Patent Description:
An event-driven vision sensor is known in which pixels that detect a change in the intensity of incident light generate signals asynchronously in time. The event-driven vision sensor is advantageous in that the sensor can operate at low power and high speed compared to a frame-type vision sensor that scans all pixels at predetermined cycles, specifically, an image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). Techniques regarding such an event-driven vision sensor are described in PTL <NUM> and PTL <NUM>, for example.

<CIT> discloses to move an event-driven sensor in order to cause a change in the intensity of incident light with the sensor elements for obtaining an image, even if the target object does not move.

The paper by <NPL>, teaches an experimental setup wherein an event-driven sensor is rotated. An optimum delay time between causing the rotary motion and the sensor's reaction is found as a result of measuring a sequence of data.

However, although the above-mentioned advantages are known regarding the event-driven vision sensor, it is hard to say that peripheral technologies in which characteristics different from those of the conventional vision sensor such as the frame-type vision sensor are considered have been sufficiently proposed.

Therefore, an object of the present invention is to provide an electronic device, an actuator control method, and a program that provide convenience by interaction of an event-driven vision sensor with an actuator.

The above problems are solved by the subject-matter of the independent claims.

According to one aspect of the present invention, provided is an electronic device according to claim <NUM> including an event-driven vision sensor that includes a sensor array having a sensor that generates an event signal when the sensor detects a change in intensity of incident light, and an actuator that displaces a module including the vision sensor, and a control unit that transmits a control signal to the actuator and reflects a correction value based on the event signal generated when the actuator displaces the module in the control signal.

According to another aspect of the present invention, provided is an actuator control method according to claim <NUM> using an event-driven vision sensor that includes a sensor array having a sensor that generates an event signal when the sensor detects a change in the intensity of incident light, and the method includes steps of driving the actuator to displace a module including the vision sensor, and reflecting the event signal generated when the actuator displaces the module in the control signal of the actuator.

According to yet another aspect of the present invention, provided is a program according to claim <NUM> causing a processing circuit connected to an event-driven vision sensor that includes a sensor array having a sensor that generates an event signal when the sensor detects a change in the intensity of incident light to execute steps of driving an actuator to displace a module including the vision sensor and reflecting the event signal generated when the actuator displaces the module in the control signal of the actuator.

According to the above configuration, convenience can be provided by interaction of the event-driven vision sensor with the actuator.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. Incidentally in the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

<FIG> is a block diagram illustrating a schematic configuration of an electronic device including a sensor module according to the first embodiment of the present invention. As illustrated in <FIG>, an electronic device <NUM> includes a sensor module <NUM> and a control unit <NUM>.

The sensor module <NUM> includes an event-driven vision sensor <NUM>, an actuator <NUM>, and a shutter <NUM>. The vision sensor <NUM> includes a sensor array <NUM> having sensors 111A, 111B,. corresponding to pixels of an image, and a processing circuit <NUM> connected to the sensor array <NUM>. The sensors 111A, 111B,. include light receiving elements, and generate event signals when detecting the intensity change of incident light, more specifically, a brightness change. The event signal is output from the processing circuit <NUM> as a time stamp, sensor identification information (a pixel position, for example), and information indicating polarity of brightness change (increasing or decreasing), for example. When the subject moves within the angle of view of the sensor array <NUM>, since the intensity of the light reflected or scattered by the subject changes, the movement of the subject can be detected in chronological order due to the event signal generated by the sensors 111A, 111B,. corresponding to the edge of the subject, for example.

Here, as already described, the event-driven vision sensor <NUM> is advantageous in that the sensor can operate at high speed with low power consumption as compared with the frame type vision sensor. This is because only the sensors that detect the change in brightness generate event signals, among the sensors 111A, 111B,. that constitute the sensor array <NUM>. Since the sensor that does not detect the brightness change does not generate an event signal, the processing circuit <NUM> can process and transmit at high speed only the event signal of the sensor that has detected the brightness change. Further, in the case where there is no change in brightness, processing and a transmission process do not occur, so that operation with low power becomes possible. On the other hand, even if the subject exists within the angle of view of the sensor array <NUM>, the brightness does not change unless the subject moves, and thus it is difficult to capture the subject which is not moving, by using the event signals generated by the sensors 111A, 111B,. That is, it is difficult to obtain information regarding the surrounding environment including a stationary subject only with the vision sensor <NUM>.

In the present embodiment, the sensor module <NUM> includes the actuator <NUM> connected to the vision sensor <NUM>. The actuator <NUM> is driven in accordance with a control signal transmitted from the control unit <NUM>, and is configured to displace the sensor array <NUM> in a direction perpendicular to the optical axis direction of the sensors 111A, 111B,. , for example. When the actuator <NUM> displaces the sensor array <NUM>, the positional relationships among all the sensors 111A, 111B,. and the subject change. That is, at this time, the same change as when all the subjects have moved within the angle of view of the sensor array <NUM> occurs. Accordingly, regardless of whether or not the subject is actually moving, the subject can be detected by the event signal generated by the sensors 111A, 111B,. corresponding to the edge of the subject, for example. Since the amount of displacement of the sensor array <NUM> required to generate the above change is not large, the actuator <NUM> may be a device such as a vibrator that slightly displaces or vibrates the sensor array <NUM>.

Note that in the above description, an example in which the direction in which the actuator <NUM> displaces the sensor array <NUM> is perpendicular to the optical axis direction of the sensors 111A, 111B,. has been described, but in the case where the displacement direction is not perpendicular to the optical axis direction, that is, for example, even if the displacement direction is parallel to the direction of the optical axis, the positional relationships among all the sensors 111A, 111B,. and the subject change. Therefore, the actuator <NUM> may displace the sensor array <NUM> in a given direction. Note that in the configuration in which the displacement direction is perpendicular to or nearly perpendicular to the optical axis direction, it is advantageous in that the displacement amount of the sensor array <NUM> required to generate the above change is minimized, and the positional relationship with the subject changes in a substantially uniform manner throughout the sensors 111A, 111B,.

Further, in the present embodiment, the sensor module <NUM> includes the shutter <NUM>. The shutter <NUM> is arranged such that the entire angle of view of the sensor array <NUM> of the vision sensor <NUM> can be shielded and opened. The shutter <NUM> may be a mechanical shutter such as a focal plane shutter or a lens shutter, or an electronic shutter such as a liquid crystal shutter. When the shutter <NUM> that has been open is closed, the entire angle of view of the sensor array <NUM> is shielded, so that the intensity of light incident on all the sensors 111A, 111B,. becomes minimal and constant in principle. Further, when the shutter <NUM> that has been closed is opened, the entire angle of view of the sensor array <NUM> is opened, which causes a change to raise brightness in all the sensors 111A, 111B,. in principle. As will be described later, in the present embodiment, such an operation is used to calibrate the sensor array <NUM> and detect a self-luminous subject.

The control unit <NUM> includes a communication interface <NUM>, a processing circuit <NUM>, and a memory <NUM>. The communication interface <NUM> receives an event signal transmitted from the processing circuit <NUM> of the vision sensor <NUM> and outputs the event signal to the processing circuit <NUM>. Further, the communication interface <NUM> transmits the control signal generated by the processing circuit <NUM> to the actuator <NUM>. The processing circuit <NUM> operates in accordance with a program stored in the memory <NUM>, for example, and processes the received event signal. For example, the processing circuit <NUM> generates images that map the positions where the brightness change occurs in chronological order on the basis of the event signal, and temporarily or continuously stores the image in the memory <NUM>, or further sends the image to another device via the communication interface <NUM>. Further, the processing circuit <NUM> generates respective control signals for driving the actuator <NUM> and the shutter <NUM>.

<FIG> is a sequence diagram illustrating a first example of the operation of the sensor module according to the first embodiment of the present invention. In the illustrated example, first, the control signal generated by the processing circuit <NUM> of the control unit <NUM> is transmitted to the actuator <NUM> (S101). When the actuator <NUM> that has received the control signal is driven (S102), the sensor array <NUM> is displaced in a predetermined direction, and the event signals generated by the sensors 111A, 111B,. corresponding to the edges of all the subjects in principle are transmitted from the vision sensor <NUM> to the control unit <NUM> (S103). The processing circuit <NUM> detects a subject from the received event signal (S104). As described above, at this time, the subject can be detected regardless of whether or not the subject is actually moving. The processing circuit <NUM> may execute procedures from transmission of the control signal to the actuator <NUM> (S101) to reception of the event signal (S103) and capture of environmental information based on the event signal (S104) as a series of procedures. For example, the processing circuit <NUM> may treat an event signal received during a predetermined time period from the transmission of the control signal to the actuator <NUM> (S101), separately from an event signal received at a time, as an event signal indicating environmental information.

<FIG> is a sequence diagram illustrating a second example of the operation of the sensor module according to the first embodiment of the present invention. In the illustrated example, first, in the state where the shutter <NUM> is open, the control signal generated by the processing circuit <NUM> of the control unit <NUM> is transmitted to the shutter <NUM> (S111). By closing the shutter <NUM> that has received the control signal (S112), the entire angle of view of the sensor array <NUM> is shielded, and the intensity of light incident on all the sensors 111A, 111B,. becomes minimal and constant. Accordingly, the event signals should not be received in principle, after the event signals indicating that the brightness has decreased due to the light being blocked are transmitted from the vision sensor <NUM> to the control unit <NUM> (S113). However, in the case where the sensor is defective or noise is detected as a brightness change due to improper setting of the threshold value of the brightness change for generating an event signal in the sensor, for example, an event signal can be generated while the shutter <NUM> is shielding the angle of view of the sensor array <NUM>. Therefore, in the control unit <NUM>, the processing circuit <NUM> maintains the shutter <NUM> to be closed for a predetermined time period, and monitors the event signal received while the shutter <NUM> is shielding the angle of view of the sensor array <NUM>. In the case where an event signal is received during this period (S114), the processing circuit <NUM> calibrates the vision sensor <NUM> on the basis of the received event signal (S115). To be specific, the processing circuit <NUM> identifies the sensor having generated the event signal, as a defective pixel (luminescent spot), or adjusts the threshold value of the brightness change for generating the event signal in the sensor.

<FIG> is a sequence diagram illustrating a third example of the operation of the sensor module according to the first embodiment of the present invention. In the illustrated example, first, in the state where the shutter <NUM> is closed, the control signal generated by the processing circuit <NUM> of the control unit <NUM> is transmitted to the shutter <NUM> (S121). When the shutter <NUM> that has received the control signal is opened (S122), the entire angle of view of the sensor array <NUM> is opened, and event signals indicating that the brightness has increased in all the sensors 111A, 111B,. in principle are transmitted from the vision sensor <NUM> to the control unit <NUM> (S123). After that, the control signal generated by the processing circuit <NUM> of the control unit <NUM> is transmitted to the shutter <NUM> again (S125), and when the shutter <NUM> is closed (S126), so that the entire angle of view of the sensor array <NUM> is shielded, event signals indicating that the brightness has decreased in all the sensors 111A, 111B,. are transmitted from the vision sensor <NUM> to the control unit <NUM> (S127). In this way, the control unit <NUM> transmits a control signal for repeating the shielding and opening of the angle of view of the sensor array <NUM> to the shutter <NUM>, and receives the event signal generated by the vision sensor <NUM> meanwhile, particularly during the period from the opening to the shielding of the angle of view.

Here, if the time period t1 from the opening (S122) to the shielding (S126) of the angle of view by the shutter <NUM> is short (specifically, <NUM> msec or less, for example), the subject hardly moves, and therefore the event signal indicating that the subject moves should not be received. As an exception, in the case where the blinking cycle of the light source in a self-luminous subject such as an illumination or a display is shorter than the time period t1, an event signal indicating the blinking of these subjects is received (S124). Accordingly, by making the time period t1, namely, the cycle of repeating the shielding and opening of the angle of view longer than the blinking cycle of the light source included in the self-luminous subject (while keeping the time period t1 short as described above), the control unit <NUM> can identify the self-luminous subject on the basis of the received event signal (S128).

In the first embodiment of the present invention as described above, due to the actuator <NUM> displacing the sensor array <NUM>, an event is forcibly generated in the vision sensor <NUM>, and information regarding the surrounding environment including a stationary subject, for example, can be obtained. Further, in the present embodiment, the sensor array <NUM> can be calibrated due to the shutter <NUM> shielding the entire angle of view of the sensor array <NUM>. Still further, by repeating the opening and closing of the shutter <NUM> at a predetermined cycle, a self-luminous subject such as an illumination or a display can be detected.

Note that in the above example, the sensor module <NUM> includes both the actuator <NUM> and the shutter <NUM>, but since these functions are independent of each other, either the actuator <NUM> or the shutter <NUM> may be included in the sensor module <NUM>. Further, although the control unit <NUM> is illustrated and described separately from the sensor module <NUM> in the above example, the control unit <NUM> may be included in the sensor module <NUM>. In this case, the processing circuit <NUM> of the sensor module <NUM> and the processing circuit <NUM> of the control unit <NUM> may be configured separately or may be common.

<FIG> is a block diagram illustrating a schematic configuration of an electronic device including a sensor module according to a second embodiment of the present invention. As illustrated in <FIG>, an electronic device <NUM> includes a sensor module <NUM>, the control unit <NUM>, and a movable support mechanism <NUM>.

The sensor module <NUM> includes an event-driven vision sensor <NUM> similar to one in the first embodiment, and the shutter <NUM>. The sensor module <NUM> is supported by the movable support mechanism <NUM> including frames 410A, 410B, and 410C and actuators 420A and 420B. In the illustrated example, the actuators 420A and 420B are rotary actuators driven in accordance with a control signal transmitted from the control unit <NUM>. The actuator 420A causes a rotational displacement of a predetermined angle between the frames 410A and 410B in accordance with the control signal, and the actuator 420B similarly causes a rotational displacement of a predetermined angle between the frames 410B and 410C. Thereby, the actuators 420A and 420B displace the sensor module <NUM> including the vision sensor <NUM>.

Also in the present embodiment, for example, by using the actuator 420B in the same manner as the actuator <NUM> of the first embodiment to forcibly generate an event in the vision sensor <NUM>, information regarding the surrounding environment including a stationary subject, for example, can be obtained. In this case, for example, the actuator 420B may be understood to be included in the sensor module <NUM>. In addition, in the present embodiment, as in the example described below, the control unit <NUM> can reflect the correction value in the control signals of actuators 420A and 420B on the basis of the event signal generated by the vision sensor <NUM> when the actuators 420A and 420B displace the sensor module <NUM>.

<FIG> is a sequence diagram illustrating a first example of the operation of the sensor module according to the second embodiment of the present invention. In the illustrated example, first, the control signal generated by the processing circuit <NUM> of the control unit <NUM> is transmitted to one or both of the actuators 420A and 420B (S131). When the actuators 420A and 420B are driven in accordance with the control signal (S132), the sensor module <NUM> is displaced, and the positional relationships among the sensors 111A, 111B,. and the subject change. At this time, the event signals generated by the sensors 111A, 111B,. are transmitted from the vision sensor <NUM> to the control unit <NUM> (S133). In the control unit <NUM>, the processing circuit <NUM> measures the delay time period d1 from the transmission of the control signal to the actuators 420A and 420B (S131) to the reception of the event signal (S133) and calibrates the actuators 420A and 420B on the basis of the delay time period d1 (S134). To be specific, the processing circuit <NUM> determines a correction value of the control signal according to the delay time period d1, and the determined correction value is reflected in the control signal subsequently generated by the processing circuit.

In the above example, for example, if the control signal is transmitted to either the actuator 420A or 420B, the actuator 420A or the actuator 420B can be calibrated independently. Further, if the control signal is transmitted to both of the actuators 420A and 420B, the composite system including the actuators 420A and 420B can be calibrated. The correction value of the control signal determined according to the delay time period d1 is used, for example, when the control unit <NUM> corrects the parameters of the proportional-integral-derivative (PID) control executed in the case where the actuators 420A and 420B are desired to implement the displacement following a specific pattern.

<FIG> is a sequence diagram illustrating a second example of the operation of the sensor module according to the second embodiment of the present invention. In the illustrated example, similarly to the example illustrated above in <FIG>, the control signal is transmitted (S131), and the actuators 420A and 420B that have received the control signal drive to cause rotation displacement in the vision sensor <NUM> (S132).

Here, for example, in the case where the actuators 420A and 420B are worn, the rotational displacement of the vision sensor <NUM> does not become stable instantaneously, and vibration occurs, for example. In this case, the event signals generated by the sensors 111A, 111B,. due to change of the positional relationships among the sensors 111A, 111B,. and the subject are transmitted from the vision sensor <NUM> to the control unit <NUM> at a plurality of timings (S133-<NUM> and S133-<NUM>). The processing circuit <NUM> measures the delay time periods d1 and d2 from the transmission of the control signal to the actuators 420A and 420B (S131) to the reception of the event signals at a plurality of timings (S133-<NUM> and S133-<NUM>), respectively. Due to this, as a result, the processing circuit <NUM> measures the elapsed time period d2-d1 from the start of reception of the event signal (S133-<NUM>) to the end of the reception (S133-<NUM>). The processing circuit <NUM> determines a correction value according to the elapsed time period d2-d1, and the determined correction value is reflected in the control signal generated by the processing circuit thereafter. To be specific, the processing circuit <NUM> sets a flag indicating that the actuators 420A and 420B are worn in the case where the elapsed time period d2-d1 exceeds a threshold value. In this case, the processing circuit <NUM> may set a value such as an operating torque different from that of the other actuators for the actuators 420A and 420B in which wear has been generated.

<FIG> is a block diagram illustrating a configuration example of a processing circuit of a control unit in the case of executing motion prediction in the second embodiment of the present invention. In the illustrated example, the processing circuit <NUM> of the control unit <NUM> includes, for example, a drive pattern generating section <NUM>, a control signal generating section <NUM>, an event signal analysis section <NUM>, an error calculating section <NUM>, and a motion predicting section <NUM>, as functions implemented by operation according to a program stored in the memory <NUM>. The drive pattern generating section <NUM> generates drive patterns for the actuators 420A and 420B. Here, the drive pattern may be predetermined by, for example, a program stored in the memory <NUM>, or is determined on the basis of the measured values of other sensors such as the acceleration sensor included in the electronic device <NUM>. The control signal generating section <NUM> generates control signals for the actuators 420A and 420B in accordance with the drive pattern generated by the drive pattern generating section <NUM>.

When the actuators 420A and 420B are driven in accordance with the control signal generated by the control signal generating section <NUM>, the event signal is transmitted from the vision sensor <NUM> to the control unit <NUM>. In the processing circuit <NUM>, the event signal analysis section <NUM> executes back calculation of the displacement of the sensor module <NUM> from the received event signal. To be specific, for example, the event signal analysis section <NUM> executes back calculation of the motion vector of the vision sensor <NUM> from the motion vector of the subject obtained by analyzing the event signal. The event signal analysis section <NUM> provides the error calculating section <NUM> with information including the displacement of the sensor module <NUM> obtained by back calculation. The error calculating section <NUM> calculates the error characteristics of the actuators 420A and 420B from the difference between the displacement of the sensor module <NUM> obtained by back calculation and the drive pattern generated by the drive pattern generating section <NUM>, while taking into consideration the delay time period d1 of the operation of the actuators 420A and 420B specified by the example described above with reference to <FIG>, for example. The error characteristics may be normalized for each type of movement of the actuators 420A and 420B (specifically, translation and rotation in each axial direction), for example, to be stored in the memory <NUM>.

After that, in the case where the drive pattern generating section <NUM> generates a new drive pattern for the actuators 420A and 420B, the control signal generating section <NUM> inputs the generated control signal to the motion predicting section <NUM> before outputting the control signal. The motion predicting section <NUM> predicts the motion of the actuators 420A and 420B with respect to the input control signal on the basis of the error characteristics of the actuators 420A and 420B calculated by the error calculating section <NUM>. The control signal generating section <NUM> corrects the control signal such that the difference between the movement predicted by the motion predicting section <NUM> and the drive pattern generated by the drive pattern generating section <NUM> becomes small. Further, the control signal generating section <NUM> inputs the corrected control signal to the motion predicting section <NUM> again, and the motion predicting section <NUM> predicts again the movements of the actuators 420A and 420B with respect to the control signal corrected on the basis of the error characteristics, and then, the control signal generating section <NUM> may correct the control signal again such that the difference between the motion predicted again and the drive pattern becomes small.

In the second embodiment of the present invention as described above, in addition to the effect of the first embodiment described above, the processing circuit <NUM> of the control unit <NUM> can calibrate the delay amount of the actuators 420A and 420B and detect the vibration due to the wear of the internal components of the actuators 420A and 420B by measuring the delay time periods d1 and d2 from transmission of the control signal to the actuators 420A and 420B to the reception of event signals. Further, in the present embodiment, the processing circuit <NUM> implements the functions of the error calculating section <NUM> and the motion predicting section <NUM> to correct the control signal in consideration of the error generated in the motion of the actuators 420A and 420B, and can operate the actuators 420A and 420B more accurately with respect to the intended drive pattern.

Note that in the above example, the calibration of the delay amount of the actuators 420A and 420B, the detection of vibration, and the correction of the control signal have been described in the same embodiment, but since these operations can be performed independently of each other, a part of these may be implemented and the rest may not be implemented in the electronic device <NUM> or the sensor module <NUM>. Further, in the above example, the vision sensor <NUM> has been described as being capable of forcibly generating an event similarly to the first embodiment, but this function is not essential. Since the shutter <NUM> is not essential either, the vision sensor <NUM> does not have to include the shutter <NUM> in the present embodiment.

Although some embodiments of the present invention have been described in detail with reference to the accompanying drawings hereinabove, the present invention is not limited to such examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various alteration examples or modification examples within the scope of the claims.

Claim 1:
An electronic device (<NUM>, <NUM>) comprising:
an event-driven vision sensor (<NUM>) that includes a sensor array (<NUM>) having a sensor (111A, 111B) that generates an event signal when the sensor detects a change in intensity of incident light;
an actuator (<NUM>) configured to displace a module (<NUM>,<NUM>) including the vision sensor (<NUM>); and
a control unit (<NUM>) configured to transmit a control signal to the actuator (<NUM>) and to determine a correction value based on the event signal generated when the actuator (<NUM>) displaces the module (<NUM>, <NUM>) in response to the control signal and to use the correction value for generating a subsequent control signal,
wherein
the control unit (<NUM>) is configured to determine the correction value by:
measuring an elapsed time period from a start of reception of the event signal to an end of the reception, and
determining the correction value according to the elapsed time period, wherein
the correction value includes a flag indicating that the actuator (<NUM>) is worn, and
the control unit (<NUM>) sets the flag when the elapsed time period exceeds a threshold value.