Patent Description:
An event-driven vision sensor in which a pixel that detects a change in incident light intensity generates a signal in a time-asynchronous manner is known. The event-driven vision sensor is advantageous in its capability to operate at low power and high speed as compared to frame-based vision sensors, i.e., image sensors such as CCDs (Charge-Coupled Devices) or CMOSs (Complementary Metal-Oxide Semiconductors), in which all pixels are scanned at every given periodicity. Technologies regarding such an event-driven vision sensor are described, for example, in PTL <NUM> and PTL <NUM>.

Other previously proposed arrangements are disclosed in <CIT>; <CIT>; <NPL>; <CIT>; and <CIT>.

However, no technology has been proposed for optimizing control over sensor readout in consideration of time-asynchronous signal generation as described above for an event-driven vision sensor.

In light of the foregoing, it is an object of the present invention to provide a sensor control apparatus, a sensor control method, and a program that allow optimization of readout control in consideration of time-asynchronous signal generation in the event-driven vision sensor.

According to an aspect of the present invention, there is provided a sensor control apparatus as defined in claim <NUM>.

According to another aspect of the present invention, there is provided a sensor control method as defined in claim <NUM>.

According to still another aspect of the present invention, there is provided a program as defined in claim <NUM>.

According to the configuration described above, it is possible to optimize readout control in consideration of time-asynchronous signal generation in an event-driven vision sensor.

A detailed description will be given below of a preferred embodiment of the present invention with reference to attached drawings. It should be noted that redundant description regarding components having substantially the same functional configuration will be omitted in the present specification and drawings by denoting such components with the same reference signs.

<FIG> is a diagram illustrating a schematic configuration of a system according to an embodiment of the present invention, and <FIG> is a diagram illustrating a configuration of an EDS illustrated in <FIG>. In the example illustrated, a system <NUM> includes an EDS <NUM> and a sensor control apparatus <NUM>.

The EDS <NUM> is an event-driven vision sensor and includes a sensor array <NUM> and address generators <NUM> and <NUM>. The sensor array <NUM> includes sensors <NUM> that generate an event signal when a change in incident light intensity (hereinafter also referred to as an event) is detected. The address generators <NUM> and <NUM> control the event signal readout from the sensors <NUM>. In the sensor array <NUM>, the sensors <NUM> are arranged in two directions orthogonal to each other (illustrated as x and y directions), and an event signal is read out from the sensor <NUM> according to an address generated by the address generators <NUM> and <NUM> for the x and y directions, respectively. Here, in a case where the sensor <NUM> at a certain address does not detect any event, no event signal is read out even if readout is performed, after which the readout is performed from the next address. Accordingly, the event signal output from the EDS <NUM> is time-asynchronous. The event signal output from the EDS <NUM> includes sensor identification information (e.g., address), polarity of luminance change (increase or decrease), and time stamp.

The sensor control apparatus <NUM> is implemented, for example, by a computer having a communication interface, a processor, and a memory and includes functional portions of a readout control section <NUM> and a region setting section <NUM> realized as a result of operation of the processor in accordance with a program stored in the memory or received via the communication interface. Also, the memory of the sensor control apparatus <NUM> stores region information <NUM> and an event history <NUM>. The configuration of each section will be described further below.

The readout control section <NUM> controls an event signal readout frequency in the EDS <NUM>. Specifically, the readout control section <NUM> sends a control signal to the EDS <NUM>, and the address generators <NUM> and <NUM> of the EDS <NUM> generate an address for reading out an event signal from the sensor <NUM> according to the control signal. Here, the readout control section <NUM> controls the EDS <NUM>, in such a manner that the event signals are read out at a first frequency r<NUM> in a first region on the sensor array <NUM> and that the event signals are read out at a second frequency r<NUM> in a second region on the sensor array <NUM>. The second frequency r<NUM> is higher than the first frequency r<NUM> (r<NUM> < r<NUM>). Here, the term "frequency" in the present specification refers to the number of times the readout is performed in unit time. The address generators <NUM> and <NUM> of the EDS <NUM> that have received the control signal as described above generate the addresses of the sensors <NUM> in the first region r<NUM> times and the addresses of the sensors <NUM> in the second region r<NUM> times per unit time.

The first and second regions set by the processes performed by the readout control section <NUM> as described above are regions including the plurality of sensors <NUM> on the sensor array <NUM> and identified, for example, by address ranges in the x and y directions, respectively. For example, the region information <NUM> includes at least one of the address range indicating the first region and the address range indicating the second region. For example, the region information <NUM> may include only the address range indicating the second region, and the region other than the second region on the sensor array <NUM> may be identified as the first region. Alternatively, the sensor array <NUM> may be divided into a given number of unit regions in advance, and the region information <NUM> may include IDs (Identifications) of the unit regions included in each of the first and second regions. Also in this case, the readout control section <NUM> may convert the ID of each unit region into the address range, and the control signal may include information indicating the addresses in each of the first and second regions.

The region setting section <NUM> changes at least part of the first region to the second region on the basis of the number of first event signals acquired by the readout in the first region within a given period of time or changes at least part of the second region to the first region on the basis of the number of second event signals acquired by the readout in the second region within the given period of time. The region setting section <NUM> updates the region information <NUM> in the changed region. Control over the EDS <NUM> by the readout control section <NUM> as described above is premised on the fact that the frequency of detection of the events in the first region on the sensor array <NUM> is lower than the frequency of detection of the events in the second region. Accordingly, in a case where the frequency of detection of the events in part of the first region becomes comparable to or higher than that in the second region, it is suitable to change the region in question to the second region. Similarly, in a case where the frequency of detection of the events in part of the second region becomes comparable to or lower than that in the first region, it is suitable to change the region in question to the first region.

Specifically, the region setting section <NUM> performs the following processes with reference to the event history <NUM>. For example, the event history <NUM> includes the addresses and time stamps of the event signals received most recently from the EDS <NUM>. In this case, the region setting section <NUM> counts the number of event signals (first event signals) for each of subordinate regions in the first region for a most recent given period of time on the basis of these addresses and time stamps and changes the subordinate region whose number of event signals is equal to or larger than a first threshold to the second region. This makes it possible to increase the frequency of event signal readout by changing the subordinate region of the first region set at a certain point in time where more event signals have been generated to the second region. Similarly, the region setting section <NUM> counts the number of event signals (second event signals) for each of the subordinate regions in the second region for the most recent given period of time and changes the subordinate region whose number of event signals is equal to or smaller than a second threshold to the first region. This makes it possible to decrease the frequency of event signal readout by changing the subordinate region of the second region set at a certain point in time where more event signals have been generated to the first region. The first and second thresholds described above may be the same value or different values.

According to the embodiment of the present invention described above, it is possible to increase the frequency of event signal readout in the region where events occur frequently as a result of a subject moving at a relatively high speed or decrease the frequency of event signal readout in the region where events occur infrequently as a result of the subject not moving or moving at a relatively low speed. This makes it possible to relatively shorten a periodicity of event signal readout in the region where events occur frequently as compared to the case where the periodicity of the event signal readout is uniform, thus ensuring improved event signal response to the subject movement.

Also, the events irrelevant to the movement of the subject caused by noise or the like commonly occur less frequently than the events that occur due to the movement of the subject. Accordingly, it is possible to decrease a ratio of event signals derived from the events irrelevant to the movement of the subject caused by noise or the like by decreasing the frequency of event signal readout in the region where the events occur infrequently as described above, thus ensuring an improved S/N ratio of the event signals as a whole.

It should be noted that, although the readout control section <NUM> controls the frequency of event signal readout in the first and second regions set on the sensor array <NUM> of the EDS <NUM> in the example described above, more regions may be set on the sensor array <NUM>. For example, a third region may be set on the sensor array <NUM> in addition to the first and second regions, and the readout control section <NUM> of the sensor control apparatus <NUM> may control the EDS <NUM>, in such a manner that the event signals are read out in the third region at a third frequency r<NUM> higher than the second frequency r<NUM> (r<NUM> < r<NUM>). Further, a fourth region may be set on the sensor array <NUM>, and the readout control section <NUM> may control the EDS <NUM>, in such a manner that the event signals are read out in the fourth region at a fourth frequency r<NUM> higher than the third frequency r<NUM> (r<NUM> < r<NUM>).

<FIG> depicts diagrams illustrating an example in which four regions are set on the sensor array in the embodiment of the present invention. At the time of <FIG>, a first region S<NUM>, a second region S<NUM>, a third region S<NUM>, and a fourth region S<NUM> are set on the sensor array <NUM> of the EDS <NUM>. The readout control section <NUM> sends a control signal to the EDS <NUM>, for example, in such a manner that the event signals are read out from the sensors <NUM> in the respective regions in a pattern P of {S<NUM>, S<NUM>, S<NUM>, S<NUM>, S<NUM>, S<NUM>, S<NUM>, S<NUM>, S<NUM>, S<NUM>}. In this case, the ratio of the readout frequencies r<NUM>:r<NUM>:r<NUM>:r<NUM> in the respective regions is <NUM>:<NUM>:<NUM>:<NUM>. Events occur frequently in the fourth region S4 and the third region S3 due to the relatively fast movement of a subject obj.

Meanwhile, at the time illustrated in <FIG>, the region where the events occur frequently has changed as a result of the movement of the subject obj. In response thereto, in the sensor control apparatus <NUM>, the region setting section <NUM> changes settings of the regions on the basis of the numbers of signals acquired by the readout in the respective regions. Specifically, the region setting section <NUM> changes the region where the events occur more frequently as a result of the movement of the subject obj from the second region S<NUM> in <FIG> to the fourth region S<NUM> in <FIG>. Meanwhile, the region setting section <NUM> changes the region where the events occur less frequently as a result of passage of the subject obj from the fourth region S<NUM> in <FIG> to the third region S<NUM> or the second region S<NUM> in <FIG>.

In the example described above, with reference to <FIG>, for example, the region where a relatively high readout frequency is set on the sensor array <NUM> of the EDS <NUM> changes as a result of the movement of the subject obj. This makes it possible to continuously improve the response of the event signal to the movement of the subject obj. In determination made among the respective regions S<NUM>, S<NUM>, S<NUM>, and S<NUM>, for example, determination may be made by comparing the number of event signals with the threshold corresponding to each region for the most recent given period of time for each subordinate region of each region. In this case, for example, there is a possibility that the subordinate region of the region S<NUM> at a certain point in time may be changed to the region S<NUM> or the region S<NUM> at a next point in time. The determination for changing the region may be made at time intervals short enough to allow response to the movement of the subject obj.

<FIG> is a diagram illustrating a modification example of the embodiment of the present invention. Although the first region S<NUM> and the second region S<NUM> are also set on the sensor array <NUM> of the EDS <NUM> in the example illustrated, the first region S<NUM> may include the single sensor <NUM> in the example illustrated in <FIG> in contrast to the example described above with reference to <FIG> and the like in which the first and second regions include the plurality of sensors <NUM>. Also, in this example, the first frequency r<NUM> at which the event signals are read out from the sensors <NUM> in the first region S<NUM> according to the control signal sent from the readout control section <NUM> is <NUM>. That is, in the example illustrated, the address generators <NUM> and <NUM> do not generate addresses for reading out the event signals from the sensors <NUM> in the first region S<NUM>.

In the example described above, the sensor <NUM> included in the first region S<NUM> is a defective pixel, i.e., a sensor that does not detect a change in light intensity. Accordingly, no effective event signal is generated from the sensor <NUM>. In such a case, it is possible to shorten the periodicity of event signal readout from other sensors <NUM> and improve an S/N ratio of the event signals as a whole by setting the frequency of event signal readout from the sensor <NUM> in question to <NUM>, i.e., by not performing the event signal readout. It should be noted, however, that once set to the first region S<NUM>, the sensor <NUM> will not be set back to the second region S<NUM> because no event signal is read out. Accordingly, only in a case where, for example, the number of event signals (second event signals) acquired by the readout from the sensor <NUM> in the second region S<NUM> is <NUM> for a period of time longer than the periodicity at which the sensor <NUM> generates the event signals due to noise, the region setting section <NUM> may change the subordinate region of the second region S<NUM> corresponding to the sensor <NUM> in question to the first region S<NUM>.

<FIG> is a flowchart illustrating examples of processes of a sensor control method according to the embodiment of the present invention. In the example illustrated, the address generators <NUM> and <NUM> generate an address according to a control signal sent from the readout control section <NUM> that has referred to the region information <NUM> in the sensor control apparatus <NUM>, thus allowing the event signals to be read out from the sensors <NUM>. Specifically, the event signals are read out from the sensors <NUM> in the first region on the sensor array <NUM> at a first frequency (step S101), and the event signals are read out from the sensors <NUM> in the second region at a second frequency (step S102). It should be noted that no event signals are generated in a case where the sensors in the respective regions do not detect any event as described earlier. Accordingly, the number of event signals read out in steps S101 and S102 varies from one time to another.

After the readout described above is repeated until a given period of time elapses (step S103), a region updating process is performed by the region setting section <NUM>. Specifically, the region setting section <NUM> changes, of the subordinate regions of the first region, the subordinate region whose number of event signals acquired within the given period of time is equal to or larger than the first threshold to the second region with reference to the event history <NUM> (step S104). Also, the region setting section <NUM> changes, of the subordinate regions of the second region, the subordinate region whose number of event signals acquired within the given period of time is equal to or smaller than the second threshold to the first region (step S105). The region setting section <NUM> updates the region information <NUM> in the changed region (step S106), and the processes in the steps S101 to S106 described above are repeated every given period of time (step S107).

The embodiment of the present invention can be used, for example, in a game controller, a smartphone, and various mobile objects (e.g., automobile, electric vehicle, hybrid electric vehicle, motorcycle, bicycle, personal mobility, airplane, drone, ship, robot) to acquire information regarding a surrounding environment, estimate a self-position from a position of a surrounding object, and detect and avoid an oncoming flying object.

Claim 1:
A sensor control apparatus (<NUM>) comprising:
a readout control section (<NUM>) configured to send a control signal to an event-driven vision sensor (<NUM>) comprising a sensor array (<NUM>) comprising a plurality of sensors (<NUM>) that are configured to generate event signals when a change in incident light intensity is detected, and two address generators (<NUM>, <NUM>) configured to generate an address for each sensor, wherein each address is respective identification information for each sensor, and periodically perform a readout on the sensors, wherein, for a given readout, if a given sensor has generated an event signal when the given readout is being performed on the given sensor, then the event signal is read out according to the address generated for the given sensor, otherwise no event signal is read out from the given sensor, after which performance of the given readout progresses onto a sensor adjacent to the given sensor, wherein the control signal is based on region information (<NUM>) comprising at least one of an address range indicating a first region (s1) of the sensor array and an address range indicating a second region (s2) of the sensor array, wherein the two address generators are configured to control, in response to the received control signal, the periodicity of the readout such that readout of event signals is performed at a first frequency on the first region and readout of event signals is performed at a second frequency higher than the first frequency on the second region; and
a region setting section (<NUM>) configured to update the region information by changing at least part of the first region to the second region on a basis of the number of first event signals acquired by the readout in the first region within a given period of time or by changing at least part of the second region to the first region on a basis of the number of second event signals acquired by the readout in the second region within the given period of time.