Patent Description:
An event-driven vision sensor in which a pixel that has detected a change in the intensity of incident light generates a signal asynchronously in time has been known. The event-driven vision sensor is advantageous in that a low-power and high-speed operation can be performed as compared to a frame-type vision sensor, specifically, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) that scans all pixels in each predetermined cycle. Techniques related to such an event-driven vision sensor are described in, for example, PTL <NUM> and PTL <NUM>.

As a method of transmitting an event signal generated by such an event-driven vision sensor, an AER (Address Event Representation) format has been known. In the AER format, for example, four pieces of data of an x coordinate and a y coordinate of a pixel in which an event has occurred, ON/OFF (polarity) of the event, and a time stamp are transferred.

Further previously proposed arrangements are disclosed in <CIT> and <CIT>. <CIT> discloses a buffer control method in which a first write end signal is received, and, after a predetermined delay, a second write end signal is generated, following which the buffer is forcibly read. <CIT> discloses an even-driven vision sensor which may comprise a FIFO buffer at each end of the columns, to increase the frame rate.

However, since signals are generated asynchronously in time in the event-driven vision sensor, an amount of data transmitted per time in the AER format varies, and thus, there has been a problem that the burst size of a buffer is not filled and a delay occurs in data processing.

Therefore, an object of the present invention is to provide a signal processing device, electronic equipment, a signal processing method, and a program capable of eliminating a processing delay of data output from an event-driven vision sensor.

The invention provides an electronic equipment as set out in appended claim <NUM> and a signal processing method as set out in appended claim <NUM>.

According to the above configuration, it is possible to suppress the processing delay of the data output from the event-driven vision sensor.

Hereinafter, several embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that constitutional elements having substantially the same functional configurations are denoted by the same reference signs in the specification and the drawings, and the duplicated description thereof is omitted.

<FIG> is a block diagram illustrating a schematic configuration of electronic equipment according to a first embodiment of the present invention. As illustrated in <FIG>, electronic equipment <NUM> includes an event-driven vision sensor <NUM> and a control unit <NUM>.

The vision sensor <NUM> includes a sensor array <NUM> including sensors 110A, 110B, and the like corresponding to pixels of an image, and a processing circuit <NUM> connected to the sensor array <NUM>. The sensors 110A, 110B, and the like include light receiving elements, and generate event signals when detecting a change in the intensity of incident light, more specifically, a change in luminance. The event signal is output from the processing circuit <NUM> as information indicating, for example, a time stamp, identification information (for example, the position of the pixel) of the sensor, and the polarity (increase or decrease) of a change in luminance. As will be described later, the processing circuit <NUM> adds a transfer end packet to the packets of the event signals. When a subject moves within the angle of view of the sensor array <NUM>, the intensity of light reflected or scattered by the subject changes, and thus, the movement of the subject can be detected in time series by the event signals generated by the sensors 110A, 110B, and the like corresponding to, for example, the edge of the subject.

The control unit <NUM> includes a communication interface <NUM>, a processing circuit <NUM>, and a memory <NUM>. The communication interface <NUM> receives packets transmitted from the processing circuit <NUM> of the vision sensor <NUM>. The packets are temporarily stored in a buffer memory <NUM>. The processing circuit <NUM> operates in accordance with, for example, a program stored in the memory <NUM>, and processes packets read out from the buffer memory <NUM>. For example, the processing circuit <NUM> generates an image in which the positions where changes in luminance have occurred are mapped in time series on the basis of the event signals included in the packets, and temporarily or continuously stores the image in the memory <NUM> or further transmits the image to another device via the communication interface <NUM>. Here, the processing circuit <NUM> includes a readout control unit <NUM> that forcibly reads out the packets from the buffer memory <NUM> when predetermined conditions are satisfied.

<FIG> are diagrams each conceptually describing the transfer end packet in the first embodiment of the present invention. In the examples illustrated in <FIG>, the minimum unit (one packet) of the event signal output from the vision sensor <NUM> is four bytes, a the size of the buffer memory <NUM> in the control unit <NUM> is <NUM> bytes (corresponding to <NUM> packets).

In the example illustrated in <FIG>, the readout control unit <NUM> is configured to read out packets <NUM> from the buffer memory <NUM> when the capacity of the buffer memory <NUM> becomes full. Here, in a case in which the number of events detected by the sensor array <NUM> is small and the vision sensor <NUM> outputs only the event signals corresponding to five packets, that is, <NUM> bytes, the capacity of the buffer memory <NUM> is still available. Therefore, the readout control unit <NUM> does not read out the packets <NUM> from the buffer memory <NUM>. This situation continues until the capacity of the buffer memory <NUM> becomes full due to the event signals received thereafter or until the packets <NUM> are forcibly read out from the buffer memory <NUM> due to timeout. Accordingly, there is a possibility that a delay occurs between the generation of the event signal by the vision sensor <NUM> and the processing of the event signal by the processing circuit <NUM>.

The situation as described above occurs because the event signal is generated asynchronously in time in the event-driven vision sensor <NUM>. Among the sensors 110A, 110B, and the like configuring the sensor array <NUM> in the vision sensor <NUM>, only the sensor that has detected a change in luminance generates the event signal, and the sensor that has not detected a change in luminance does not generate the event signal, so that a low-power and high-speed operation can be performed as compared to a frame-type vision sensor. In contrast, in the vision sensor <NUM>, the amount of data transmitted per time varies, unlike the frame-type vision sensor which transmits data corresponding to all the pixels at the time of scanning, and accordingly, there is a possibility that the delay as described above may occur.

Thus, the processing circuit <NUM> in the vision sensor <NUM> adds a transfer end packet <NUM> to the packets <NUM> of the event signals in the embodiment as in the examples illustrated in <FIG>. More specifically, when scanning of the sensors 110A, 110B, and the like is completed, the processing circuit <NUM> adds the transfer end packet <NUM> to the packets <NUM> of the event signals. In this case, when the transfer end packet <NUM> is stored in the buffer memory <NUM>, the readout control unit <NUM> forcibly reads out the packets <NUM> from the buffer memory <NUM> irrespective of the free capacity of the buffer memory <NUM>. In the example illustrated in <FIG>, when finding the transfer end packet <NUM>, the readout control unit <NUM> stores padding data <NUM> in the free capacity of the buffer memory <NUM> to fill up the capacity of the buffer memory <NUM>, and the packets <NUM> are accordingly sent to the process in the processing circuit <NUM>. In addition, in the example illustrated in <FIG>, when finding the transfer end packet <NUM>, the readout control unit <NUM> may be also configured to forcibly read out the packets <NUM> from the buffer memory <NUM> without padding and send the packets to the process in the processing circuit <NUM>.

<FIG> is a flowchart illustrating an example of a process of the processing circuit of the vision sensor in the first embodiment of the present invention. In the illustrated example, the processing circuit <NUM> executes scanning of the sensor array <NUM> (Step S101). Here, the scanning of the sensor array <NUM> is executed by scanning the sensors 110A, 110B, and the like in a predetermined order and acquiring the event signal only from the sensor that has detected a change in luminance. For example, although scanning is executed in the same manner even in the frame-type vision sensor, the scanning of the event-driven vision sensor <NUM> is executed at a higher speed because a process for the sensor that has not detected a change in luminance is skipped. When the scanning in Step S101 is completed, the processing circuit <NUM> adds the transfer end packet to the packets of the event signals (Step S102). The processing circuit <NUM> repeats these processes (Step S103).

<FIG> is a flowchart illustrating an example of a process of the readout control unit in the first embodiment of the present invention. In the illustrated example, the readout control unit <NUM> stores the packets of the event signals received from the vision sensor <NUM> in the buffer memory <NUM> (Step S121). The readout control unit <NUM> monitors the packets of the event signals stored in the buffer memory <NUM>, and in a case in which the transfer end packet <NUM> is received or stored (Step S122), the packets <NUM> are read out from the buffer memory <NUM> (Step S123). Even in a case in which the capacity of the buffer memory <NUM> becomes full (Step S124), the readout control unit <NUM> reads out the packets <NUM> from the buffer memory <NUM> (Step S123). The readout control unit <NUM> repeats these processes (Step S125).

In the above example, in a case in which the number of events detected by the sensor array <NUM> is large and the packets <NUM> to be transmitted exceed the capacity of the buffer memory <NUM>, the packets <NUM> are sequentially read out from the buffer memory <NUM> by the process in Step S105 described above. If the transfer efficiency in a case in which the transmission amount is large as described above is considered, it is preferable that the capacity of the buffer memory <NUM> is large to some extent. In the embodiment, in a case in which the number of event signals output from the vision sensor <NUM> is small, the forcible readout is executed by the transfer end packet <NUM> as described above, so that the processing delay of the event signals in the control unit <NUM> can be suppressed to a time corresponding to one scan of the sensor array <NUM> at most while securing the capacity of the buffer memory <NUM>.

In the second embodiment, the processing circuit <NUM> of the vision sensor <NUM> adds a transfer start packet to the packets of the event signals in the same configuration of the electronic equipment as the above-described first embodiment. The process of the processing circuit <NUM> in the embodiment may be a process in which the order of Step S101 and Step S102 is switched and the transfer end packet is replaced with the transfer start packet in, for example, the example described above by using <FIG>. In other respects, the configuration of the embodiment is the same as that of the above-described first embodiment, and thus, the duplicated and detailed description thereof is omitted.

<FIG> are diagrams each conceptually describing the transfer start packet in the second embodiment of the present invention. In the examples illustrated in <FIG>, the buffer memory <NUM> is configured in the same manner as the examples described above with reference to <FIG>. Note that a case in which the transfer start packet is not received is the same as the example described with reference to <FIG>, and is accordingly omitted.

In the example illustrated in <FIG>, when a transfer start packet <NUM> is newly received, the readout control unit <NUM> forcibly reads out the packets <NUM> of the event signals stored in the buffer memory <NUM> at that time, and sends the packets <NUM> to the process in the processing circuit <NUM>. The readout control unit <NUM> may perform the padding as in the example illustrated in <FIG>. In this case, the processing delay of the event signals in the control unit <NUM> is suppressed to a time corresponding to one scan of the sensor array <NUM> at most.

Meanwhile, in the example illustrated in <FIG>, when the number of packets <NUM> of the event signals stored in the buffer memory <NUM> reaches a predetermined number n counted from the transfer start packet <NUM>, the readout control unit <NUM> forcibly reads out the packets <NUM> of the event signals stored in the buffer memory <NUM> at that time, and sends the packets to the process in the processing circuit <NUM>. For example, the predetermined number n can be statistically determined as the mean value or the maximum value of the numbers of event signals output in one scan of the sensor array <NUM>.

Also in the embodiment, the processing delay of the event signals in the control unit <NUM> can be suppressed while securing the capacity of the buffer memory <NUM> as similar to the above-described first embodiment. In addition, the embodiment can be also applied to, for example, a case in which the end of scanning is not specified in the processing circuit <NUM> because the event signals generated by the vision sensor <NUM> is asynchronous in time.

In the above-described first and second embodiments of the present invention, the processing circuit <NUM> of the vision sensor <NUM> adds a delimiter (separator) packet such as the transfer end packet <NUM> or the transfer start packet <NUM> to the packets corresponding to the event signals, so that the processing delay of the event signals is prevented. Note that the processing circuit <NUM> does not necessarily execute the scanning of the sensor array <NUM> as described above. In this case, for example, the processing circuit <NUM> transmits the delimiter packet to the control unit <NUM> in each predetermined time cycle, and the readout control unit <NUM> handles the delimiter packet in the same manner as the transfer end packet <NUM> and the transfer start packet <NUM> in the above example, so that the processing delay of the event signals can be prevented.

Next, a third embodiment of the present invention will be described; this embodiment does not fall within the scope of the claims. In the third embodiment, in the same configuration of the electronic equipment as the above-described first embodiment, the processing circuit <NUM> of the vision sensor <NUM> does not add the delimiter packet to the packets of the event signals, and the readout control unit <NUM> of the control unit <NUM> forcibly reads out the packets <NUM> of the event signals stored in the buffer memory <NUM> in accordance with predetermined conditions, so that the processing delay of the event signals can be prevented. In other respects, the configuration of the present embodiment is the same as that of the above-described first embodiment, and thus, the duplicated and detailed description thereof is omitted.

<FIG> is a flowchart illustrating an example of a process of the readout control unit in the third embodiment of the present invention. In the illustrated example, the readout control unit <NUM> stores the packets of the event signals received from the vision sensor <NUM> in the buffer memory <NUM> (Step S301). The readout control unit <NUM> monitors the packets of the event signals stored in the buffer memory <NUM>, and in a case in which the identification information (for example, the position of the pixel) of the sensor indicated by the event signal circulates (Step S302), the packets <NUM> are read out from the buffer memory <NUM> (Step S303). Even in a case in which the capacity of the buffer memory <NUM> becomes full (Step S304), the readout control unit <NUM> reads out the packets <NUM> from the buffer memory <NUM> (Step S303). The readout control unit <NUM> repeats these processes (Step S305).

In the above example, the readout control unit <NUM> forcibly reads out the packets <NUM> from the buffer memory <NUM> on condition that the identification information of the sensor indicated by the event signal circulates. Specifically, for example, in a case in which the identification information is a coordinate (x, y) indicating the position of the pixel corresponding to the sensor, the readout control unit <NUM> may determine that the identification information circulates on condition that x<NUM> < x<NUM> or x<NUM> = x<NUM> and y<NUM> < y<NUM> between a coordinate (x<NUM>, y<NUM>) associated with a time stamp t<NUM> and a coordinate (x<NUM>, y<NUM>) associated with a time stamp t<NUM> (t<NUM> > t<NUM>). Alternatively, the readout control unit <NUM> may determine that the identification information circulates in a case in which an event signal indicating the same identification information associated with a different time stamp is received.

Even in the present embodiment, the processing delay of the event signals in the control unit <NUM> can be suppressed while securing the capacity of the buffer memory <NUM> as similar to the above-described first embodiment and second embodiment. In addition, the present embodiment can be also applied to, for example, a case in which it is not easy to implement the function of adding the delimiter packet to the event signals in the processing circuit <NUM>.

Claim 1:
Electronic equipment comprising:
an event-driven vision sensor (<NUM>) including a sensor array (<NUM>) including sensors (110A, 110B) generating event signals when a change in intensity of incident light is detected and a processing circuit (<NUM>) configured to generate one or more packets including packets corresponding to the event signals and characterised in that the processing circuit adds a delimiter packet to the packets corresponding to the event signals; and
a signal processing device (<NUM>) comprising:
a communication unit (<NUM>) which receives packets transmitted from the event-driven vision sensor;
a buffer memory (<NUM>) in which the packets are temporarily stored; and
a readout control unit (<NUM>) which forcibly reads out the packets from the buffer memory if the delimiter packet is received by the communication unit or stored in the buffer memory.