Patent Description:
An event-driven type vision sensor is known in which pixels that detect an intensity change of incident light generate signals time-asynchronously. The event-driven type vision sensor is advantageous in that it can operate at a high speed with low power consumption in comparison with a frame type vision sensor in which all pixels are scanned in every predetermined cycle, particularly with CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor) image sensors and so forth. A technology related to such an event-driven type vision sensor as just described is disclosed, for example, in PTL <NUM> and PTL <NUM>. Previously proposed arrangements are disclosed by PTL <NUM>, PTL <NUM>, NPTL <NUM>, NPTL <NUM> and NPTL <NUM>.

However, regarding the event-driven type vision sensor, although such an advantage as described above is known, sufficient proposals have not been made for a utilization method of using the event-driven type vision sensor in combination with another device, for example, a frame type vision sensor.

Therefore, it is an object of the present invention to provide an image processing apparatus, an imaging apparatus, a system, an image processing method, and a program that can achieve advantageous effects by using an event-driven type vision sensor in combination with a frame type vision sensor.

In the following, an embodiment of the present invention is described with reference to the accompanying drawings. It is to be noted that, in the present specification and the drawings, components having substantially same functional configurations are denoted by like reference signs and overlapping description of them is omitted herein.

<FIG> is a block diagram depicting a general configuration of a system according to an embodiment of the present invention. As depicted in <FIG>, a system <NUM> includes an RGB (Red-Green-Blue) camera <NUM>, an EDS (Event Driven Sensor) <NUM>, and an image processing apparatus <NUM>.

The RGB camera <NUM> is an example of a first vision sensor that captures an image by synchronous scanning and includes an image sensor <NUM> and a processing circuit <NUM> connected to the image sensor <NUM>. The image sensor <NUM> captures an RGB image <NUM> by synchronously scanning all pixels, for example, in a predetermined cycle or at a predetermined timing according to a user operation. The processing circuit <NUM> converts the RGB image <NUM> into a format, for example, suitable for storage or transmission. Further, the processing circuit <NUM> provides a timestamp <NUM> to the RGB image <NUM>. For example, the processing circuit <NUM> provides a timestamp <NUM> indicative of at least any of start or end of scanning by the image sensor <NUM> to the RGB image <NUM>. For example, in the case of a still image, the period of time from the start to the end of scanning is referred to also as exposure period of time.

The EDS <NUM> is an example of a second vision sensor that generates an event signal when a sensor detects an intensity change of light and includes sensors <NUM> configuring a sensor array and a processing circuit <NUM> connected to the sensors <NUM>. Each of the sensors <NUM> includes a light reception element and generates an event signal <NUM> when it detects an intensity change of light incident thereto, more particularly, a luminance change. Since any sensor <NUM> that does not detect an intensity change of incident light does not generate the event signal <NUM>, in the EDS <NUM>, the event signal <NUM> is generated time-asynchronously. The event signal <NUM> outputted through the processing circuit <NUM> includes identification information of the sensor <NUM> (for example, the position of a pixel), a polarity (increase or decrease) of a luminance change, and a timestamp <NUM>.

Here, in the present embodiment, the timestamp <NUM> provided to the RGB image <NUM> and the timestamp <NUM> provided to the event signal <NUM> are in synchronism with each other. In particular, the timestamp <NUM> can be synchronized with the timestamp <NUM>, for example, by providing time information, which is used to generate the timestamp <NUM> in the EDS <NUM>, to the RGB camera <NUM>. Alternatively, in a case where the time information for generating the timestamps <NUM> and <NUM> is independent between the RGB camera <NUM> and the EDS <NUM>, by calculating an offset amount between the timestamps with reference to time at which a specific event (for example, a change of an imaging target over an overall image) occurs, the timestamp <NUM> and the timestamp <NUM> can be synchronized with each other ex post facto.

Further, in the present embodiment, by a calibration procedure for the RGB camera <NUM> and the EDS <NUM> executed in advance, each sensor <NUM> of the EDS <NUM> is associated with one or a plurality of pixels of the RGB image <NUM>, and the event signal <NUM> is generated corresponding to an intensity change of light at one or a plurality of pixels of the RGB image <NUM>.

<FIG> is a view schematically illustrating calibration between a camera and a sensor in the embodiment of the present invention. In the example depicted, a calibration pattern <NUM> common to the RGB camera <NUM> and the EDS <NUM> is imaged (in the case of the EDS <NUM>, by causing the overall region of the calibration pattern <NUM> to flicker, for example, using a light source <NUM>, the calibration pattern can be imaged), and corresponding parameters between the camera and the sensor are calculated from internal parameters and external parameters of the RGB camera <NUM> and the EDS <NUM>. By this, each sensor <NUM> can be associated with one or a plurality of pixels of the RGB image <NUM>. For example, an imaging apparatus that includes a combination of the RGB camera <NUM> and the EDS <NUM> for which such synchronization of timestamps and a calibration procedure as described above are carried out in advance may be provided.

<FIG> is a view illustrating an example of matching between an image and an event signal in the embodiment of the present invention. In the example depicted, an RGB image <NUM> captured by the RGB camera <NUM> and an event signal <NUM> outputted from the EDS <NUM> at time corresponding to scanning of the RGB image <NUM> are depicted as arranged at positions of pixels. By calculating such corresponding parameters between the camera and the sensor as described hereinabove with reference to <FIG> in advance, it is possible to associate the event signal <NUM> with one or a plurality of pixels of the RGB image <NUM> as depicted in <FIG>, in short, to superpose the event signal <NUM> on the RGB image <NUM>.

Referring again to <FIG>, the image processing apparatus <NUM> includes functions of a movement estimation unit <NUM>, an inverse filter generation unit <NUM>, and a filter application unit <NUM> that are incorporated by a computer including, for example, a communication interface, a processor, and a memory and are implemented by operation of the processor in accordance with a program stored in the memory or received through the communication interface. In the following, the functions of the components are described further.

The movement estimation unit <NUM> estimates a movement of an imaging target in the RGB image <NUM> on the basis of the timestamp <NUM> provided to the RGB image <NUM>, the event signal <NUM>, and the timestamp <NUM> provided to the event signal <NUM>. In a case where the imaging target of the RGB image <NUM> moves, an intensity change of light, particularly a luminance change, which appears at an edge portion of the imaging target, is detected from the event signal <NUM>. In other words, in the present embodiment, even if the movement estimation unit <NUM> does not refer to the RGB image <NUM> itself, it can estimate a movement of the imaging target in the RGB image <NUM> on the basis of the event signal <NUM>. The movement estimation unit <NUM> can estimate, from a position change in time series of pixels with regard to which occurrence of a luminance change is indicated by the event signal <NUM>, a movement region in which the movement of the imaging target occurs in the RGB image <NUM> and a movement vector that indicates the movement of the imaging target in the RGB image <NUM>.

In particular, for example, the movement estimation unit <NUM> estimates a movement on the basis of the event signals <NUM> having timestamps <NUM> included within a period of time from the start to the end of scanning for capturing the RGB image <NUM>. Here, the period of time from the start to the end of scanning is specified, for example, from two timestamps provided to the RGB image <NUM>. Alternatively, even in a case where only a timestamp indicative of any of the start or the end of scanning is provided to the RGB image <NUM>, if the duration of the scanning is known already, then the period of time from the start to the end of the scanning can be specified. As hereinafter described, for example, by applying an inverse filter generated on the basis of a movement of an imaging target occurring during a period of time from the start to the end of scanning, the influence of blur appearing in the RGB image <NUM> by a movement of an imaging target can be reduced.

The inverse filter generation unit <NUM> generates an inverse filter <NUM> on the basis of a movement of an imaging target in the RGB image <NUM> estimated by the movement estimation unit <NUM>. Here, the inverse filter is a filter that intends to bring the RGB image <NUM> closer to an original image of the imaging target by causing a change (filter) inverse to a change (filter) from an original picture of the imaging target caused in the RGB image <NUM> by a movement of the imaging target. The filter application unit <NUM> applies the inverse filter <NUM> to the RGB image <NUM> to obtain an output image <NUM>. As hereinafter described, the filter application unit <NUM> may apply a filter for compensating for a change caused in the RGB image <NUM> by application of the inverse filter <NUM> (for example, for filling up a blank region by enlarging the background) separately to the RGB image <NUM>.

<FIG> are views illustrating estimation of a movement and an inverse filter based on an event signal in the embodiment of the present invention. In the examples depicted, an imaging target obj is captured in the RGB image <NUM>. Since the imaging target obj moves during scanning by the image sensor <NUM>, blurring occurs in the RGB image <NUM> and the picture of the imaging target obj is elongated as depicted in <FIG>. On the other hand, as depicted in <FIG>, the event signal <NUM> indicates that an event has occurred with a pixel group P<NUM> immediately after the start of scanning of the RGB image <NUM>, and the pixel group P<NUM> gradually moves (pixel groups P<NUM>,. , Pn-<NUM>) until an event occurs with the pixel group Pn immediately before the end of the scanning.

In this case, the movement estimation unit <NUM> estimates a movement region R and a movement vector V in the RGB image <NUM> as depicted in <FIG>. For example, the movement region R is a region that includes the pixel groups P<NUM>, P<NUM>,. , Pn-<NUM>, Pn at which events have occurred during a period of time from the start to the end of the scanning (the region corresponds to the picture of the imaging target obj elongated as a result of occurrence of blur), and the movement vector V is a vector for moving each pixel of the pixel group P<NUM> corresponding to the start point of the movement to each pixel of the pixel group Pn corresponding to the end point of the movement. Since the sensor <NUM> for generating an event signal <NUM> is associated with one or a plurality of pixels of the RGB image <NUM> as described hereinabove, the movement estimation unit <NUM> need not refer to the RGB image <NUM> itself in order to estimate the movement region R and the movement vector V.

In a case where such a movement region R and a movement vector V as described above are estimated, the inverse filter generation unit <NUM> applies them to the RGB image <NUM> to generate an inverse filter <NUM> from which such an output image <NUM> as depicted in <FIG> is to be obtained. In the example depicted, the inverse filter <NUM> acts in a limited way on pixels in the movement region R to cancel the movement vector V. In particular, the inverse filter <NUM> moves the pixels in the movement region R by a vector -kV obtained by multiplying an inverse vector -V of the movement vector V by a coefficient k. The coefficient k gradually increases such that, for example, it is <NUM> at the start point (pixel group P<NUM>) of the movement vector V and becomes <NUM> at the end point (pixel group Pn). In the output image <NUM>, the enlarged picture of the imaging target obj is compressed to its original size and the influence of the blur caused by the movement of the imaging target obj is reduced.

In such an embodiment of the present invention as described above, the movement estimation unit <NUM> of the image processing apparatus <NUM> estimates a movement of an imaging target in the RGB image <NUM> from the event signal <NUM>. Since the event signal <NUM> is generated only in a case where an intensity change of light is detected at one or a plurality of pixels of the RGB image <NUM>, the processing can be speeded up in comparison with that in an alternative case in which, for example, pixels of a plurality of RGB images <NUM> consecutive in time are compared with each other to estimate a movement. Further, since the inverse filter <NUM> generated by the inverse filter generation unit <NUM> acts in a limited way on the movement region R of the RGB image <NUM>, occurrence of artifact can be suppressed, for example, in comparison with that in an alternative case in which a filter is applied to the overall RGB image <NUM> including a region other than the movement region R.

It is to be noted that the system <NUM> described in connection with the example described above may be incorporated in a single apparatus or may be distributed and implemented in a plurality of devices. For example, the RGB image <NUM> acquired by the RGB camera <NUM> and the event signal <NUM> acquired by the EDS <NUM> may be stored into a memory together with the timestamps <NUM> and <NUM>, and as post processing, estimation of a movement by the image processing apparatus <NUM>, generation of an inverse filter <NUM>, and application of the inverse filter <NUM> may be executed. Alternatively, when the RGB image <NUM> and the event signal <NUM> are acquired, the processes up to the generation of an inverse filter <NUM> by the image processing apparatus <NUM> may be executed, and the inverse filter <NUM> may be stored together with the RGB image <NUM>. In this case, when the RGB image <NUM> is displayed, the inverse filter <NUM> may be applied to the RGB image <NUM>, for example, in accordance with an operation of a user to generate an output image <NUM>.

Further, although, in the example depicted in <FIG>, an inverse filter <NUM> for compressing an elongated picture of an imaging target obj as a result of occurrence of blur to the start point side of the movement vector V is generated, in another example, an inverse filter <NUM> that compresses a picture of the imaging target obj to the end side or to an intermediate point of the movement vector V may be generated. Further, the inverse filter <NUM> to be generated on the basis of the event signal <NUM> is not limited to the example described above, and, for example, various filters each known as filter for reducing the influence of blur occurring by a movement of an imaging target can be used. Also in those cases, for example, if the movement estimation unit <NUM> estimates the movement region R of the system <NUM> and the inverse filter generation unit <NUM> generates an inverse filter <NUM> that acts in a limited way upon the movement region R, occurrence of artifact can be suppressed, for example, in comparison with that in an alternative case in which a filter is applied to the overall RGB image <NUM>.

<FIG> is a flow chart depicting an example of an image processing method according to the embodiment of the present invention. In the example depicted, the RGB camera <NUM> captures an RGB image <NUM> (step S101), and the EDS <NUM> simultaneously generates an event signal <NUM> (step S102). It is to be noted that the step S102 at which the event signal <NUM> is generated is executed only in a case where the sensor <NUM> corresponding to one or a plurality of pixels of the RGB image <NUM> detects an intensity change of light. A timestamp <NUM> is provided to the RGB image <NUM> (step S103), and to the event signal, a timestamp <NUM> is provided (step S104).

Then, processing by the image processing apparatus <NUM> is executed. First, the movement estimation unit <NUM> estimates a movement of an imaging target in the RGB image <NUM> on the basis of the timestamp <NUM> of the RGB image <NUM>, the event signal <NUM>, and the timestamp <NUM> of the event signal <NUM> (step S105). Then, the inverse filter generation unit <NUM> generates an inverse filter <NUM> on the basis of the estimated movement (step S106), and the filter application unit <NUM> applies the inverse filter <NUM> to the RGB image <NUM> (step S107). By such processes as described above, for example, an output image <NUM> can be obtained in which the influence of blur appearing in the RGB image <NUM> by a movement of the imaging target is reduced.

Claim 1:
An image processing apparatus (<NUM>) comprising:
a movement estimation unit (<NUM>) that estimates, on a basis of a first timestamp (<NUM>) provided to an image (<NUM>) captured by synchronous scanning, an event signal (<NUM>) generated corresponding to an intensity change of light at one or a plurality of pixels of the image, and a second timestamp (<NUM>) that is provided to the event signal and is in synchronism with the first timestamp, a movement of an imaging target in the image;
an inverse filter generation unit (<NUM>) that generates an inverse filter (<NUM>) on a basis of the movement; and
a filter application unit (<NUM>) that applies the inverse filter to the image, wherein
the movement estimation unit estimates a movement region in which the movement of the imaging target has occurred in the image, and
the inverse filter generation unit generates the inverse filter that acts in a limited way on the movement region of the image but not on the overall image.