Patent ID: 12262101

DETAILED DESCRIPTION

Overview of Embodiment 1

1. An embodiment disclosed in the present specification relates to a gating camera. The gating camera is structured to divide the depth direction into multiple ranges, and to generate multiple slice images that correspond to the multiple ranges. The gating camera includes: an illumination apparatus structured to emit probe light; a first image sensor; a second image sensor; and a controller structured to control a timing of emission of the probe light by the illumination apparatus and a timing of image capture by the first image sensor and the second image sensor. The controller controls the first image sensor and the second image sensor such that they receive reflected light from different respective ranges corresponding to one emission of the probe light from the illumination apparatus.

With a conventional gating camera provided with a single camera, when a given single range is set to a target, the light reflected by an object in a different range is not used. Accordingly, such an arrangement has a problem of poor efficiency of use of the probe light. In contrast, the gating camera according to the embodiment is capable of acquiring multiple slice images that correspond to multiple ranges in one emission of the probe light. This provides improved probe light use efficiency.

Also, the first image sensor may be assigned to a relatively near-distance range. Also, the second image sensor may be assigned to a relatively far-distance range. Also, the first image sensor may be structured to have an angle of view that is larger than that of the second image sensor. This enables a smaller difference in the size of an object image between the slice image in a far-distance range and the slice image in a near-distance range.

Description will be made below with reference to the drawings regarding an embodiment 1.

FIG.1is a block diagram showing an object identification system10according to the embodiment 1. The object identification system10is mounted on a vehicle such as an automobile, motorcycle, or the like. The object identification system10judges the kind (category) of an object OBJ that exists in the vicinity of the vehicle.

The object identification system mainly includes a gating camera20and a processing device40. The gating camera20includes an illumination apparatus22, a first image sensor24A, a second image sensor24B, and a controller26. The gating camera20captures images for a plurality of N (N≥2) ranges RNG1through RNGNdivided in the depth direction. The ranges may be designed such that adjacent ranges overlap at their boundaries in the depth direction.

The illumination apparatus22emits probe light L1in front of the vehicle in synchronization with a light emission timing signal S1supplied from the controller26. As the probe light L1, infrared light is preferably employed. However, the present invention is not restricted to such an arrangement. Also, as the probe light L1, visible light having a predetermined wavelength or ultraviolet light may be employed.

The first image sensor24A and the second image sensor24B are configured to support exposure control in synchronization with image timing signals S2A and S2B supplied from the controller26, and to be capable of generating slice images IMGA and IMGB. The first image sensor24A and the second image sensor24B are sensitive to the same wavelength as that of the probe light L1. The first image sensor24A and the second image sensor24B capture images of reflected light (returned light) L2reflected by the object OBJ.

The controller26controls the emission of the probe light L1by the illumination apparatus22and timings at which the first image sensor24A and the second image sensor24B each capture an image.

FIG.2is a diagram for explaining the basic operation of the gating camera20. For ease of understanding, description will be made directing attention to the operation of one (24A) from among the first image sensor24A and the second image sensor24B.FIG.2shows the operation when the i-th range RNGiis measured. The illumination apparatus22emits light during a light-emitting period τ1from the time point t0to t1in synchronization with the light emission timing signal S1. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance. The distance between the gating camera20and the near-distance boundary of the range RNGiis represented by dMINi. The distance between the gating camera20and the far-distance boundary of the range RNGiis represented by dMAXi.

The round-trip time TMINi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMINi, up to the return of the reflected light to the first image sensor24A, is represented by TMINi=2×dMINi/c. Here, c represents the speed of light.

Similarly, the round-trip time TMAXi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMAXi, up to the return of the reflected light to the first image sensor24A, is represented by TMAXi=2×dMAXi/c.

When only an image of an object OBJ included in the range RNGiis to be captured, the controller26generates the image capture timing signal S2A so as to start the exposure at the time point t2=t0+TMINi, and so as to end the exposure at the time point t3=t1+TMAXi. This is a single exposure operation.

When an image is captured for the i-th range RNGi, the exposure may be executed multiple times. In this case, preferably, the controller26may repeatedly execute the above-described exposure operation multiple times with a predetermined period τ2.

FIGS.3A and3Bare diagrams for explaining a slice image generated by the gating camera20including a single camera.FIG.3Ashows an example in which an object (pedestrian) OBJ2exists in the range RNG2, and an object (vehicle) OBJ3exists in the range RNG3.FIG.3Bshows multiple slice images IMG1through IMG3acquired in the situation shown inFIG.3A. When the slice image IMG1is captured, the image sensor is exposed by only the reflected light from the range RNG1. Accordingly, the image IMG1includes no object image.

When the slice image IMG2is captured, the image sensor is exposed by only the reflected light from the range RNG2. Accordingly, the slice image IMG2includes only the object OBJ2. Similarly, when the slice image IMG3is captured, the image sensor is exposed by only the reflected light from the range RNG3. Accordingly, the slice image IMG3includes only the object OBJ3. As described above, with the gating camera20, this arrangement is capable of capturing object images in the form of separate images for the respective ranges.

Returning toFIG.1, in the present embodiment, the first image sensor24A and the second image sensor24B are assigned to different ranges. More specifically, the controller26controls the first image sensor24A and the second image sensor24B such that they receive the reflected light from the respective different ranges in response to one emission of the probe light L1by the illumination apparatus22.

The gating camera20generates multiple slice images IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN. As the i-th slice image IMGi, only an image of an object included in the corresponding range RNGiis acquired.

The processing device40is configured to identify the kind of an object based on multiple slice images IMG1through IMGNthat correspond to the multiple range RNG1through RNGNgenerated by the gating camera20. The processing device40is provided with a classifier42implemented based on a prediction model generated by machine learning. The algorithm employed by the classifier42is not restricted in particular. Examples of the algorithms that can be employed include You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), Region-based Convolutional Neural Network (R-CNN), Spatial Pyramid Pooling (SPPnet), Faster R-CNN, Deconvolution-SSD (DSSD), Mask R-CNN, etc. Also, other algorithms that will be developed in the future may be employed.

The processing device40may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, or the like, and a software program to be executed by the processor (hardware component). Also, the processing device40may be configured as a combination of multiple processors. Alternatively, the processing device40may be configured as a hardware component alone.

The above is the configuration of the object identification system10including the gating camera20. Next, description will be made regarding the operation of the gating camera20.FIG.4is a diagram for explaining the operation of the gating camera20shown inFIG.1.

The first image sensor24A is assigned to the i-th range RNGithat is arranged at a relatively near distance. The second image sensor24B is assigned to the j-th (j>i) range RNGjarranged at a relatively far distance. The illumination apparatus22emits light during a light emission period τ1from the time point to t0the time point t1in synchronization with the light emission timing signal S1. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance d. The distance between the gating camera20and the near-distance boundary of the range RNGiis represented by dMINi. Similarly, the distance between the gating camera20and the far-distance boundary of the range RNGiis represented by dMAXi.

In order to acquire an image of the object OBJiincluded in the range RNGi, the image capture timing signal S2A is generated so as to start the exposure of the first image sensor24A at the time point t2=t0+TMINi, and so as to end the exposure of the first image sensor24A at the time point t3=t1+TMAXi.

Furthermore, in order to acquire an image of the object OBJjincluded in the range RNGjusing the same probe light, the image capture timing signal S2B is generated so as to start the exposure of the second image sensor24B at the time point t4=t0+TMINj, and so as to end the exposure of the second sensor24B at the time point t5=t1+TMAXj.
TMINj=2×dMINj/c
TMAXj=2×dMAXj/c

FIGS.5A and5Bare diagrams for explaining slice images generated by the gating camera20.FIG.5Ashows an example in which an object (pedestrian) OBJiexists in the range RNGi, and an object (vehicle) OBJjexists in the range RNGj.FIG.5Bshows two slice images IMGiand IMGjgenerated by the first image sensor24A and the second image sensor24B in a situation shown inFIG.5A. It should be noted that the first image sensor24A and the second image sensor24B have the same angle of view. Accordingly, an image of the object OBJjacquired in the far-distance range has a small size.

The above is the operation of the gating camera20. Next, description will be made regarding the advantages thereof. With a conventional gating camera provided with a single camera, when a given range is an image capture target, the reflected light from an object in a different range is not used. That is to say, it can be said that such an arrangement provides low probe light use efficiency. In contrast, with the gating camera20according to the present embodiment, multiple slice images that correspond to the multiple ranges can be acquired in one emission of the probe light L1. That is to say, such an arrangement provides improved probe light use efficiency.

Next, description will be made regarding the assignment of multiple ranges to the first image sensor24A and the second image sensor24B. Description will be made based on several examples.

Example 1-1

For simplification of description, description will be made assuming that all the ranges RNGi(i=1 to N) have the same length (dMAXi−dMINi) in the depth direction, and generation of the slice image for each range RNGirequires the same number of exposures.

FIG.6Ashows the correspondence relation between the ranges and the camera according to Example 1.1. In the drawing, “A” indicates the first image sensor, and “B” indicates the second image sensor. In this example, the depth direction is divided into N(=6) ranges. However, the present invention is not restricted to such an example. In Example 1-1, the odd-numbered ranges RNG1, RNG3, and RNG5are assigned to the first image sensor24A. The even-numbered ranges RNG2, RNG4, and RNG6are assigned to the second image sensor24B.

FIG.6Bis a timing chart that corresponds to the assignment shown inFIG.6A. A set of K emissions by the illumination apparatus22required to generate a single slice image and the accompanying K exposures by the first image sensor24A and the second image sensor24B will be referred to as “one slot”.FIG.6Bshows an example in which one slot includes one emission and one exposure. Also, one slot may include multiple light emissions and multiple exposures. In a case of employing a single camera, image capture of N=6 ranges requires six slots. However, in a case of employing two cameras, image capture of N=6 ranges requires only three slots.

The first image sensor24A captures images of the three ranges RNG1, RNG3, and RNG5assigned to the first image sensor24A itself in a time sharing manner. In the same manner, the second image sensor24B captures images of the three ranges RNG2, RNG4, and RNG6assigned to the second image sensor24B itself in a time sharing manner.

Example 1-2

FIG.7Ais a diagram showing the correspondence relation between the ranges and the camera according to Example 1-2. In the drawing, “A” indicates the first image sensor, and “B” indicates the second image sensor. The first image sensor24A, which is one from among the two cameras24A and24B, is assigned to multiple ranges arranged at a relatively near distance. On the other hand, the other camera, i.e., the second image sensor24B is assigned to the ranges arranged at a relatively far distance. Specifically, the ranges RNG1through RNG3are assigned to the first image sensor24A. The ranges RNG4through RNG6are assigned to the second image sensor24B.

FIG.7Bis a timing chart that corresponds to the assignment shown inFIG.7A. Also in this example, one slot includes one emission and one exposure. However, the present invention is not restricted to such an example. Also, one slot may include multiple light emissions and multiple exposures. The image capture of N=6 ranges requires three slots to be completed. The first image sensor24A captures images of the three ranges RNG1through RNG3assigned to the first image sensor24A itself in a time sharing manner. In the same manner, the second image sensor24B captures images of the three ranges RNG4through RNG6assigned to the second image sensor24B itself in a time sharing manner.

The method according to Example 1-2 has the following advantage as compared with the method according to Example 1-1. That is to say, with the control according to Example 1-2, in a case in which the output image of the first image sensor24A and the output image of the second image sensor24B are processed in parallel by means of two classifiers, the two classifiers can be implemented by an algorithm for near distances and an algorithm for far distances.

Preferably, in Example 1-2, the first image sensor24A and the second image sensor24B may be configured to have different angles of view.FIG.8is a diagram showing the gating camera20according to Example 1-2. The first image sensor24A assigned to the near ranges is configured to be capable of capturing an image with a relatively wide field of view. The second image sensor24B assigned to the far ranges is configured to be capable of capturing an image with a relatively narrow field of view.

For example, the first image sensor24A and the second image sensor24B may be configured with the same size. In this case, the first image sensor24A may be combined with a wide-angle lens, and the second image sensor24B may be combined with a telephoto lens.

Also, the first image sensor24A and the second image sensor24B may each be provided with a lens having the same focal distance. In this case, the first image sensor24A may be configured to have a relatively large image sensor size, and the second image sensor24B may be configured to have a relatively small image sensor size, such that they have different fields of view. Also, the first image sensor24A and the second image sensor24B may be configured such that they have different sensor sizes and different focal distances. In the drawing, the range in which an image is captured by the first image sensor24A and the range in which an image is captured by the second image sensor24B are hatched with different patterns. The above is the configuration of the gating camera20according to Example 1-2.

FIGS.9A and9Bare diagrams for explaining the operation of the gating camera20shown inFIG.8.

For comparison, description will be made with reference toFIG.5. In a case in which the images of the near-distance range RNGiand the far-distance range RNGjare captured by means of a camera having the same field of view, as shown inFIG.5B, the object OBJjimage in the far-distance range RNGjhas a relatively small size.

Returning toFIG.9, with Example 1-2 in which images of the near-distance ranges RNGi(i=1 to 3) and the far-distance ranges RNGj(j=4 to 6) are captured by means of cameras having different angles of view, this allows the object OBJjimage in the far-distance range RNGjto be captured with a large size. This provides the processing device40provided in the next stage of the gating camera20with an improved object identification rate.

Example 1-3

As a result of investigating conventional gating cameras, the present inventors have further recognized the following additional problems.

With the gating camera, the image sensor captures an image of the reflected light reflected by an object included in an image capture target range after the light emission device emits the probe light. That is to say, as the distance to the target range becomes larger, the intensity of the reflected light received by the image sensor becomes smaller.

With typical cameras, such a camera is able to compensate for a decrease in the intensity of the received light by increasing the exposure time. However, with a gating camera, the exposure time is determined according to the distance from the gating camera to the near-distance boundary of the range for which an image is to be captured and the distance from the gating camera to the far-distance boundary of the range for which an image is to be captured. Accordingly, the exposure time cannot be increased.

Accordingly, in order to provide a high-quality image of a far-distance range, a countermeasure is required. Examples of such a countermeasure include: (i) raising the intensity of the probe light to be emitted; and (ii) raising the image sensor sensitivity.

In some cases, such a countermeasure cannot be supported. Alternatively, in many cases, only an insufficient countermeasure can be supported. In this case, a set of light emission and exposure (i.e., the slot described above) is required to be executed a multiple of Kj(Kj≥2) times for the far-distance range RNGj. Furthermore, the Kjslice images obtained as a result of the multiple slots are combined (or multiple exposures are provided) so as to generate a single image. That is to say, as the distance to the range becomes larger, the number of slots required for generating a single slice image becomes larger.

For ease of understanding, description will be made regarding an example in which K1=K2=1, K3=K4=2, K5=K6=3. In this case, 12 slots are required in order to generate the slice images IMG1through IMG6of the six ranges RNG1through RNG6. With the time required for one slot as Ts, a conventional gating camera including a single camera requires a time of 12×Ts to process the 12 slots for each frame.

In Example 1-3, a technique is provided for reducing the time required for each frame.

FIGS.10A through10Care diagrams showing the range assignment according to Example 1-3. In Example 1-3, a total number of required slots are assigned fifty/fifty to the first image sensor24A and the second image sensor24B such that they are processed synchronously in parallel. In the same manner as in Example 1-2, the near-distance ranges RNG1through RNG4are assigned to the first image sensor24A, and the far-distance ranges RNG5and RNG6are assigned to the second image sensor24B.

In the same manner as in Example 1-1,FIG.10Bshows an example in which the slots of the odd-numbered ranges RNG1, RNG3, and RNG5are assigned to the first image sensor24A, and the slots of the even-numbered ranges RNG2, RNG4, and RNG6are assigned to the second image sensor24B.

FIG.10Cshows an example in which two cameras are assigned to different slots for the same ranges.

With Example 1-3, the processing for 12 slots is assigned to the two cameras24A and24B such that they are processed in parallel. This allows the processing time for each frame to be reduced to (6×Ts), thereby raising the frame rate.

Description has been made above regarding Example 1. Next, description will be made regarding modifications relating to Example 1.

Modification 1-1

The assignment of the cameras for the multiple ranges or multiple slots has been described for exemplary purposes only. It can be readily conceived by those skilled in this art that various modifications may be made, which are also encompassed in the technical scope of the present invention.

Modification 1-2

Description has been made in the embodiment regarding the gating camera20provided with two cameras24A and24B. Also, the number of the cameras may be three or more.

Modification 1-3

Description has been made in the embodiment regarding an arrangement in which the slice images of different ranges are processed by the same classifier42. However, the present invention is not restricted to such an arrangement. Also, different classifiers may be employed for the respective ranges.

Usage

FIGS.11A and11Bare diagrams showing an automobile300provided with the gating camera20. Referring toFIG.11A, the automobile300includes headlamps (lamps)302L and302R. The automobile300shown inFIG.11Aincludes a single illumination apparatus22at a central position of the vehicle. Furthermore, the left and right headlamps302L and302R respectively include the first image sensor24A and the second image sensor24B as built-in components. The position of the illumination apparatus22is not restricted in particular. For example, the illumination apparatus22may be provided to a front bumper (i) or a front grille (ii). Also, the illumination apparatus22may be provided to the back side of a rear-view mirror on the inner side of the front window (iii). Also, the position of the controller26is not restricted in particular. The controller26may be provided in the engine compartment or the vehicle interior. Also, the controller26may be built into the headlamp.

Referring toFIG.11B, the single illumination apparatus22includes multiple (e.g., two) light sources22A and22B. The multiple light sources22A and22B emit light at the same timing such that the output light of the multiple light sources22A and22B forms one probe light. The multiple light sources22A and22B are built into the left and right headlamps302L and302R, respectively.

The first image sensor24A and the second image sensor24B may be built into one from among the headlamps302L and302R. Alternatively, the first image sensor24A and the second image sensor24B may each be arranged as an external component of the headlamps302L and302R. For example, the first image sensor24A and the second image sensor24B may be provided in the vicinity of the illumination apparatus22.

FIG.12is a block diagram showing an automotive lamp200provided with an object detection system210. The automotive lamp200forms a lamp system310together with an in-vehicle ECU304. The automotive lamp200includes a light source202, a lighting circuit204, and an optical system206. Furthermore, the automotive lamp200includes the object detection system210. The object detection system210corresponds to the object identification system10described above. The object detection system210includes the gating camera20and the processing device40.

Also, the information with respect to the object OBJ detected by the processing device40may be used to support the light distribution control operation of the automotive lamp200. Specifically, a lamp ECU208generates a suitable light distribution pattern based on the information with respect to the kind of the object OBJ and the position thereof generated by the processing device40. The lighting circuit204and the optical system206operate so as to provide the light distribution pattern generated by the lamp ECU208.

Also, the information with respect to the object OBJ detected by the processing device40may be transmitted to the in-vehicle ECU304. The in-vehicle ECU may support autonomous driving based on the information thus transmitted.

Embodiment 2

First, description will be made regarding the problem relating to the embodiment 2.

FIG.13is a diagram showing an example of a situation in which multiple objects exist. As viewed from the sensor2toward the side in the arrow direction, two objects OBJ1and OBJ2exist such that they overlap. Description will be made assuming that the two objects OBJ1and OBJ2have the same color, and the object OBJ2has an upper-half surface portion and a lower-half surface portion formed of different materials.

FIG.14Ashows a visible-light image generated by capturing an image of the situation shown inFIG.13by means of a visible-light camera. When the two objects have the same color or very similar colors, such a visible-light image leads to a blurred edge (boundary) between the two objects. Accordingly, in a case in which such a visible-light image is subjected to image processing, such an arrangement has the potential to provide false recognition of the two objects OBJ1and OBJ2as the same object.

FIG.14Bshows an infrared image generated by capturing an image of the situation shown inFIG.13by means of an infrared camera. With the infrared camera, a portion where the reflection ratio with respect to infrared light changes appears as an edge. Accordingly, as shown inFIG.14B, a clear edge is generated at a boundary of the object OBJ2between the upper-half portion and the lower-half portion formed of different materials. Accordingly, in a case in which such an infrared image is subjected to image processing, such an arrangement has the potential to provide false recognition of the single object OBJ2as two objects OBJ2Aand OBJ2B.

The present invention relating to the embodiment 2 has been made in view of such a situation. Accordingly, it is an exemplary purpose of an embodiment of the present invention to provide an object identification system that is capable of accurately separating multiple objects.

Overview of Embodiment 2

An embodiment disclosed in the present specification relates to an object identification system. The object identification system includes: a visible-light camera structured to generate a visible-light image; a gating camera structured to divide the depth direction into multiple ranges, and to generate multiple infrared images that correspond to the multiple ranges; and a processing device structured to process the visible-light image and the multiple infrared images. The processing device detects a separate object for each of the multiple infrared images, extracts an edge of the separate object, and applies the edge of the object to the visible-light image so as to separate multiple object images included in the visible-light image.

With such an arrangement configured to combine the visible-light image generated by the visible-light camera and the multiple infrared images generated by the gating camera, the object image can be separated giving consideration to depth information in addition to color information and reflection ratio information. This allows multiple objects to be separated with high precision. With this, for example, the objects can be distinguished even when objects having the same color overlap in the front-rear direction.

Also, the processing device may judge a single separate region or multiple consecutive regions as the separate object for each infrared image. This allows a single object having portions formed of different materials to be recognized as a single object.

Also, when precision of the object identification with the visible-light image alone is low, the gating camera may be set to an active state. With such an arrangement in which the operation of the gating camera is controlled to a minimum necessary level, such an arrangement suppresses an increase of an amount of calculation.

Also, the gating camera may be operated at a frame rate that is lower than that of the visible-light camera. This suppresses an increase of an amount of calculation.

Also, once an edge is extracted for a given object, the edge thus extracted may be continuously used, and edge re-extraction may be suspended. Once the object is separated, the edge thereof can be easily tracked. Accordingly, in this case, by suspending the operation of the gating camera, this allows an amount of calculation to be reduced.

Description will be made below regarding the embodiment 2 with reference to the drawings.

FIG.15is a block diagram showing an object identification system10according to an embodiment 2. The object identification system10is mounted on a vehicle such as an automobile, motorcycle, or the like. The object identification system10judges the kind (which will also be referred to as the “category” or “class”) of an object OBJ that exists in the vicinity of the vehicle.

The object identification system10mainly include a visible-light camera12, a gating camera20, and a processing device40. The visible-light camera12is configured as a color camera or a monochrome camera. The visible-light camera12generates a visible-light image IMG_VS.

The gating camera20includes an illumination apparatus22, an infrared camera24, and a controller26. The gating camera20captures images for a plurality of N (N≥2) ranges RNG1through RNGNdivided in the depth direction. The ranges may be designed such that adjacent ranges overlap at their boundaries in the depth direction.

The illumination apparatus22emits probe light L1in front of the vehicle in synchronization with a light emission timing signal S1supplied from the controller26. As the probe light L1, infrared light is preferably employed.

The infrared camera24is configured to support exposure control in synchronization with an image capture timing signal S2supplied from the controller26, and to be capable of generating an infrared image IMG_IR. The infrared camera24is sensitive to the same wavelength as that of the probe light L1. The infrared camera24captures an image of reflected light (returned light) L2reflected by the object OBJ.

The controller26controls the emission of the probe light L1by means of the illumination apparatus22and the timing of the image capture by means of the infrared camera24.

FIG.16is a diagram for explaining the basic operation of the gating camera20.FIG.16shows the operation when the i-th range RNGiis measured. The illumination apparatus22emits light during a light emission period τ1from the time points t0to t1in synchronization with the light emission timing signal S1. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance. The distance between the gating camera20and the near-distance boundary of the range RNGiis represented by dMINi. The distance between the gating camera20and the far-distance boundary of the range RNGiis represented by dMAXi.

The round-trip time TMINi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMINi, up to the return of the reflected light to the infrared camera24, is represented by TMINi=2×dMINi/c. Here, c represents the speed of light.

Similarly, the round-trip time TMAXi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMAXi, up to the return of the reflected light to the infrared camera24, is represented by TMAXi=2×dMAXi/c.

When only an image of an object OBJ included in the range RNGiis to be captured, the controller26generates the image capture timing signal S2A so as to start the exposure at the time point t2=t0+TMINi, and so as to end the exposure at the time point t3=t1+TMAXi. This is a single exposure operation.

When an image is captured for the i-th range RNGi, the exposure may be executed multiple times. In this case, preferably, the controller26may repeatedly execute the above-described exposure operation multiple times with a predetermined period τ2.

FIGS.17A and17Bare diagrams for explaining an image generated by the gating camera20.FIG.17Ashows an example in which an object (pedestrian) OBJ2exists in the range RNG2, and an object (vehicle) OBJ3exists in the range RNG3.FIG.17Bshows multiple infrared images IMG_IR1through IMG_IR3acquired in the situation shown inFIG.17A. When the slice image IMG_IR1is captured, the image sensor is exposed by only the reflected light from the range RNG1. Accordingly, the image IMG_IR1includes no object image.

When the infrared image IMG_IR2is captured, the image sensor is exposed by only the reflected light from the range RNG2. Accordingly, the infrared image IMG_IR2includes only the object OBJ2. Similarly, when the infrared image IMG_IR3is captured, the image sensor is exposed by only the reflected light from the range RNG3. Accordingly, the infrared image IMG_IR3includes only the object OBJ3. As described above, with the gating camera20, this arrangement is capable of capturing object images in the form of separate images for the respective ranges.

Returning toFIG.15, the processing device40is configured to receive a visible-light image IMG_VS and multiple infrared images IMG_IR1through IMG_IRN, and to identify the kind of an object in front of the vehicle based on the images thus received.

The processing device40may include the classifier42and a separation processing unit50. The processing device40may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, or the like, and a software program to be executed by the processor (hardware). Also, the processing device40may be configured as a combination f multiple processors. Alternatively, the processing device40may be configured as a hardware component only.

The classifier42may be implemented based on a prediction model generated by machine learning. The classifier42judges the kind (category or class) of an object included in an input image. The algorithm employed by the classifier42is not restricted in particular. Examples of algorithms that can be employed include You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), Region-based Convolutional Neural Network (R-CNN), Spatial Pyramid Pooling (SPPnet), Faster R-CNN, Deconvolution-SSD (DSSD), Mask R-CNN, etc. Also, other algorithms that will be developed in the future may be employed.

As an upstream stage of the classifier42, the separation processing unit50is provided. The separation processing unit50does not distinguish the kind of an object. The separation processing unit50isolates an object image IMG_OBJ that corresponds to the object OBJ from the visible-light image IMG_VS. The classifier42identifies the kind of each object image IMG_OBJ.

FIG.18is a function block diagram showing the processing device40according to an embodiment. In this example, the separation processing unit50includes a first processing unit52and a second processing unit54. The first processing unit52receives multiple infrared images IMG_IR1through IMG_IRN, and sequentially processes the infrared images thus received. The first processing unit52refers to each infrared image IMG_IRi, and judges that a single separate region is, or multiple consecutive regions are, the separate object OBJij. Specifically, a region surrounded by an edge will be referred to as the “separate object”. The object OBJijrepresents the j-th object included in the infrared image IMG_IRi. With this, the processing device40outputs edge data EDGEijthat indicates the edge (outer circumference) and the position of the separate object OBJij. When the i-th infrared image IMGiincludes M separate objects OBJi1through OBJiM, M items of edge data EDGEi1through EDGEiMare generated. A region surrounded by an edge will be referred to as a “region”. The edge data EDGEijhas no relation with the object color.

FIGS.19A and19Bare diagrams for explaining the judgment of a separate object based on a single infrared image IMG_IRi, i.e., for explaining the operation of the first processing unit52. The infrared image IMG_IRishown inFIG.19Aincludes five regions A through E. It should be noted that such an arrangement ensures that each single infrared image IMG_IRiis included in the same corresponding range RNGi.

The region A is a separate region. Accordingly, judgement is made that the region A is a single separate object OBJi1. In contrast, the regions B and C are continuous. Accordingly, judgment is made that the regions B and C are combined as a single separate object OBJi2. Similarly, the regions D and E are continuous. Accordingly, judgment is made that the regions D and E are combined as a single separate object OBJi3.

FIG.19Bis a diagram showing edge data. In this example, the three separate objects OBJi1through OBJi3are detected. Accordingly, three items of edge data EDGEi1through EDGEi3each indicating the corresponding edge and the position thereof are generated.

Returning toFIG.18, the second processing unit54receives the input of the edge data EDGE acquired for all the infrared images IMG_IR1through IMG_IRN. The second processing unit54applies the object edge data EDGE to the visible-light image IMG_VS, so as to separate a plurality of K object images IMG_OBJ1through IMG_OBJK.

The above is the configuration of the object identification system10. Next, description will be made regarding the operation thereof.

FIGS.20A through20Care diagrams for explaining the operation of the object identification system10. Description will be made below assuming that the situation shown inFIG.13is measured by the object identification system10. Also, description will be made assuming that, inFIG.13, the object OBJ1exists in the first range RNG1, and the object OBJ2exists in the range RNG2.

Description will be made with reference toFIGS.20A and20Bregarding the processing of the first processing unit52.FIG.20Ashows the infrared image IMG_IR1generated by the gating camera20. The infrared image IMG_IR1includes an image of the object OBJ1. The edge of the object OBJ1is detected by the first processing unit52of the separation processing unit50. Subsequently, the edge data EDGE11is generated.

FIG.20Bshows the infrared image IMG_IR2generated by the gating camera20. The infrared image IMG_IR2includes an image of the object OBJ2. The edge of the object OBJ2is detected by the first processing unit52of the separation processing unit50. Subsequently, the edge data EDGE12is generated. There is a difference in the material between the upper portion and the lower portion of the object OBJ2. However, the upper portion and the lower portion of the object OBJ2are recognized as a single separate object.

Referring toFIG.20C, description will be made regarding the processing of the second processing unit54. The left-side drawing inFIG.20Cshows the visible-light image IMG_VS generated by the visible-light camera12. As described above, the two objects OBJ1and OBJ2have the same color. Accordingly, the visible-light image IMG_VS provides only a blurred boundary between the objects OBJ1and OBJ2. As shown in the central drawing inFIG.20C, the edge data EDGE11and EDGE21are applied to the visible-light image IMG_VS. With this, as shown in the right-side drawing inFIG.20C, such an arrangement is capable of isolating two object images IMG_OBJ1and IMG_OBJ2. Preferably, the second processing unit54may sequentially apply the edge data from the near-distance range so as to sequentially isolate the object image IMG_OBJ from the near-distance side.

The above is the operation of the object identification system10. The object identification system10is capable of distinguishing multiple objects even if it is difficult for an arrangement employing only a visible-light camera to distinguish such multiple objects.

Furthermore, with the processing using a typical infrared camera, as shown inFIG.2B, such an arrangement has the potential to provide false recognition of a single object OBJ2having two portions formed of different materials as two objects OBJ2Aand OBJ2B. In contrast, with the present embodiment, this arrangement is capable of recognizing a single object even if such a single object has multiple portions formed of different materials.

Description will be made regarding the frame rate of the gating camera20. Description will be made regarding the gating camera20assuming that multiple infrared images IMG_IR1through IMG_IRNfor all of multiple ranges RNG1through RNGNare regarded as one frame.

1. The gating camera20may operate at the same frame rate as that of the visible-light camera12. In this case, the same processing is performed for all the frames, thereby providing the object identification with improved precision in a steady manner.

2. However, in a case in which the visible-light camera12and the gating camera20are operated at the same frame rate, such an arrangement leads to a problem of an increased amount of processing by the processing device40. In order to solve such a problem, the frame rate of the gating camera20may be designed to be lower than that of the visible-light camera12. With this, object identification based on only the visible-light image IMG_VS and object identification based on both the visible-light image IMG_VS and the infrared image IMG_IR may be performed in a time sharing manner.

In this case, the intermittently generated edge data EDGE may be displayed so as to track the position of a subject included in the visible-light image IMG_VS using a so-called tracking technique.

3. In the normal state, the object identification may be performed based on only the image generated by the visible-light camera12. When the identification precision decreases, the gating camera20may be set to the active state so as to start the object identification based on both the visible-light image IMG_VS and the infrared image IMG_IR. A decrease in the identification precision may be detected based on a decrease in the matching probability. Subsequently, after the identification rate improves, the gating camera20may be set to the non-active state so as to return to the object identification based on only the visible-light camera12.

Description will be made regarding modifications relating to the embodiment 2.

Modification 2-1

The configuration of the processing device40is not restricted to the example described above.FIG.21is a function block diagram showing a processing device40A according to a modification 2-1. In this modification, the separation processing unit50includes a first processing unit52. The functions of the processing unit52are the same as those of the processing unit52shown inFIG.18. Specifically, the processing unit52refers to each infrared image IMG_IRi, and judges that a single separate region is, or multiple consecutive regions are, a separate object OBJij. Subsequently, the first processing unit52outputs the edge data EDGEijthat indicates the edge (outer circumference) and the position of the separate object OBJij. The classifier42receives the input of a set of the edge data EDGE together with the visible-light image IMG_VS. The classifier42detects the object image IMG_OBJ included in the visible-light image IMG_VS using each edge data EDGE as a clue, and judges the kind of the object image IMG_OBJ.

For example, the edge data EDGE may be sequentially selected so as to use the edge data EDGE thus selected as a mask. With this, only an image within the edge may be set to a target to be subjected to the object identification.

Usage

FIGS.22A and22Bare diagrams showing an automobile300provided with the gating camera20. Referring toFIG.22A, the automobile300includes headlamps (lamps)302L and302R. The automobile300shown inFIG.22Aincludes a single illumination apparatus22at a central position of the vehicle. Furthermore, the left and right headlamps302L and302R respectively include the visible-light camera12and the infrared camera24as built-in components thereof. In a case in which parallax between the visible-light camera12and the infrared camera24is large, parallax correction processing may be implemented in the processing device40. The position of the illumination apparatus22is not restricted in particular. For example, the illumination apparatus22may be provided to a front bumper (i) or a front grille (ii). Also, the illumination apparatus22may be provided to the back side of a rear-view mirror on the inner side of the front window (iii). Also, the position of the controller26is not restricted in particular. The controller26may be provided in the engine compartment or the vehicle interior. Also, the controller26may be built into the headlamp.

Referring toFIG.22B, the single illumination apparatus22includes multiple (e.g., two) light sources22A and22B. The multiple light sources22A and22B emit light at the same timing such that the output light of the multiple light sources22A and22B forms one probe light. The multiple light sources22A and22B are built into the left and right headlamps302L and302R, respectively.

The visible-light camera12and the infrared camera24may be built into one from among the headlamps302L and302R. This arrangement does not require parallax correction between the visible-light camera12and the infrared camera24, thereby allowing the processing by the processing device40to be reduced.

The visible-light camera12and the infrared camera24may each be arranged as an external component of the headlamps302L and302R. For example, the visible-light camera12and the infrared camera24may be provided in the vicinity of the illumination apparatus22.

FIG.23is a block diagram showing an automotive lamp200provided with an object detection system210. The automotive lamp200forms a lamp system310together with an in-vehicle ECU304. The automotive lamp200includes a light source202, a lighting circuit204, and an optical system206. Furthermore, the automotive lamp200includes the object detection system210. The object detection system210corresponds to the object identification system10described above. The object detection system210includes the gating camera20and the processing device40.

Also, the information with respect to the object OBJ detected by the processing device40may be used to support the light distribution control operation of the automotive lamp200. Specifically, a lamp ECU208generates a suitable light distribution pattern based on the information with respect to the kind of the object OBJ and the position thereof generated by the processing device40. The lighting circuit204and the optical system206operate so as to provide the light distribution pattern generated by the lamp ECU208.

Also, the information with respect to the object OBJ detected by the processing device40may be transmitted to the in-vehicle ECU304. The in-vehicle ECU may support autonomous driving based on the information thus transmitted.

Embodiment 3

First, description will be made regarding a problem relating to an embodiment 3.

The object identification system employing visible light has a problem of a decrease in the identification rate at night. Accordingly, the present inventor investigated an arrangement in which an active sensor is mounted on an automobile so as to provide object identification using an image generated by the active sensor. The active sensor is configured to emit probe light such as infrared light or the like, and to capture an image of reflected light from an object by means of a camera.

As the light (infrared light) propagation distance becomes longer, attenuation of the light becomes larger. Accordingly, as the distance to an object becomes larger, the amount of light that reaches the object becomes smaller, and the amount of light reflected from the object becomes smaller. That is to say, in a case in which an image of the same object is captured, as the distance to the object becomes larger, the pixel values of an image of the object become lower, leading to the generation of a dark image.

The classifier used for the object identification is implemented based on a prediction model generated by performing machine learning using an image of an object to be identified as teacher data. Accordingly, in a case in which the prediction model is generated using an image of an object that exists at a predetermined distance as the teacher data, such an arrangement has the potential to have a problem of drastic reduction of the identification rate for an image captured when the same object exists at a different distance. In order to solve this problem, in a case in which an image of the same object is captured at various distances so as to generate the teacher data, this leads to an enormous learning cost.

The invention relating to the embodiment 3 has been made in view of such a situation. Accordingly, it is an exemplary purpose of an embodiment of the present invention to provide an object identification system employing an active sensor with an improved identification rate.

Overview of Embodiment 3

An embodiment disclosed in the present specification relates to an object identification system. The object identification system includes: a gating camera structured to divide the depth direction into multiple ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate multiple slice images that correspond to the multiple ranges; and a processing device structured to have multiple first correction characteristics defined corresponding to the multiple ranges, to correct each of the multiple slice images using the corresponding first correction characteristics, and to judge the kind of an object based on the multiple slice images thus corrected.

In this embodiment, the gating camera is employed as an active sensor. With the gating camera, the distance to each range is known. Accordingly, the first correction characteristics are determined giving consideration to attenuation characteristics of the light propagation path or the light divergence angle of the light to be emitted. Furthermore, an image is corrected using the first correction characteristics. Such an arrangement allows similar images to be acquired regardless of the position of the object. This provides an improved object identification rate.

Also, the first correction characteristics that correspond to each range may be changed according to a measurement environment. Infrared light has a characteristic of being readily absorbed by water. Accordingly, with the gating camera using infrared light, there is a difference in the attenuation rate between rainy weather, dense fog, and clear weather. Also, the attenuation rate differs due to humidity. Also, dust particles such as PM2.5 can have an effect on the attenuation rate. Accordingly, with such an arrangement in which the first correction characteristics are changed according to the measurement environment, this provides a further improved identification rate.

Also, the processing device may hold multiple second correction characteristics defined corresponding to the multiple ranges. Also, the processing device may correct each of the multiple slice images using the corresponding second correction characteristics, and may combine the multiple slice images thus corrected.

Also, an image obtained by combining the multiple corrected slice images may be displayed on a display.

Another embodiment of the present invention relates to a processing device. The processing device is used so as to form an object identification system together with a gating camera structured to divide the depth direction into multiple ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate multiple slice images that correspond to the multiple ranges. The processing device includes: a first correction unit structured to have multiple first correction characteristics defined corresponding to the multiple ranges, and to correct each of the multiple slice images using the corresponding first correction characteristics; and a classifier structured to judge the kind of an object based on the multiple corrected slice images.

Also, the processing device may further include a second correction unit structured to have multiple second correction characteristics defined corresponding to the multiple ranges, and to correct each of the multiple slice images using the corresponding second correction characteristics; a combining unit structured to combine the multiple slice images corrected by the second correction unit; and an output unit structured to output an image generated by the combining unit on a display.

Description will be made below regarding the embodiments 3-1 through 3-4 with reference to the drawings.

Embodiment 3-1

FIG.24is a block diagram showing an object identification system10A according to an embodiment 3-1. The object identification system10A is mounted on a vehicle such as an automobile, motorcycle, or the like. The object identification system10A judges the kind (which will also be referred to as a “category” or “class”) of an object OBJ that exists in the vicinity of the vehicle.

The object identification system10A mainly include a gating camera20and a processing device40A. The gating camera20includes an illumination apparatus22, an image sensor24, and a controller26. The gating camera20captures images for a plurality of N (N≥2) ranges RNG1through RNGNdivided in the depth direction. The ranges may be designed such that adjacent ranges overlap at their boundaries in the depth direction.

The illumination apparatus22emits probe light L1in front of the vehicle in synchronization with a light emission timing signal S1supplied from the controller26. As the probe light L1, infrared light is preferably employed. However, the present invention is not restricted to such an arrangement. Also, visible light having a predetermined wavelength may be employed. Also, ultraviolet light may be employed.

The image sensor24is configured to support exposure control in synchronization with an image capture timing signal S2supplied from the controller26, and to be capable of generating a slice image IMG. The image sensor24is sensitive to the same wavelength as that of the probe light L1. The image sensor24captures an image of reflected light (returned light) L2reflected by the object OBJ.

The controller26changes the light emission timing signal S1and the image capture timing signal S2for each range RNG, so as to change the difference in the timing between the light emission operation of the illumination apparatus22and the exposure operation of the image sensor24. The gating camera20generates the slice images IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN. The i-th slice image IMGiincludes only an image of the object included in the corresponding range RNGi.

FIG.25is a diagram for explaining the basic operation of the gating camera20.FIG.25shows the operation when the i-th range RNGiis measured. The illumination apparatus22emits light during a light emission period τ1from the time points t0to t1in synchronization with the light emission timing signal S1. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance. The distance between the gating camera20and the near-distance boundary of the range RNGiis represented by dMINi. The distance between the gating camera20and the far-distance boundary of the range RNGiis represented by dMAXi.

The round-trip time TMINi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMINi, up to the return of the reflected light to the image sensor24, is represented by TMINi=2×dMINi/c. Here, c represents the speed of light.

Similarly, the round-trip time TMAXi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMAXi, up to the return of the reflected light to the image sensor24, is represented by TMAXi=2×dMAXi/c.

When only an image of an object OBJ included in the range RNGiis to be captured, the controller26generates the image capture timing signal S2A so as to start the exposure at the time point t2=t0+TMINi, and so as to end the exposure at the time point t3=t1+TMAXi. This is a single exposure operation.

When an image is captured for the i-th range RNGi, the exposure may be executed multiple times. In this case, preferably, the controller26may repeatedly execute the above-described exposure operation multiple times with a predetermined period τ2.

FIGS.26A and26Bare diagrams for explaining an image generated by the gating camera20.FIG.26Ashows an example in which an object (pedestrian) OBJ2exists in the range RNG2, and an object (vehicle) OBJ3exists in the range RNG3.FIG.26Bshows multiple slice images IMG1through IMG3acquired in the situation shown inFIG.26A. When the slice image IMG1is captured, the image sensor is exposed by only the reflected light from the range RNG1. Accordingly, the slice image IMG1includes no object image.

When the slice image IMG2is captured, the image sensor is exposed by only the reflected light from the range RNG2. Accordingly, the slice image IMG2includes only the object image OBJ2. Similarly, when the slice image IMG3is captured, the image sensor is exposed by only the reflected light from the range RNG3. Accordingly, the slice image IMG3includes only the object image OBJ3. As described above, with the gating camera20, this arrangement is capable of capturing object images in the form of separate images for the respective ranges.

Returning toFIG.24, the processing device40A is configured to identify the kind of an object based on the multiple slice images IMG1through IMGNthat correspond to the range RNG1through RNGNgenerated by the gating camera20. The processing device40A is provided with the classifier42implemented based on a prediction model generated by machine learning. The algorithm employed by the classifier42is not restricted in particular. Examples of the algorithms that can be employed include You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), Region-based Convolutional Neural Network (R-CNN), Spatial Pyramid Pooling (SPPnet), Faster R Deconvolution-SSD (DSSD), Mask R-CNN, etc. Also, other algorithms that will be developed in the future may be employed.

The processing device40A may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, or the like, and a software program to be executed by the processor (hardware component). Also, the processing device40A may be configured as a combination of multiple processors. Alternatively, the processing device40A may be configured as a hardware component alone.

The processing device40includes a first correction unit50provided as an upstream stage of the classifier42. Multiple correction characteristics (which will be referred to as a “set of correction characteristics”) that correspond to the multiple ranges RNG1through RNGNare defined in the first correction unit50. The correction characteristics can be represented in the form of a function p′=f(p) with the pixel value p before correction as an argument (input), and with the output of the pixel value after correction as an output.

The first correction unit50corrects each of the multiple slice images IMG1through IMGNusing the corresponding one from among the correction characteristics f1(p) through fN(p), so as to generate the slice images IMGa1through IMGaNafter the correction.

The correction characteristics fi(p) are defined corresponding to the i-th range RNGi. In an example, the correction characteristics fi(p) is defined with only the pixel value as its argument without depending on the pixel position. In this case, as described above, the following relation holds true between the pixel value p of a given pixel included in an image before correction and the pixel value of the pixel included in the image after correction.
p′=fi(p)

FIGS.27A through27Dare diagrams showing several examples of the correction characteristics f. Description will be made with reference toFIG.27A. In this example, a set of the correction characteristics f1(p) through fN(p) are each defined as a linear function, and are represented by the following Expression. Here, “αi” represents a correction coefficient defined for each range. As the range becomes farther, the value of the correction coefficient αibecomes larger. It should be noted that p′ is clamped when it reaches its maximum value.
p′=fi(p)=αi×p

The correction using the set of the correction characteristics corresponds to the exposure correction.

Description will be made with reference toFIG.27B. In this example, a set of the correction characteristics fi(p) through fN(p) each have a so-called S-curve. Such an S-curve provides an effect of raising the image contrast.

With the normalized S-curve as S(p), the correction characteristics may be defined as follows.
p′=f′(p)=αi×S(p)

Alternatively, the correction characteristics may be defined as follows.
p′=fi(p)=S(αi×p)

Description will be made with reference toFIG.27C. In this example, a set of the correction characteristics f1(p) through fN(p) each have a so-called inverse S-curve. Such an inverse S-curve provides an effect of reducing the image contrast. With the normalized inverse S-curve as INVS(p), the correction characteristics may be defined as follows.
p′=fi(p)=αi×INVS(p)
Alternatively, the correction characteristics may be defined as follows.
p′=fi(p)=INVS(αi×p)

Description will be made with reference toFIG.27D. In this example, a set of the correction characteristics f1(p) through fN(p) each provide so-called gamma correction.
p′=fi(p)=p{circumflex over ( )}γi
Here, γirepresents a gamma correction value for the i-th range.

It should be noted that the correction characteristics are not restricted to those shown inFIGS.27A through27D. Also, the correction characteristics may be defined as a quadratic function or a higher-order function. Also, the correction characteristics may be defined using an exponential function or a trigonometric function. Also, the correction characteristics are not necessarily required to be defined in the form of a function. That is to say, the processing device40may hold a function f(p) that represents the correction characteristics, and may input a value p to the function f(p) so as to acquire the output p. Also, the processing device40may hold a lookup table for defining the relation between the input p and the output p′.

Description has been made with reference toFIGS.27A through27Dregarding an arrangement in which a set of the correction characteristics f1(p) through fN(p) are represented as the same type of function with different parameters (α or γ). However, the present invention is not restricted to such an arrangement. Also, different types of functions may be employed for the respective ranges.

That is to say, such a set of the correction characteristics f1(p) through fN(p) may preferably be defined so as to provide the classifier42configured as a downstream stage with an improved object identification rate.

Returning toFIG.24, the first correction unit50supplies the corrected slice images IMGa1through IMGaNto the classifier42. The classifier42distinguishes the kind of an object included in each slice image IMGaifor each of the corrected slice images IMGa1through IMGaN.

For example, the output (which will be referred to as “detection data”) OUTi(i=1, 2, . . . . N) of the classifier42includes size information (position information) with respect to each object image included in the i-th image data IMGaiand the kind (category) information thereof. The detection data OUT may include information with respect to a bounding box for each object. The kind information may indicate the possibility (matching probability) of the object matching each of multiple kinds. Also, the kind information may include an identifier that indicates the kind that matches a possible object with the highest belonging probability.

The above is the configuration of the object identification system10A. Next, description will be made regarding the operation thereof.

FIGS.28A and28Bare diagrams for explaining the image capture by the gating camera20.FIG.28Ais a diagram showing a given measurement situation as viewed from the side.FIG.28Bshows images captured by the gating camera20. In this example, objects (humans) OBJ2, OBJi, and OBJNexist in the ranges RNG2, RNGi, and RNGN.

In the images shown inFIG.28B, the hatching density represents the magnitude of the pixel value. In a case in which an image of an object (human) having the same reflection ratio is captured, the pixel values of the object image when it exists in the slice image IMG2that corresponds to a near-distance range are larger than those when it exists in the slice image IMGNthat corresponds to a far-distance range.

FIGS.29A and29Bare diagrams for explaining the correction by the first correction unit50. Description will be made below assuming that the correction characteristics shown inFIG.27Aare employed. With such an arrangement, different exposure correction is applied for each range, thereby generating the corrected slice images IMGa2, IMGai, and IMGaN. Specifically, the correction characteristics f2(p), which apply negative exposure correction, are applied to the slice image IMG2in a near-distance range. On the other hand, the correction characteristics fN(p), which apply positive exposure correction, are applied to the slice image IMGNin a far-distance range. As a result, the pixel values (hatching density in the drawings) of each object image are corrected such that they become approximately the same values for the corrected slice images IMGa2, IMGai, and IMGaN.

If the slice images IMG2, IMGi, and IMGNbefore correction are input to the classifier42, correct judgement of the existence of a human can be made for a particular image (e.g., IMGi). However, such an arrangement has the potential to have a problem in that correct judgment of the existence of a human cannot be made for different images (e.g., IMG2and IMGN). In contrast, with such an arrangement in which the corrected slice images IMGa2, IMGai, and IMGaNare input to the classifier42, this enables the judgement that an object included in each slice image matches a human.

The above is the operation of the object identification system10A. With the object identification system10A, this provides an improved object identification rate.

Embodiment 3-2

FIG.30is a block diagram showing an object identification system10B according to an embodiment 3-2. In the embodiment 3-2, the first correction characteristics for each range are changed according to the measurement environment. Infrared light has a characteristic of being readily absorbed by water. Accordingly, in rainy weather or dense fog, the first correction characteristics may preferably be changed so as to raise the correction level.

The first correction unit50of a processing device40B holds multiple sets of first correction characteristics defined corresponding to multiple environments. The set of the first correction characteristics may preferably be defined for each environment having different propagation characteristics (attenuation characteristics) of the probe light L1and the reflected light L2. For example, multiple sets of the first correction characteristics may be defined according to the weather (sunny, rainy, cloudy, foggy). Also, multiple sets of the first correction characteristics may be defined according to the measurement time of day (day, night, dusk). In a case in which the object identification system10is mounted on a moving vehicle such as an automobile or the like, multiple sets of the first correction characteristics may be defined according to multiple driving situations.

The image display system10B includes an environment judgment unit70. The environment judgment unit70judges the current measurement environment (weather, time of day, driving situation), and notifies the first correction unit50of this information. The first correction unit50selects one from among the multiple first correction characteristics based on the notice thus received from the environment judgment unit70. Subsequently, the first correction unit50corrects the image for each range using the set of the correction characteristics thus selected.

With the embodiment 3-2, this provides a further improved object identification rate as compared with the embodiment 3-1.

Embodiment 3-3

FIG.31is a diagram showing an object identification system10C according to an embodiment 3-3. In the object identification system10C, a processing device40C further includes a second correction unit52and a combining unit54.

The second correction unit52holds multiple second correction characteristics (which will be referred to as a “set of multiple second correction characteristics”) defined corresponding to the multiple ranges RNG1through RNGN. In the same manner as the first correction characteristics, the second correction characteristics can each be represented by a function p′=g (p) with the pixel value p before correction as an argument (input) and with the pixel value after the correction as an output.

The second correction unit52corrects each of the multiple slice images IMG1through IMGNusing the corresponding second correction characteristics from among g1(p) through gN(p), and generates the corrected slice images IMGb1through IMGbN.

The second correction characteristics gi(p) are defined corresponding to the i-th range RNGi. In an example, the second correction characteristics gi(p) are defined with only the pixel value as an argument without depending on the pixel position. In this case, as described above, the following relation holds true between the pixel value p of a given pixel included in an image before the correction and the pixel value p′ of the pixel in the corrected image.
p′=gi(p)

The corrected slice images IMGb1through IMGbNare input to the combining unit54. The combining unit54recombines the corrected slice images IMGb1through IMGbNSO as to generate a single slice image IMGc. The slice image IMGc thus combined is displayed on a display80, so as to present the slice image IMGc thus combined to the user.

The object identification system10C is mounted on an automobile, for example. In this case, the display80is arranged at the driver's seat so as to allow the infrared slice image IMGc thus recombined to be displayed to the driver.

The above is the configuration of the object identification system10C. Next, description will be made regarding the operation thereof. Description has been made regarding the classifier42with an improved object identification rate. Accordingly, description will be made regarding an image display operation of the display80.

FIGS.32A through32Dare diagrams for explaining the image display by the object identification system10C.FIG.32Ais a perspective view showing a measurement situation. An object (human) OBJ2exists in the range RNG2, and an object (automobile) OBJ6exists in the range RNG6.

FIG.32Bshows the slice images IMG2and IMG6generated by the gating camera20in the situation shown inFIG.32A. In the slice image IMG2in the near-distance range RNG2, the object image of the object OBJ2has large pixel values. Conversely, in the slice image IMG6in the far-distance range, the object image of the object OBJ6has small pixel values. InFIGS.32B through32D, the pixel value is represented by the dot density. That is to say, a portion with a low dot density represents a dark image portion on the display.

In a case in which the two slice images IMG2and IMG6shown inFIG.32Bare combined as they are and the image thus combined is displayed on a display, a human image OBJ2in the near distance is displayed as a bright object image. However, the automobile image OBJ6in the far distance is displayed as a very dark image. Accordingly, it is difficult for the user to recognize the automobile OBJ6.

In order to solve this problem, with the embodiment 3-3, the second correction unit52corrects the slice images IMG2and IMG6. In the corrected slice images IMGb2and IMGb6, the two object images OBJ2and OBJ6are corrected such that they have pixel values on the same order or a similar order. Furthermore, the image contrast is optimized as necessary so as to allow the user to easily view the object images. Subsequently, the corrected slice images IMGb1through IMGbNare combined so as to generate the combined slice image IMGc. The combined slice image IMGc is corrected such that the two object images OBJ2and OBJ6have pixel values on the same order or a similar order. Accordingly, when the combined slice image IMGc is displayed on the display80, this allows the user to easily recognize both the human OBJ2and the automobile OBJ6.

Embodiment 3-4

FIG.33is a diagram showing an object identification system10D according to an embodiment 3-4. In the embodiment 3-4, the first correction characteristics fiand the second correction characteristics giare designed to be the same for the same range RNGi. In this case, the second correction unit52may preferably be omitted. Also, the corrected slice images IMGa1through IMGaNcorrected by the first correction unit50may preferably be input to the combining unit54.

Description will be made regarding a modification relating to the embodiment 3.

Modification 3-1

Description has been made in the embodiment regarding an arrangement in which, when a single image is corrected, all the pixels are corrected according to the same correction characteristics. However, the correction characteristics may have position dependance. That is to say, the correction characteristics may be defined as a function with the pixel position as an argument in addition to the pixel value p.

Usage

FIG.34is a block diagram showing an automobile provided with the object identification system10. An automobile300is provided with headlamps302L and302R. The object identification system10is built into at least one from among the headlamps302L and302R. Each headlamp302is positioned at a frontmost end of the vehicle body, which is most advantageous as a position where the gating camera20is to be installed for detecting an object in the vicinity.

FIG.35is a block diagram showing an automotive lamp200provided with an object detection system210. The automotive lamp200forms a lamp system310together with an in-vehicle ECU304. The automotive lamp200includes a light source202, a lighting circuit204, and an optical system206. Furthermore, the automotive lamp200includes the object detection system210. The object detection system210corresponds to the object identification system10described above. The object detection system210includes the gating camera20and the processing device40.

Also, the information with respect to the object OBJ detected by the processing device40may be used to support the light distribution control operation of the automotive lamp200. Specifically, a lamp ECU208generates a suitable light distribution pattern based on the information with respect to the kind of the object OBJ and the position thereof generated by the processing device40. The lighting circuit204and the optical system206operate so as to provide the light distribution pattern generated by the lamp ECU208.

Also, the information with respect to the object OBJ detected by the processing device40may be transmitted to the in-vehicle ECU304. The in-vehicle ECU may support autonomous driving based on the information thus transmitted.

Embodiment 4

First, description will be made regarding the problems relating to an embodiment 4.

When the vehicle is traveling at night or at dusk, it is difficult for the driver to see an object in the vicinity. In order to compensate for this difficulty, the present inventor has investigated an image display system in which an active sensor using infrared light is installed on a vehicle, and a captured image is displayed on a display so as to be presented to the driver.

FIGS.36A and36Bare diagrams for explaining the characteristics of a conventional active sensor.FIG.36Ais a diagram showing an image capture situation as viewed from the side.FIG.36Bshows an image captured by the active sensor in the image capture situation shown inFIG.36A.

Light (infrared light) attenuates as the propagation distance becomes longer. Accordingly, as the distance to an object becomes larger, the amount of light that reaches the object becomes smaller, and the amount of light reflected from the object becomes smaller. That is to say, in a case of employing the active sensor, an image of a near-distance object is captured as a bright image, and an image of a far-distance object is captured as a dark image. Accordingly, in a case in which such images are displayed as they are, there is a large difference in appearance between the object images depending on the distance to the object.

The invention relating to the embodiment 4 has been made in view of such a situation. Accordingly, it is an exemplary purpose of an embodiment of the invention to provide an image display system that is capable of displaying an image in an easy-to-see form.

Overview of Embodiment 4

An embodiment disclosed in the present specification relates to an image display system. The image display system includes a gating camera structured to divide the depth direction into multiple ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate multiple slice images that correspond to the multiple ranges; and a processing device structured to combine the multiple slice images so as to generate a combined image to be displayed on a display.

With this embodiment, the gating camera is employed as an active sensor. With the gating camera, by adjusting the exposure (shutter speed or sensitivity) and the light emission intensity of the probe light for each range, this solves a problem of variation depending on the distance to an object. By combining the multiple slice images thus obtained, this allows an easy-to-see image to be generated.

Also, the processing device may be structured to have multiple correction characteristics defined corresponding to the multiple ranges, to correct each of the multiple slice images using the corresponding correction characteristics, and to combine the multiple corrected slice images.

With the gating camera, an image of an individual object can be captured for each range. Furthermore, the distance to each range is known. Accordingly, the correction characteristics are determined for each range giving consideration to the attenuation characteristics of the light propagation path or the light divergence angle of the light to be emitted. Furthermore, an image is corrected using the correction characteristics thus determined. This allows similar images to be acquired regardless of the distance to the object. In this case, such an arrangement does not require adjustment of the shutter speed and the sensitivity for each range.

Also, the correction characteristics that correspond to each range may be changed according to a measurement environment. For example, infrared light has a characteristic of being readily absorbed by water. Accordingly, with the gating camera using infrared light, there is a difference in the attenuation rate between rainy weather, dense fog, and clear weather. Also, the attenuation rate differs due to humidity. Also, dust particles such as PM2.5 can have an effect on the attenuation rate. Accordingly, with such an arrangement in which the first correction characteristics are changed according to the measurement environment, this provides a further improved visibility.

Description will be made below regarding the embodiments 4-1 through 4-3 with reference to the drawings.

Embodiment 4-1

FIG.37is a block diagram showing an image display system11E according to an embodiment 4-1. The image display system11E is mounted on a vehicle such as an automobile, motorcycle, or the like. The image display system11E presents an image in the vicinity of (e.g., in front of) the vehicle to the driver.

The image display system11E mainly include a gating camera20E, a processing device40E, and a display80. The gating camera20E includes an illumination apparatus22, an image sensor24, and a controller26. The gating camera20E captures images for a plurality of N (N≥2) ranges RNG1through RNGNdivided in the depth direction. The ranges may be designed such that adjacent ranges overlap at their boundaries in the depth direction.

The illumination apparatus22emits probe light L1in front of the vehicle in synchronization with a light emission timing signal S1supplied from the controller26. As the probe light L1, infrared light is preferably employed. However, the present invention is not restricted to such an arrangement. Also, visible light having a predetermined wavelength may be employed.

The image sensor24is configured to support exposure control in synchronization with an image capture timing signal S2supplied from the controller26, and to be capable of generating a slice image IMG. The image sensor24is sensitive to the same wavelength as that of the probe light L1. The image sensor24captures an image of reflected light (returned light) L2reflected by the object OBJ.

The controller26changes the light emission timing signal S1and the image capture timing signal S2for each range RNG, so as to change the difference in the timing between the light emission operation of the illumination apparatus22and the exposure operation of the image sensor24. The gating camera20E generates the slice images IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN. The i-th slice image IMGiincludes only an image of the object included in the corresponding range RNGi.

FIG.38is a diagram for explaining the basic operation of the gating camera20E.FIG.38shows the operation when i-th range RNGiis measured. The illumination apparatus22emits light during a light emission period τ1from the time points t0to t1in synchronization with the light emission timing signal S1. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance. The distance between the gating camera20E and the near-distance boundary of the range RNGiis represented by dMINi. The distance between the gating camera20E and the far-distance boundary of the range RNGiis represented by dMAXi.

The round-trip time TMINi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMINi, up to the return of the reflected light to the image sensor24, is represented by TMINi=2×dMINi/c. Here, c represents the speed of light.

Similarly, the round-trip time TMAXi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMAXi, up to the return of the reflected light to the image sensor24, is represented by TMAXi=2×dMAXi/c.

When only an image of an object OBJ included in the range RNGiis to be captured, the controller26generates the image capture timing signal S2A so as to start the exposure at the time point t2=t0+TMINi, and so as to end the exposure at the time point t3=t1+TMAXi. This is a single exposure operation.

When an image is captured for the i-th range RNGi, the exposure may be executed multiple times. In this case, preferably, the controller26may repeatedly execute the above-described exposure operation multiple times with a predetermined period τ2.

In the present embodiment, in order to prevent the occurrence of variation in the exposure (brightness value of an object image in an image) for each range, the gating camera20E is designed to provide an optimized shutter speed (exposure time), an optimized number of exposures, an optimized sensitivity, an optimized illumination intensity of the probe light, etc.

FIGS.39A and39Bare diagrams for explaining an image generated by the gating camera20E.FIG.39Ashows an example in which an object (pedestrian) OBJ2exists in the range RNG2, and an object (vehicle) OBJ3exists in the range RNG3.FIG.39Bshows multiple slice images IMG1through IMG3acquired in the situation shown inFIG.39A. When the slice image IMG1is captured, the image sensor is exposed by only the reflected light from the range RNG1. Accordingly, the slice image IMG1includes no object image.

When the slice image IMG2is captured, the image sensor is exposed by only the reflected light from the range RNG2. Accordingly, the slice image IMG2includes only the object image OBJ2. Similarly, when the slice image IMG3is captured, the image sensor is exposed by only the reflected light from the range RNG3. Accordingly, the slice image IMG3includes only the object image OBJ3. As described above, with the gating camera20E, this arrangement is capable of capturing object images in the form of separate images for the respective ranges.

Returning toFIG.37, the processing device40E combines the multiple slice images IMG1through IMGNthat correspond to the ranges RNG1through RNGNgenerated by the gating camera20E, so as to generate a combined slice image IMGc to be displayed on the display80.

The processing device40E may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, Graphics Processing Unit (GPU), or the like, and a software program to be executed by the processor (hardware component). Also, the processing device40E may be configured as a combination of multiple processors. Alternatively, the processing device40E may be configured as a hardware component alone.

The above is the image display system11E. Next, description will be made regarding the operation thereof.FIGS.40A and40Bare diagrams for explaining the image capture operation of the gating camera20E.FIG.40Ais a diagram showing a measurement situation as viewed from the side.FIG.40Bshows images captured by the gating camera20E. The situation shown inFIG.40Ais the same as that shown inFIG.36A. In this example, the objects (humans) OBJ2, OBJi, and OBJNexist in the ranges RNG2, RNGi, and RNGN, respectively.

In the images shown inFIG.40B, the hatching density represents the magnitude of the pixel value. In the present embodiment, when an image of an object (human) having the same reflection ratio is captured, the gating camera20E is optimized in a hardware manner or a software manner such that the object image has approximately the same pixel values regardless of whether it exists in the slice image IMG2that corresponds to a near-distance range or in the slice image IMGNthat corresponds to a far-distance range.

FIG.41Ais a diagram showing a combined image obtained by combining the multiple slice images shown inFIG.40B. As a comparison,FIG.41Bshows an image (i.e., image shown inFIG.36B) generated by a conventional infrared active sensor in a situation shown inFIG.40A. As shown inFIG.41B, with such a conventional infrared active sensor, when an image of the same object is captured, as the distance to the object becomes larger, the pixel values become smaller. Conversely, as the distance to the object becomes smaller, the pixel values become larger. Accordingly, this leads to a problem in that bright object images and dark object images are mixed in a single image. In a case in which such an image is displayed on the display, it is difficult for the user to see the object images.

In contrast, with the image display system11E according to the embodiment 4-1, the gating camera20E generates the slice images IMG1through IMGNwith approximately the same exposure (brightness). With such an arrangement in which such slice images IMG1through IMGNare combined, this provides improved visibility as compared with conventional arrangements.

Embodiment 4-2

Description has been made in the embodiment 4-1 regarding an arrangement in which multiple slice images IMG1through IMGNare corrected to have uniform brightness by means of the gating camera20E optimized in a hardware manner or a software manner. However, in some cases, it is difficult to optimize the gating camera20E depending on the kind or model of the gating camera. Also, in some cases, such optimization is able to provide only insufficient effects. The embodiment 4-2 is particularly effective in such a case.

FIG.42is a block diagram showing an image display system11F according to an embodiment 4-2. In the image display system11F, the processing device40F includes a correction unit52and a combining unit54.

Multiple correction characteristics (which will be referred to as a “set of correction characteristics”) that correspond to the multiple ranges RNG1through RNGNare defined in the correction unit52. The correction characteristics can each be represented in the form of a function p′=f(p) with the pixel value p before correction as an argument (input), and with the output of the pixel value after correction as an output.

The correction unit52corrects each of the multiple slice images IMG1through IMGNusing the corresponding one from among the characteristics g1(p) through gN(p), so as to generate the slice images IMGb1through IMGbNafter the correction.

The correction characteristics gi(p) are defined corresponding to the i-th range RNGi. In an example, the correction characteristics gi(p) are defined with only the pixel value as an argument without depending on the pixel position. In this case, as described above, the following relation holds true between the pixel value p of a given pixel included in an image before correction and the pixel value of the pixel included in the image after the correction.
p′=gi(p)

FIGS.43A through43Dare diagrams showing several examples of the correction characteristics g. Description will be made with reference toFIG.43A. In this example, a set of the correction characteristics gi(p) through gN(p) are linear functions, and are represented by the following Expression. Here, “αi” represents a correction coefficient defined for each range. As the range becomes farther, the value of the correction coefficient αibecomes larger. It should be noted that p′ is clamped when it reaches its maximum value.
p′=gi(p)=αi×p

The correction using this set of correction characteristics corresponds to exposure correction.

Description will be made with reference toFIG.43B. In this example, a set of the correction characteristics g1(p) through gN(p) are S-curves. Such S-curves provide an effect of raising the image contrast.

With the normalized S-curve as S(p), the correction characteristics may be defined as follows.
p′=g′(p)=αi×S(p)

Alternatively, the correction characteristics may be defined as follows.
p′=gi(p)=S(αi×p)

Description will be made with reference toFIG.43C. In this example, a set of the correction characteristics g1(p) through gN(p) are so-called inverse S-curves. Such inverse S-curves provide an effect of reducing the image contrast. With the normalized inverse S-curve as INVS(p), the correction characteristics may be defined as follows.
p′=gi(p)=αi×INVS(p)

Alternatively, the correction characteristics may be defined as follows.
p′=gi(p)=INVS(αi×p)

Description will be made with reference toFIG.43D. In this example, a set of the correction characteristics g1(p) through gN(p) each represent so-called gamma correction.
p′=gi(p)=p{circumflex over ( )}γi
Here, γirepresents a gamma correction value for the i-th range.

It should be noted that the correction characteristics are not restricted to the functions shown inFIGS.43A through43D. Also, the correction characteristics may be defined as a quadratic function or a higher-order function. Also, the correction characteristics may be defined using an exponential function or a trigonometric function. Also, the correction characteristics are not necessarily required to be defined in the form of a function. That is to say, the processing device40may hold a function g (p) that represents the correction characteristics, and may input a value p to the function f(p) so as to acquire the output p. Also, the processing device40may hold a lookup table for defining the relation between the input p and the output p′.

Description has been made with reference toFIGS.43A through43Dregarding an arrangement in which a set of the correction characteristics g1(p) through gN(p) are represented as the same type of function with different parameters (α or γ). However, the present invention is not restricted to such an arrangement. Also, different types of functions may be employed for the respective ranges.

That is to say, such a set of the correction characteristics g1(p) through gN(p) may preferably be defined such that, when an image of the same object is captured, a similar object image is acquired regardless of the range.

Returning toFIG.42, the corrected slice images IMGb1through IMGbNare input to the combining unit54. The combining unit54recombines the corrected slice images IMGb1through IMGbNso as to generate a single slice image IMGc. The slice image IMGc thus combined is displayed on a display80, so as to present the slice image IMGc thus combined to the user.

The above is the configuration of the image display system11F. Next, description will be made regarding the operation thereof.FIGS.44A and44Bare diagrams for explaining the processing of the correction unit52shown inFIG.42.FIG.44Ashows multiple slice images IMG1through IMGNgenerated by the gating camera20. As the range becomes farther, the pixel values become smaller. Conversely, as the range becomes nearer, the pixel values become larger.

FIG.44Bshows the corrected slice images IMGb1through IMGbNcorrected by the correction unit52. With such an arrangement in which the corrected slice images IMGb1through IMGbNare combined, this allows an easy-to-see combined slice image IMGc to be acquired as shown inFIG.41A.

Embodiment 4-3

FIG.45is a block diagram showing an object identification system11G according to an embodiment 4-3. In this embodiment, the correction characteristics for each range are changed according to the measurement environment. Infrared light has a characteristic of being readily absorbed by water. Accordingly, in rainy weather or dense fog, the first correction characteristics may preferably be changed so as to raise the correction level.

The correction unit52of a processing device40G holds multiple sets of correction characteristics defined corresponding to multiple environments. The set of the correction characteristics may preferably be defined for each environment having different propagation characteristics (attenuation characteristics) of the probe light L1and the reflected light L2. For example, the multiple sets of the correction characteristics may be defined according to the weather (sunny, rainy, cloudy, foggy). Also, the multiple sets of the correction characteristics may be defined according to the measurement time of day (day, night, dusk). In a case in which the object identification system11is mounted on a moving vehicle such as an automobile or the like, multiple sets of the correction characteristics may be defined according to multiple driving situations.

The image display system11G includes an environment judgment unit70. The environment judgment unit70judges the current measurement environment (weather, time of day, driving situation), and notifies the correction unit52of this information. The correction unit52selects one from among the multiple correction characteristics based on the notice thus received from the environment judgment unit70. Subsequently, the correction unit52corrects the image for each range using the set of the correction characteristics thus selected.

With the embodiment 4-3, this provides a further improved object identification rate as compared with the embodiment 4-2.

Description will be made regarding a modification relating to the embodiment 4.

Modification 4-1

Description has been made in the embodiment regarding an arrangement in which, when a single image is corrected, all the pixels are corrected according to the same correction characteristics. However, the correction characteristics may be designed to have position dependance. That is to say, the correction characteristics may be defined as a function with the pixel position as an argument in addition to the pixel value p.

Embodiment 5

First, description will be made regarding problems relating to an embodiment 5.

In the operation testing of the gating camera or a system including the gating camera as a built-in component, the following test is required. That is to say, several or all the ranges are selected as test target ranges. After a reference reflector is arranged for each of the test target ranges, images are captured by means of the gating camera, and judgement is made regarding whether or not images can be captured normally. In a case in which the gating camera is employed as an in-vehicle camera, such an arrangement requires capturing images of objects in a range several dozen or several hundred meters or more ahead of the gating camera. Accordingly, in a case in which such a far-distance range is selected as the test target range, this leads to a problem in that a test requires an extensive space.

Even if such an extensive space can be used, in a case in which multiple test target ranges are selected, such a test must be executed while changing the position of the reference reflector. This leads to an increased workload required for the test, which becomes a factor of an increased cost.

The invention relating to the embodiment 5 has been made in view of such a situation. Accordingly, it is an exemplary purpose of an embodiment of the invention to provide a gating camera that can be tested in a state in which the gating camera is focused on a given range using a reference reflector arranged in a different range.

Overview of Embodiment 5

An embodiment disclosed in the present specification relates to a gating camera structured to divide the depth direction into multiple ranges, and to generate multiple slice images that correspond to the multiple ranges. The gating camera includes: an illumination apparatus structured to emit probe light according to a light emission timing signal; an image sensor structured to capture an image of reflected light according to an image capture timing signal; and a controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range. In a testing process, the gating camera is structured to be capable of applying an offset to the time difference between the light emission timing signal and the image capture timing signal.

In a case of operating the controller in a state in which a given range is set to be an image capture target, the light emission timing signal and the image capture signal are generated with a time difference that corresponds to the image capture target range. In a case of applying a negative offset to the time difference between the light emission timing signal and the image capture timing signal, an image of an object that exists in a range that is nearer than the image capture target range thus set is captured. In a case of applying a positive offset to the time difference between the light emission timing signal and the image capture timing signal, an image of an object that exists in a range that is farther than the image capture target range thus set is captured. With this, in a state in which a given range is set to the image capture target, a test can be executed using a reference reflector arranged in a different range.

Also, the gating camera may be structured to allow an external delay unit to be connected on a path of the light emission timing signal. Also, the gating camera may further include a delay unit to be inserted on a signal path of the light emission timing signal in a testing process. With this, a test can be executed using a reference reflector arranged in a relatively near-distance range in a state in which a relatively far-distance range is set to be an image capture target. This allows the test space to be reduced.

Also, the gating camera may be structured to allow an external delay unit to be connected on a signal path of the image capture timing signal. Also, the gating camera may further include a delay unit to be inserted on a signal path of the image capture timing signal. With this, a test can be executed using a reference reflector arranged in a relatively far-distance range in a state in which a relatively near-distance range is set to be an image capture target.

With an embodiment, a gating camera includes: an illumination apparatus structured to emit probe light according to a light emission timing signal; an image sensor structured to capture an image of reflected light according to an image capture timing signal; and a controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range. In a testing process, the gating camera is structured to be capable of outputting the light emission timing signal to an external reference light source via an external delay unit. This allows the gating camera to be tested using an external reference light source. Furthermore, this allows the space required for the test to be reduced.

With an embodiment, a gating camera includes: an illumination apparatus structured to emit probe light according to a light emission timing signal; an image sensor structured to capture an image of reflected light according to an image capture timing signal; and a controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range. In a testing process, the gating camera is structured to be capable of outputting the image capture timing signal to an external image sensor via an external delay unit. This allows the gating camera to be tested using an external image sensor. Furthermore, this allows the space required for the test to be reduced.

Description will be made with reference to the drawings regarding an embodiment 5.

Embodiment 5-1

FIG.46is a block diagram showing a gating camera20according to an embodiment 5-1. The gating camera20captures images for a plurality of N (N≥2) ranges RNG1through RNGNdivided in the depth direction, so as to generate multiple images (slice images) IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN. The ranges may be designed such that adjacent ranges overlap at their boundaries in the depth direction.

The gating camera20includes an illumination apparatus22, an image sensor24, and a controller26.

The illumination apparatus22emits probe light L1in front of the vehicle in synchronization with a light emission timing signal S1supplied from the controller26. As the probe light L1, infrared light is preferably employed. However, the present invention is not restricted to such an arrangement. Also, visible light having a predetermined wavelength or ultraviolet light may be employed.

The image sensor24is configured to support exposure control in synchronization with an image capture timing signal S2supplied from the controller26, and to be capable of generating a slice image IMG. The image sensor24is sensitive to the same wavelength as that of the probe light L1. The image sensor24captures an image of reflected light (returned light) L2reflected by the object OBJ.

The controller26defines a pulse width of each of the light emission timing signal S1and the image capture timing signal S2and a time difference between them for each range. The light emission timing signal S1and the image capture timing signal S2thus defined for the i-th (1≤i≤N) range are represented by S1iand S2iwith the range number i as a suffix. The operation in which i-th range RNGiis set to be the image capture target will be referred to as “focusing on the i-th range RNGi”. This corresponds to the selection of the light emission timing signal S1iand the image capture timing signal S21. The range RNGiin this state will also be referred to as a “focus range”. It should be noted that “focus” in this usage differs from “focus” in the optical meaning.

FIG.47is a diagram for explaining the basic operation of the gating camera20.FIG.47shows the operation when the i-th range RNGiis measured. The illumination apparatus22emits light during a light emission period τ1from the time points t0to t1in synchronization with the light emission timing signal S1i. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance. The distance between the gating camera20and the near-distance boundary of the range RNGiis represented by dMINi. The distance between the gating camera20and the far-distance boundary of the range RNGiis represented by dMAXi.

The round-trip time TMINi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMINi, up to the return of the reflected light to the image sensor24, is represented by TMINi=2×dMINi/c. Here, c represents the speed of light.

Similarly, the round-trip time TMAXi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMAXi, up to the return of the reflected light to the image sensor24, is represented by TMAXi=2×dMAXi/c.

When only an image of an object OBJ included in the range RNGiis to be captured, the controller26generates the image capture timing signal S2so as to start the exposure at the time point t2=t0+TMINi, and so as to end the exposure at the time point t3=t1+TMAXi. This is a single exposure operation.

When an image is captured for the i-th range RNGi, the exposure may be executed multiple times. In this case, preferably, the controller26may repeatedly execute the above-described exposure operation multiple times with a predetermined period τ2.

FIGS.48A and48Bare diagrams for explaining an image generated by the gating camera20.FIG.48Ashows an example in which an object (pedestrian) OBJ2exists in the range RNG2, and an object (vehicle) OBJ3exists in the range RNG3.FIG.48Bshows multiple slice images IMG1through IMG3acquired in the situation shown inFIG.48A. When the slice image IMG1is captured, the image sensor is exposed by only the reflected light from the range RNG1. Accordingly, the slice image IMG1includes no object image.

When the slice image IMG2is captured, the image sensor is exposed by only the reflected light from the range RNG2. Accordingly, the slice image IMG2includes only the object image OBJ2. Similarly, when the slice image IMG3is captured, the image sensor is exposed by only the reflected light from the range RNG3. Accordingly, the slice image IMG3includes only the object image OBJ3. As described above, with the gating camera20, this arrangement is capable of capturing object images in the form of separate images for the respective ranges.

The gating camera20generates multiple slice images IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN. As the i-th slice image IMGi, only an image of an object included in the corresponding range RNGiis captured.

Returning toFIG.46, in the testing process, the gating camera20is configured such that an offset t can be applied to the time difference Δt between the light emission timing signal S1and the image capture timing signal S2. In the present embodiment, the gating camera20is configured to be capable of providing the offset τ as a negative value. Such a negative offset −τ can be provided by delaying the light emission timing signal S1jgenerated as a preceding signal.

In order to provide the negative offset −τ, the gating camera20is provided with a delay unit28configured as an external component, or includes such a delay unit28as a built-in component. In the normal image capture operation, the delay unit28has no effect on the image capture. That is to say, the light emission timing signal S1generated by the controller26is supplied to the illumination apparatus22as it is (path i). On the other hand, in the testing process, the light emission timing signal S1generated by the controller26is delayed by the time τ that corresponds to the offset by means of the delay unit28. The delayed light emission timing signal S1′ is supplied to the illumination apparatus22(path ii).

The above is the configuration of the gating camera20. Next, description will be made regarding the test of the gating camera20.

FIG.49is a diagram showing a layout of the gating camera20and the reference reflector REF in the testing process. In this example, testing is made regarding whether or not an image has been captured normally for the j-th range RNGj. The reference reflector REF is arranged in the i-th range RNGithat is at a near distance as compared with the j-th range RNGj. The controller26of the gating camera20is set to a state in which an image can be captured for the j-th range RNGj. That is to say, the focus is set to the j-th range RNGj. In other words, the controller26outputs the light emission timing signal S1jand the image capture timing signal S2j. In this case, the negative offset −τ is applied to the time difference Δtjbetween the light emission timing signal S1jand the image capture timing signal S2j. The offset amount t is represented by τ=(dMINj−dMINi)/c×2.

FIG.50is a diagram for explaining the testing of the gating camera20shown inFIG.46. The time difference between the light emission timing signal S1; and the image capture timing signal S2; is represented by Δtj. The light emission timing signal S1jis delayed by t by means of the delay unit28. With this, the time difference between the delayed light emission timing signal S1j′ and the image capture timing signal Ω; is represented by Δtj−τ.

In the upper diagram shown inFIG.50, the line of alternately long and short dashes indicates the light beam when the illumination apparatus22emits light according to the light emission timing signal S1j. This light is reflected in the j-th range RNGj, and an image of the reflected light is captured according to the image capture timing signal S2j. In the testing process, the illumination apparatus22emits light according to the delayed light emission timing signal S1j′ instead of the light emission timing signal S1j. In this case, the light beam is indicated by the solid line. This light beam is reflected by an object (reference reflector REF) in the range RNGi, and an image of the reflected light is captured according to the image capture timing signal S2j.

As a result, when the slice image IMGjthus captured includes an image of the reference reflector REF, judgment is made that the gating camera20operates normally. Otherwise, judgement is made that the gating camera20operates abnormally. The above is the testing of the gating camera20.

With the gating camera20, the negative offset −τ can be applied to the time difference Δtjbetween the light emission timing signal S1jand the image capture timing signal S2j. This arrangement is capable of capturing an image of the reference object REF that exists in the range RNGithat is at a near distance as compared with the focus range RNGj.

FIG.51is a diagram showing a first example configuration of the gating camera20shown inFIG.46. The gating camera20includes the delay unit28as a built-in component. For example, the gating camera20includes selectors30and32respectively configured as an upstream stage and a downstream stage of the delay unit28. In the normal image capture operation (non-testing process), the delay unit28is bypassed by the selectors30and32so as to directly supply the light emission timing signal S1to the illumination apparatus22. In the testing process, the light emission timing signal S1is transmitted via the delay unit28by means of the selectors30and32, thereby supplying the delayed light emission timing signal S1′ to the illumination apparatus22.

FIG.52is a diagram showing a second example configuration of the gating camera20shown inFIG.46. The point of difference between the gating cameras20shown inFIGS.52and51is that, in the gating camera20shown inFIG.52, the delay unit28is detachably mounted on the gating camera20.

Embodiment 5-2

FIG.53is a block diagram showing a gating camera20A according to an embodiment 5-2. In the embodiment 5-2, the gating camera20A is configured such that, in the testing process, a positive offset can be applied to the time difference Δt between the light emission timing signal S1and the image capture timing signal S2. The positive offset can be applied by delaying the image capture timing signal S2generated as a subsequent timing signal.

In order to provide the positive offset τ, the gating camera20A is provided with a delay unit28configured as an external component, or includes such a delay unit28as a built-in component. In the normal image capture operation, the delay unit28has no effect on the image capture. That is to say, the image capture timing signal S2generated by the controller26is supplied to the image sensor24as it is (path i). On the other hand, in the testing process, the image capture timing signal S2generated by the controller26is delayed by the time τ that corresponds to the offset by means of the delay unit34. The delayed image capture timing signal S2′ is supplied to the image sensor24(path ii).

Next, description will be made regarding the testing of the gating camera20A.FIG.54is a diagram showing a layout of the gating camera20A and the reference reflector REF in the testing process. In this example, testing is made regarding whether or not an image has been captured normally for the i-th range RNGi. The reference reflector REF is arranged in the j-th range RNGjthat is at a far distance as compared with the i-th range RNGi. The controller26of the gating camera20A is set to a state in which an image can be captured for the i-th range RNGi. That is to say, the controller26outputs the light emission timing signal S1iand the image capture timing signal S2i. In this case, the positive offset +τ is applied to the time difference Δtibetween the light emission timing signal S1iand the image capture timing signal S2i. The offset amount t is represented by τ=(dMINj−dMINi)/c×2.

FIG.55is a diagram for explaining the testing of the gating camera20A shown inFIG.53. The time difference between the light emission timing signal S1iand the image capture timing signal S2iis represented by Δti. The image capture signal S2iis delayed by t by means of the delay unit28. With this, the time difference between the light emission timing signal S1iand the delayed image capture timing signal S2i′ is represented by Δti+τ.

In the testing process, the image capture is executed based on the delayed image capture timing signal S2i′ instead of the image capture timing signal S2i. As a result, when the slice image IMGithus captured includes an image of the reference reflector REF, judgment is made that the gating camera20A operates normally. Otherwise, judgement is made that the gating camera20A operates abnormally.

With the gating camera20A shown inFIG.53, the positive offset +τ can be applied to the time difference Δtibetween the light emission timing signal S1iand the image capture timing signal S2i. This arrangement is capable of capturing an image of the reference object REF that exists in the range RNGjthat is at a far distance as compared with the focus range RNGi.

FIG.56is a diagram showing a first example configuration of the gating camera20A shown inFIG.53. The gating camera20A includes the delay unit34as a built-in component. For example, the gating camera20A includes selectors36and38respectively configured as an upstream stage and a downstream stage of the delay unit34. In the normal image capture operation (non-testing process), the delay unit34is bypassed by the selectors36and38so as to directly supply the image capture timing signal S2to the image sensor24. In the testing process, the image capture timing signal S2is transmitted via the delay unit34by means of the selectors36and38, thereby supplying the delayed image capture timing signal S2′ to the image sensor24.

FIG.57is a diagram showing a second example configuration of the gating camera20A shown inFIG.53. The point of difference between the gating cameras20A shown inFIGS.57and56is that, in the gating camera20A shown inFIG.57, the delay unit34is detachably mounted on the gating camera20A.

Embodiment 5-3

FIG.58is a diagram showing a gating camera20B according to an embodiment 5-3. The gating camera20B includes an illumination apparatus22, an image sensor24, and a controller26. The gating camera20B is configured such that, in the testing process, the light emission timing signal S1can be output via an external delay unit60to an external reference light source62. With the embodiment 5-3, in the testing process, the illumination apparatus22is not used. In this state, testing is executed regarding whether or not the image sensor24and the controller26operate normally. The reference light source62is arranged such that it faces the gating camera20B. The reference light source62emits light according to the delayed light emission timing signal S1′. The above is the configuration of the gating camera20B.

The gating camera20B provides substantially the same operation as that shown inFIG.46. The point of difference from that shown inFIG.46is that the external reference light source62is employed instead of the reference reflector REF. Furthermore, in the testing process, the illumination apparatus22is set to a non-emission mode. The reference light source62is arranged in the i-th range RNGi. The j-th range RNGjis the focus of the gating camera20B.

InFIG.58, the light path length of the emitted light L3from the reference light source62is half the light path length in a case in which the probe light L1is emitted from the illumination apparatus22, and reflected light L2returns by reflection. Accordingly, the delay amount t of the delay unit60is represented by τ=(dMINj−dMINi)/c.

As a result of image capture, when the slice image IMGjthus captured includes an image of the emitted light from the reference light source62, judgment is made that the gating camera20B operates normally.

It should be noted that the same testing system can be built using the gating camera20shown inFIG.52as the system using the gating camera20B shown inFIG.58.

Embodiment 5-4

FIG.59is a diagram showing a gating camera20C according to an embodiment 5-4. The gating camera20C includes an illumination apparatus22, an image sensor24, and a controller26. The gating camera20C is configured such that, in the testing process, the light emission timing signal S2can be output via an external delay unit64to an external reference camera66. With the embodiment 5-4, in the testing process, the image sensor24is not used. In this state, testing is executed regarding whether or not the illumination apparatus22and the controller26operate normally. The reference camera66is arranged such that it faces the gating camera20C. The reference camera66captures an image of the emitted light L4from the illumination apparatus22based on the delayed image capture timing signal S2′. The above is the configuration of the gating camera20C.

Description will be made regarding the testing of the gating camera20C. In the testing process, the image sensor24is not used. The reference camera66is arranged in the j-th range RNGj. The gating camera20C is set to a state in which it is able to measure the i-th range RNGi.

InFIG.59, the light path length of the emitted light L4from the illumination apparatus22is half the light path length in a case in which, as shown inFIG.46, the probe light L1is emitted from the illumination apparatus22, and reflected light L2returns by reflection. Accordingly, the delay amount τ of the delay unit64is represented by τ=(dMINj−dMINi)/c.

As a result of image capture, when the output image of the reference camera66includes an image of the emitted light from the illumination apparatus22, judgment is made that the gating camera20C operates normally.

It should be noted that the same testing system can be built using the gating camera20A shown inFIG.57as the system using the gating camera20C shown inFIG.59.

Usage

FIG.60is a block diagram showing the automotive lamp200provided with the object detection system210. The automotive lamp200forms the lamp system310together with the in-vehicle ECU304. The automotive lamp200includes a light source202, a lighting circuit204, and an optical system206. Furthermore, the automotive lamp200is provided with the object detection system210. The object detection system210includes the gating camera20described above and a processing device40.

The processing device40is configured to identify the kind of an object based on the multiple slice images IMG1through IMGNthat correspond to the range RNG1through RNGNgenerated by the gating camera20. The processing device40is provided with the classifier42implemented based on a prediction model generated by machine learning. The algorithm employed by the classifier42is not restricted in particular. Examples of the algorithms that can be employed include You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), Region-based Convolutional Neural Network (R-CNN), Spatial Pyramid Pooling (SPPnet), Faster R-CNN, Deconvolution-SSD (DSSD), Mask R-CNN, etc. Also, other algorithms that will be developed in the future may be employed.

The processing device40may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, or the like, and a software program to be executed by the processor (hardware component). Also, the processing device40may be configured as a combination of multiple processors. Alternatively, the processing device40may be configured as a hardware component alone.

Also, the information with respect to the object OBJ detected by the processing device40may be used to support the light distribution control operation of the automotive lamp200. Specifically, a lamp ECU208generates a suitable light distribution pattern based on the information with respect to the kind of the object OBJ and the position thereof generated by the processing device40. The lighting circuit204and the optical system206operate so as to provide the light distribution pattern generated by the lamp ECU208.

Also, the information with respect to the object OBJ detected by the processing device40may be transmitted to the in-vehicle ECU304. The in-vehicle ECU may support autonomous driving based on the information thus transmitted.

Embodiment 6

First, description will be made regarding problems relating to an embodiment 6. As a result of investigating an arrangement in which multiple slice images captured by a gating camera are combined so as to generate a single image, the present inventors have recognized the following problems.

FIGS.61A and61Bare diagrams for explaining the problems relating to image combining.FIG.61Ashows multiple slice images IMG1through IMGN. Here, for ease of understanding, description will be made assuming that the images are each configured as a one-dimensional image plotted with the pixel position as the horizontal axis and with the pixel value as the vertical axis. In this example, description will be made assuming that an object exists only in the i-th range. Accordingly, the i-th slice image IMGiincludes an image of the object. If light (disturbance light) that differs from the probe light is input to the infrared camera, this light is recorded as a noise component. The noise component is included in all the slice images.FIG.61Bshows an image obtained by combining the multiple slice images by simple addition. In a case in which the slice images are combined in such a simple manner, this leads to drastic degradation of the S/N ratio.

It should be noted that such problems described above are by no means within the scope of common and general knowledge of those skilled in this art. Furthermore, it can be said that the knowledge described above has been uniquely recognized by the present inventors.

The present invention relating to an embodiment 6 has been made in view of such a situation. Accordingly, it is an exemplary purpose of an embodiment of the present invention to provide an image capture apparatus that is capable of generating a combined image with improved image quality.

Overview of Embodiment 6

An embodiment disclosed in the present specification relates to an image capture apparatus. The image capture apparatus incudes: a gating camera structured to divide a field of view in a depth direction into multiple ranges, to capture an image for each range while changing a time difference between light emission and exposure, and to generate multiple slice images that correspond to the multiple ranges; and an image processing device structured to combine the multiple slice images so as to generate a combined image. The image processing device is structured to detect a no-object-existing region in which no object exists for each of the multiple slice images, to apply noise reduction processing to the no-object-existing region, and to combine the multiple slice images thus subjected to the noise reduction processing.

With such an arrangement in which multiple slice images are combined after noise is removed from the no-object-existing region, this provides the combined image with improved image quality.

Also, when the image processing device detects an object-existing region in which an object exists in a given slice image, the image processing device may judge that a region in a different slice image that overlaps the object-existing region thus detected is a no-object-existing region. In principle, the gating camera is configured to detect the reflected light from an object positioned at the nearest distance in the depth direction. Accordingly, in a case in which the ranges are designed such that they do not overlap in the depth direction, when reflection is detected from an object in a region of a given slice image, an image detected in the same region in a different slice image is regarded as a noise image. This allows such a noise image to be removed.

Also, the image capture apparatus may set pixel values of multiple pixels included in the no-object-existing region to zero.

Also, when multiple pixels having pixel values that are larger than a threshold value consecutively exist, the image processing device may judge a region including these pixels to be a region in which an object exists.

Description will be made below regarding an embodiment 6 with reference to the drawings.

Embodiment 6-1

FIG.62is a block diagram showing an image capture apparatus10according to an embodiment 6-1. The image capture apparatus10includes a gating camera20and an image processing device30. For example, the image capture apparatus10is mounted on a vehicle such as an automobile, motorcycle, or the like. The image capture apparatus10captures an image of an object OBJ that exists in the vicinity of the vehicle.

The image capture apparatus10mainly include a gating camera20and an image processing device30. The gating camera20includes an illumination apparatus22, an image sensor24, and a controller26. The gating camera20captures images for a plurality of N (N≥2) ranges RNG1through RNGNdivided in the depth direction. The ranges may be designed such that adjacent ranges overlap at their boundaries in the depth direction.

The illumination apparatus22emits probe light L1in front of the vehicle in synchronization with a light emission timing signal S1supplied from the controller26. As the probe light L1, infrared light is preferably employed. However, the present invention is not restricted to such an arrangement. Also, visible light having a predetermined wavelength may be employed. Also, ultraviolet light may be employed.

The image sensor24is configured to support exposure control in synchronization with an image capture timing signal S2supplied from the controller26, and to output a slice image IMG. The image sensor24is sensitive to the same wavelength as that of the probe light L1. The image sensor24captures an image of reflected light (returned light) L2reflected by the object OBJ.

The controller26changes the light emission timing signal S1and the image capture timing signal S2for each range RNG, so as to change the difference in the timing between the light emission operation of the illumination apparatus22and the exposure operation of the image sensor24. The gating camera20generates the slice images IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN. Accordingly, in principle, the i-th slice image IMGiincludes only an image of an object included in the corresponding range RNGi.

FIG.63is a diagram for explaining the basic operation of the gating camera20.FIG.63shows the operation when the i-th range RNGiis measured. The illumination apparatus22emits light during a light-emitting period τi from the time points t0to t1in synchronization with the light emission timing signal S1. In the upper diagram, a light beam diagram is shown with the horizontal axis as time and with the vertical axis as distance. The distance between the gating camera20and the near-distance boundary of the range RNGiis represented by dMINi. The distance between the gating camera20and the far-distance boundary of the range RNGiis represented by dMAXi.

The round-trip time TMINi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMINi, up to the return of the reflected light to the image sensor24, is represented by TMINi=2×dMINi/c. Here, c represents the speed of light.

Similarly, the round-trip time TMAXi, which is a period from the departure of light from the illumination apparatus22at a given time point, to the arrival of the light at the distance dMAXi, up to the return of the reflected light to the image sensor24, is represented by TMAXi=2×dMAXi/c.

When only an image of an object OBJ included in the range RNGiis to be captured, the controller26generates the image capture timing signal S2so as to start the exposure at the time point t2=t0+TMINi, and so as to end the exposure at the time point t3=t1+TMAXi. This is a single exposure operation.

When an image is captured for the i-th range RNGi, the exposure may be executed multiple times. In this case, preferably, the controller26may repeatedly execute the above-described exposure operation multiple times with a predetermined period τ2.

FIGS.64A and64Bare diagrams for explaining an image generated by the gating camera20.FIG.64Ashows an example in which an object (pedestrian) OBJ2exists in the range RNG2, and an object (vehicle) OBJ3exists in the range RNG3.FIG.64Bshows multiple slice images IMG1through IMG3acquired in the situation shown inFIG.64A. When the slice image IMG1is captured, the image sensor is exposed by only the reflected light from the range RNG1. Accordingly, the slice image IMG1includes no object image.

When the image IMG2is captured, the image sensor is exposed by only the reflected light from the range RNG2. Accordingly, the image IMG2includes only the object image OBJ2. Similarly, when the image IMG3is captured, the image sensor is exposed by only the reflected light from the range RNG3. Accordingly, the image IMG3includes only the object image OBJ3. As described above, with the gating camera20, this arrangement is capable of capturing object images in the form of separate images for the respective ranges.

Returning toFIG.61, the processing device30combines the multiple slice images IMG1through IMGNthat correspond to the multiple ranges RNG1through RNGN, which are captured by the gating camera20, so as to generate a combined image IMGc.

The image processing device30judges a region (no-object-existing region) in which no object exists for each of the multiple slice images IMG1through IMGN. For example, for each slice image IMG #, when pixels having a pixel value that is larger than a predetermined threshold value consecutively exist, the image processing device30may judge a region including such pixels to be an object-existing region. Furthermore, the image processing device30judges the regions other than the object-existing region to be no-object-existing regions. Alternatively, the image processing device30may execute edge extraction for each slice image IMG#. With this, the image processing device30may judge that a region surrounded by an edge is an object-existing region. Conversely, the image processing device30may judge the area outside the edge to be a no-object-existing region.

The image processing device30applies noise reduction processing to the no-object-existing region for each of the multiple slice images IMG1through IMGN.

For example, as the noise removal, the pixel values of the no-object-existing region may be set to zero for each slice image IMGj. Alternatively, the image processing device30may multiply the pixel values of the no-object-existing region by a factor that is smaller than 1. The slice image thus subjected to noise removal is represented by IMGaj.

Subsequently, the image processing device30combines the multiple slice images IMGa1through IMGaNthus subjected to noise removal so as to generate a combined image IMGc.

FIG.65is a function block diagram showing the image processing device30. The image processing device30includes an object detection unit32, a noise reducing unit34, and a combining unit36. The object detection unit32detects the object-existing region and the no-object-existing region for each of the multiple slice images IMG1through IMGN. The object detection unit32supplies information that indicates the no-object-existing region for each slice image to the noise reducing unit34. The noise reducing unit34applies noise reduction processing to the no-object-existing regions for each of the multiple slice images IMG1through IMGN. The combining unit36combines the slice images IMGa1through IMGaNthus subjected to the noise reduction processing, so as to generate a combined image.

The image processing device30may be configured as a hardware component such as a field programmable gate array (FPGA), application specified IC (ASIC), or the like. Alternatively, the image processing device30may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, or the like, and a software program to be executed by the processor (hardware component). Also, the image processing device30may be configured as a combination of multiple processors.

The above is the configuration of the image capture apparatus10. Next, description will be made regarding the image capture apparatus10.FIG.66is a perspective view showing an image capture situation. Description will be made below assuming that the number N of the ranges is six. Description will be made regarding an example in which an object (human) OBJ2and an object (automobile) OBJ6exist in the ranges RNG2and RNG6, respectively.

FIG.67A through67Care diagrams for explaining the combining processing by the image processing apparatus30.FIG.67Ashows the multiple slice images IMG1through IMG6acquired in the situation shown inFIG.66. Here, “N1” indicates noise due to extraneous light. “N2” indicates noise due to an object included in the adjacent range.

Directing attention to the slice image IMG2, the slice image IMG2includes a clear object image of the object OBJ2. The region including the object image OBJ2is judged to be an object-existing region A1. Furthermore, the area outside the object-existing region A1is judged to be a no-object-existing region. In the same manner, directing attention to the slice image IMG6, the slice image IMG6includes a clear object image of the object OBJ6. The region including the object image OBJ6is judged to be an object-existing region A2. Furthermore, the area outside the object-existing region A2is judged to be a no-object-existing region.

Directing attention to the slice images IMG3and IMG4, there is no region in which the pixel values thereof exceed a threshold value. Furthermore, no edge is detected. Accordingly, judgment is made that the entire region is a no-object-existing region.

Directing attention to the slice image IMG1, the slice image IMG1includes an image of the noise N2that occurs due to the object OBJ2included in the adjacent range. In this example, the pixel values of the noise N2region are sufficiently small. Furthermore, no edge is detected. Accordingly, the noise N2region is judged to be a no-object-existing region. The same can be said of the slice image IMG5. That is to say, the noise N2region is judged to be a no-object-existing region.

FIG.67Bshows the slice images IMGa1through IMGa6subjected to the noise reduction processing. The pixel values of the no-object-existing regions are set to zero for each slice image IMGa, thereby removing the noise components N1and N2.

FIG.67Cshows the combined image IMGc obtained by combining the slice images IMGa1through IMGa6thus subjected to the noise reduction. As shown inFIG.67C, with the image capture apparatus10, the noise N1due to the extraneous light can be removed. Furthermore, this also allows the noise N2due to an object in the adjacent range to be removed.

Referring toFIGS.68A through68C, description will be made regarding problems that can occur in the processing provided by the embodiment 6-1.FIGS.68A through68Care diagrams showing another example of the operation of the image capture apparatus10according to the embodiment 6-1.

FIG.68Ashows the slice images IMG1through IMG6acquired in the situation shown inFIG.66as with the example shown inFIG.67A. Directing attention to the slice image IMG1, the slice image IMG1includes the noise N2due to the object OBJ2in the adjacent range that is stronger than that inFIG.67A. Description will be made assuming that the pixel values of the noise N2region are large to an extent that allows an edge to be detected. In this case, the region including the noise N2is falsely judged to be an object-existing region A3. The same can be said of the slice image IMG5. That is to say, the region including the noise N2is falsely judged to be an object-existing region A4.

As a result, inFIG.68B, the noise N2remains in each of the slice images IMGa1and IMG5thus subjected to the noise reduction processing. As a result, in some cases, the combined image shown inFIG.68Chas a problem of the occurrence of a blurred edge in an object image or a problem of degraded image contrast. In order to solve this problem, there is a need to carefully determine the threshold value to be used for detection of an object-existing region. In a case in which the threshold value is designed to be an excessively large value, such an arrangement has the potential to lead to false judgment of a noise region although an actual object exists.

Embodiment 6-2

With an embodiment 6-2, the problems that occur in the embodiment 6-1 are solved. The image capture apparatus10has the same overall configuration as that shown inFIG.62. However, there is a difference in the operation of the image processing device30between the embodiments 6-1 and 6-2.

In the embodiment 6-2, when an object-existing region is detected in a given slice image IMGi, the same region in a different slice image IMGj(j≠i) that overlaps the object-existing region thus detected is determined to be a no-object-existing region.

FIGS.69A through69Care diagrams for explaining combining processing according to the embodiment 6-2.FIG.69Ais the same asFIG.68A. Referring to the slice image IMG1shown inFIG.69A, a strong edge is detected for the noise N2. However, an object-existing region A1is detected in a different slice image IMG2. Accordingly, correct judgement can be made for the slice image IMG1that the region thereof including the noise N2that overlaps the object-existing region A1is a no-object-existing region. Also, in the same manner, correct judgment can be made for the slice image IMG5that the region thereof including the noise N2is a no-object-existing region.

As a result, as shown inFIG.69B, the effects of the noise N2are removed from each of the slice images IMG1and IMG5thus subjected to the noise correction. As shown inFIG.69C, this allows the combined image IMGc to be acquired with high image quality.

In the embodiment 6-1, the slice images are each processed independently so as to judge the object-existing regions and the no-object-existing regions. In contrast, in the embodiment 6-2, the object-existing regions and the no-object-existing regions are judged with reference to the multiple slice images, i.e., based on the relation between the multiple slice images. This arrangement is capable of preventing false judgement. In particular, the gating camera has a characteristic of detecting only the reflected light from an object at the nearest distance. Accordingly, it is not possible for two (or three or more) slice images to include the same image of an object at the same pixel position or in the same region. In the embodiment 6-2, with such an arrangement using this feature, the noise and the object can be distinguished in a sure manner.

It should be noted that, when two slice images have an overlapping object-existing region detected at the same time, the object-existing region including larger pixel values may be extracted as a correct object-existing region from among the two slice images. On the other hand, the object-existing region detected in the other slice images may be judged to be a no-object existing region.

Embodiment 6-3

Description has been made in the embodiments 6-1 and 6-2 regarding an arrangement in which, after the object-existing regions and the no-object-existing regions are judged, the noise removal processing is executed. However, the present invention is not restricted to such an arrangement. Also, such processing may be executed for each pixel.

That is to say, the pixel values at the same pixel position are compared for all the slice images IMG1through IMGN. Furthermore, the pixel having the largest pixel value from among the slice images IMG1through IMGNis judged to be an effective pixel. The other pixels are each judged to be noise pixels. The pixel values of the noise pixels are each set to zero, or are each multiplied by a coefficient that is smaller than 1 so as to reduce the pixel value. After this processing is executed for all the pixels, the multiple slice images IMG1through IMGNmay be combined.

Usage

FIG.70is a block diagram showing an object identification system400provided with an image capture apparatus. The image identification system400includes an image capture apparatus410and a processing device420. The image capture apparatus410corresponds to the image capture apparatus10described in the embodiments 6-1 and 6-2. The image capture apparatus410generates a combined image IMGc.

The processing device420is configured to be capable of identifying the position and the kind (category, class) of an object based on the combined image IMGc. The processing device420may include a classifier422. The processing device420may be configured as a combination of a processor (hardware component) such as a Central Processing Unit (CPU), Micro Processing Unit (MPU), microcontroller, or the like, and a software program to be executed by the processor (hardware component) such as a microcontroller. Also, the processing device420may be configured as a combination of multiple processors. Alternatively, the processing device420may be configured as a hardware component alone.

The classifier422may be implemented based on a prediction model generated by machine learning. The classifier422judges the kind (category or class) of an object included in an input image. The algorithm employed by the classifier422is not restricted in particular. Examples of the algorithms that can be employed include You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), Region-based Convolutional Neural Network (R-CNN), Spatial Pyramid Pooling (SPPnet), Faster R-CNN, Deconvolution-SSD (DSSD), Mask R-CNN, etc. Also, other algorithms that will be developed in the future may be employed. The processing device420and the image processing apparatus30of the image capture apparatus410may be implemented on the same processor or the same FPGA.

Also, the output of the object identification system400may be used for the light distribution control of the automotive lamp, Also, the output of the object identification system400may be transmitted to the in-vehicle ECU so as to support autonomous driving control.

FIG.71is a block diagram showing a display system500provided with an image capture apparatus. The display system500includes an image capture apparatus510and a display520. The image capture apparatus510corresponds to the image capture apparatus10described in the embodiments 6-1 and 6-2. The image capture apparatus510generates a combined image IMGc with improved image quality. The display520displays the combined image IMGc. The display system500may be configured as a digital mirror. Also, the display system500may be configured as a front view monitor or a rear view monitor that covers a blind spot.

FIG.72is a block diagram showing an automobile300provided with the object identification system400. The automobile300is provided with headlamps302L and302R. All or part of the components of the object identification system400are built into at least one from among the headlamps302L and302R. Each headlamp302is positioned at a frontmost end of the vehicle body, which is most advantageous as a position where the gating camera20is to be installed for detecting an object in the vicinity.

FIG.73is a block diagram showing the automotive lamp200provided with the object detection system210. The automotive lamp200forms the lamp system310together with the in-vehicle ECU304. The automotive lamp200includes a light source202, a lighting circuit204, and an optical system206. Furthermore, the automotive lamp200is provided with the object detection system210. The object detection system210corresponds to the object identification system400described above. The object detection system210includes the gating camera20, the image processing device30, and the processing device40.

Also, the information with respect to the object OBJ detected by the processing device40may be used to support the light distribution control operation of the automotive lamp200. Specifically, a lamp ECU208generates a suitable light distribution pattern based on the information with respect to the kind of the object OBJ and the position thereof generated by the processing device40. The lighting circuit204and the optical system206operate so as to provide the light distribution pattern generated by the lamp ECU208.

Also, the information with respect to the object OBJ detected by the processing device40may be transmitted to the in-vehicle ECU304. The in-vehicle ECU may support autonomous driving based on the information thus transmitted. The function of the processing device40for supporting the object detection may be implemented on the in-vehicle ECU304.

Description has been made regarding the present invention with reference to the embodiments using specific terms. However, the above-described embodiments show only an aspect of the mechanisms and applications of the present invention. Rather, various modifications and various changes in the layout can be made without departing from the spirit and scope of the present invention defined in appended claims.

Clauses Describing Features of the Disclosure

Clause 18. An object identification system comprising:a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate a plurality of slice images that correspond to the plurality of ranges; anda processing device structured to have a plurality of first correction characteristics defined corresponding to the plurality of ranges, to correct each of the plurality of slice images using the corresponding first correction characteristics, and to judge a kind of an object based on the plurality of slice images thus corrected.

Clause 19. The object identification system according to Clause 18, wherein the first correction characteristics that correspond to each range are changed according to a measurement environment.

Clause 20. The object identification system according to Clause 18, wherein the processing device holds a plurality of second correction characteristics defined corresponding to the plurality of ranges,and wherein the processing device corrects each of the plurality of slice images using the corresponding second correction characteristics, and combines the plurality of slice images thus corrected.

Clause 21. The object identification system according to Clause 20, wherein an image obtained by combining the plurality of corrected slice images is displayed on a display.

Clause 22. An automobile comprising the object identification system according to Clause 18.

Clause 23. An automotive lamp comprising the object identification system according to Clause 18.

Clause 24. A processing device to be used so as to form an object identification system together with a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate a plurality of slice images that correspond to the plurality of ranges, the processing device comprising:a first correction unit structured to have a plurality of first correction characteristics defined corresponding to the plurality of ranges, and to correct each of the plurality of slice images using the corresponding first correction characteristics; anda classifier structured to judge a kind of an object based on the plurality of corrected slice images.

Clause 25. The processing device according to Clause 24, further comprising:a second correction unit structured to have a plurality of second correction characteristics defined corresponding to the plurality of ranges, and to correct each of the plurality of slice images using the corresponding second correction characteristics; anda combining unit structured to combine the plurality of slice images corrected by the second correction unit so as to generate an image to be displayed on a display.

Clause 26. An image display system comprising a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate a plurality of slice images that correspond to the plurality of ranges; anda processing device structured to combine the plurality of slice images so as to generate a combined image to be displayed on a display.

Clause 27. The image display system according to Clause 26, wherein the processing device is structured to have a plurality of correction characteristics defined corresponding to the plurality of ranges, to correct each of the plurality of slice images using the corresponding correction characteristics, and to combine the plurality of corrected slice images.

Clause 28. The image display system according to Clause 27, wherein the correction characteristics that correspond to each range are changed according to a measurement environment.

Clause 29. An automobile comprising the image display system according to Clause 26.

Clause 30. A processing device to be used so as to form an image display system together with a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to capture an image for each range while changing a time difference between light emission and exposure, so as to generate a plurality of slice images that correspond to the plurality of ranges, the processing device comprising:a combining unit structured to combine the plurality of slice images so as to generate an image to be displayed on a display.

Clause 31. The processing device according to Clause 30, further comprising a correction unit structured to have a plurality of correction characteristics defined corresponding to the plurality of ranges, and to correct each of the plurality of slice images using the corresponding correction characteristics,wherein the combining unit is structured to combine the plurality of slice images corrected by the correction unit.

Clause 32. A gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to generate a plurality of slice images that correspond to the plurality of ranges, the gating camera comprising:an illumination apparatus structured to emit probe light according to a light emission timing signal;an image sensor structured to capture an image of reflected light according to an image capture timing signal; anda controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range,wherein, in a testing process, the gating camera is structured to be capable of applying an offset to the time difference between the light emission timing signal and the image capture timing signal.

Clause 33. The gating camera according to Clause 32, structured to allow an external delay unit to be connected on a path of the light emission timing signal.

Clause 34. The gating camera according to Clause 32, further comprising a delay unit to be inserted on a signal path of the light emission timing signal in a testing process.

Clause 35. The gating camera according to Clause 32, structured to allow an external delay unit to be connected on a signal path of the image capture timing signal.

Clause 36. The gating camera according to Clause 32, further comprising a delay unit to be inserted on a signal path of the image capture timing signal.

Clause 37. A gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to generate a plurality of slice images that correspond to the plurality of ranges, the gating camera comprising:an illumination apparatus structured to emit probe light according to a light emission timing signal;an image sensor structured to capture an image of reflected light according to an image capture timing signal; anda controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range,wherein, in a testing process, the gating camera is structured to be capable of outputting the light emission timing signal to an external reference light source via an external delay unit.

Clause 38. A gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to generate a plurality of slice images that correspond to the plurality of ranges, the gating camera comprising:an illumination apparatus structured to emit probe light according to a light emission timing signal;an image sensor structured to capture an image of reflected light according to an image capture timing signal; anda controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range,wherein, in a testing process, the gating camera is structured to be capable of outputting the image capture timing signal to an external image sensor via an external delay unit.

Clause 39. A testing method for a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to generate a plurality of slice images that correspond to the plurality of ranges,wherein the gating camera comprises:an illumination apparatus structured to emit probe light according to a light emission timing signal;an image sensor structured to capture an image of reflected light according to an image capture timing signal; anda controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range,and wherein the testing method comprises:arranging a reflector in the i-th (i represents an integer) range;operating the controller in a state in which an image of the j-th (j>i) range can be captured; andapply a predetermined delay to the light emission timing signal.

40. A testing method for a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, and to generate a plurality of slice images that correspond to the plurality of ranges,wherein the gating camera comprises:an illumination apparatus structured to emit probe light according to a light emission timing signal;an image sensor structured to capture an image of reflected light according to an image capture timing signal; anda controller structured to generate the light emission timing signal and the image capture timing signal with a time difference defined for each range,and wherein the testing method comprises:arranging a reflector in the j-th (j represents an integer) range;operating the controller in a state in which an image of the i-th (I<j) range can be captured; andapply a predetermined delay to the image capture timing signal.

Clause 41. An image capture apparatus comprising:a gating camera structured to divide a field of view in a depth direction into a plurality of ranges, to capture an image for each range while changing a time difference between light emission and exposure, and to generate a plurality of slice images that correspond to the plurality of ranges; andan image processing device structured to combine the plurality of slice images so as to generate a combined image,wherein the image processing device is structured to detect a no-object-existing region in which no object exists for each of the plurality of slice images, to apply noise reduction processing to the no-object-existing region, and to combine the plurality of slice images thus subjected to the noise reduction processing.

Clause 42. The image capture apparatus according to Clause 41, wherein, when the image processing device detects an object-existing region in which an object exists in a given slice image, the image processing device judges that a region in a different slice image that overlaps the object-existing region thus detected is a no-object-existing region.

Clause 43. The image capture apparatus according to Clause 41, structured to set pixel values of a plurality of pixels included in the no-object-existing region to zero.

Clause 44. The image capture apparatus according to Clause 41, wherein, when a plurality of pixels having pixel values that are larger than a threshold value consecutively exist, the image processing device judges a region including these pixels to be a region in which an object exists.

Clause 45. An automobile comprising:the image capture apparatus according to Clause 41;and a classifier structured to judge a king of an object included in a combined image generated by the image capture apparatus based on the combined image.

Clause 46. An automobile comprising:the image capture apparatus according to Clause 41;and a display apparatus structured to display a combined image generated by the image capture apparatus.

Clause 47. An image processing device structured to execute:processing for detecting a no-object-existing region in which no object exists for each of a plurality of slice images captured by a gating camera;processing for noise reduction in the no-object-existing region; andprocessing for combining the plurality of slice images thus subjected to noise reduction.

Clause 48. The image processing device according to Clause 47, wherein, when the image processing device detects an object-existing region in which an object exists in a given slice image, the image processing device judges that a region in a different slice image that overlaps the object-existing region thus detected is a no-object-existing region.