Patent ID: 12217473

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of an object detection device, an object detection method, and a storage medium of the present invention will be described with reference to the drawings. The object detection device is, for example, mounted on a mobile object. The mobile object is, for example, a four-wheeled vehicle, a two-wheeled vehicle, a micromobility, or a robot that moves by itself, or a portable device such as a smartphone placed on a mobile object that moves by itself or moving by being carried by a person. In the following description, it is assumed that the mobile object is a four-wheeled vehicle, and the mobile object is referred to as a “vehicle” for the following description. The object detection device is not limited to a device mounted on the mobile object, and may perform processing to be described below based on a captured image captured by a camera for fixed point observation or a camera of a smartphone.

FIG.1is a diagram which shows a configuration of an object detection device100and an example of peripheral devices. The object detection device100communicates with a camera10, a traveling control device200, a notification device210, and the like.

The camera10is attached to a back surface of a windshield of a vehicle, captures at least an image in which a road is captured in a traveling direction of the vehicle, and outputs the captured image to the object detection device100. A sensor fusion device or the like may be interposed between the camera10and the object detection device100, but description thereof will be omitted.

The traveling control device200is, for example, an automatic driving control device for causing a vehicle to autonomously travel, or a driving support device for performing inter-vehicle distance control, automatic brake control, automatic lane change control, and the like. The notification device210is a speaker, a vibrator, a light emitting device, a display device, or the like for outputting information to an occupant of the vehicle.

The object detection device100includes, for example, an acquirer110, a low-resolution image generator120, a grid definition unit140, an extractor150, and a high-resolution processor170. The extractor150includes a feature amount difference calculator152, an aggregation unit154, a first addition unit156, a second addition unit158, and a point of interest extractor160. These components are realized by, for example, a hardware processor such as a central processor (CPU) executing a program (software). Some or all of these components may be realized by hardware (circuit part; including circuitry) such as large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processor (GPU), or may be realized by software and hardware in cooperation. A program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a hard disk drive (HDD) or a flash memory, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or a CD-ROM and installed by the storage medium being mounted on a drive unit.

FIG.2is a diagram which schematically shows functions of each part of the object detection device100. In the following description, each part of the object detection device100will be described with reference toFIG.2. The acquirer110acquires a captured image from the camera10. The acquirer110stores the acquired captured image (data thereof) in a working memory such as a random access memory (RAM).

The low-resolution image generator120performs thinning processing on the captured image to generate a low-resolution image whose image quality is lower than that of the captured image. The low-resolution image is, for example, an image having the number of pixels smaller than that of the captured image.

The mask area determiner130determines a mask area that is not processed by the grid definition unit140and subsequent constituents. Details will be described below.

The grid definition unit140defines a plurality of partial area sets in a low-resolution image. Each of the plurality of partial area sets is defined by cutting out a plurality of partial areas (hereinafter referred to as grids) from a low-resolution image. A partial area is set, for example, in a rectangular shape without a gap. The grid definition unit140defines the plurality of partial area sets so that the number of pixels in the grids differs between the plurality of partial area sets. In the following description, each of the plurality of partial area sets may be referred to as a first partial area set PA1, a second partial area set PA2, . . . , and a kthpartial area set PAk. Detailed functions of the grid definition unit140will be described below.

The extractor150derives an aggregated value obtained by aggregating a difference in feature amount between grids included in each of the plurality of partial area sets and the peripheral grids, and adds an aggregated value between the plurality of partial area sets to extract a point of interest (a point that is discontinuous with the surroundings inFIG.2). Detailed functions of each part of the extractor150will be described below.

The high-resolution processor170cuts out a portion corresponding to the point of interest in the captured image (synchronous cutting inFIG.2), performs high-resolution processing on it, and determines whether an object on a road is an object that a vehicle needs to avoid contact with. The high-resolution processor170determines whether an image reflected in the point of interest is a road marking, a falling object, or an unknown object (an unlearned object), by using, for example, a learned model that recognizes road markings (an example of an object that a vehicle does not have to avoid contact with) and falling objects (an example of an object that a vehicle needs to avoid contact) from an image. At this time, the high-resolution processor170may further narrow down the processing to the portion recognized to correspond to a road marking or a falling object in the point of interest in the captured image.

FIG.3is a diagram for describing processing of the mask area determiner130, the grid definition unit140, and the extractor150. For example, the mask area determiner130extracts edge points in a right and left direction in a low-resolution image, and detects a position in an image of a road lane marking, a road shoulder, or the like (white line, traveling road boundary) by connecting the edge points arranged in a straight line. Then, it detects an area sandwiched between the right and left road lane markings and the like and including a center point in the right and left direction on a front side of the image as a traveling road of the vehicle. Next, the mask area determiner130determines portions other than the traveling road of the vehicle (a portion above a vanishing point where the road lane markings intersect in the distance, and portions closer to the left and right ends than the road lane markings) as a mask area. The grid definition unit140and the extractor150perform processing by excluding the mask area.

As described above, the grid definition unit140defines the partial area sets in the following order from the first partial area set PA1having the largest number of pixels in the grids, the second partial area set PA2having the second largest number of pixels in the grids, and up to the kthpartial area set PAk having the smallest number of pixels in the grids. “Define” is to determine a boundary line of the grids for a low-resolution image.

In the following description, processing of the feature amount difference calculator152, the aggregation unit154, and the first addition unit156will be described. The processing of these functional units described with reference toFIGS.4to7is performed after one partial area set is first selected and a grid of interest is selected one by one in the selected partial area set. Then, when all the grids of a selected partial area set are selected as the grids of interest and the processing is completed, a next partial area set is selected and the processing is performed in the same manner. When the processing for all the partial area sets is completed, the second addition unit158adds an aggregated value (a second aggregated value V2described below) between the partial area sets to generate extraction target data PT which is one image, and passes it to the point of interest extractor160.

FIG.4is a diagram for describing the processing of the feature amount difference calculator152, the aggregation unit154, and the first addition unit156. The feature amount difference calculator152calculates a difference in feature amount for each pixel of a comparison destination grid and a comparison source grid. It is assumed that the feature amount is, for example, a luminance value for each of R, G, and B components, and a set of R, G, and B is one pixel. The comparison destination grid and the comparison source grid are selected from the grid of interest and peripheral grids.FIG.5is a diagram which shows a definition example of peripheral grids. As shown inFIG.5, grids2to9adjacent to each other in vertical, horizontal, and diagonal directions of the grid of interest are defined as peripheral grids. A method of selecting peripheral grids is not limited to this, and grids on the top, bottom, left, and right may be selected as the peripheral grids, or the peripheral grids may be selected using another rule.

The comparison destination grid and the comparison source grid are, for example, selected in order in a combination shown inFIG.6.FIG.6is a diagram which shows an example of a rule for selecting a comparison destination grid and a comparison source grid. The comparison destination grid is the grid of interest, and the comparison source grid is selected in order from the grids2to9. A relationship between the comparison destination grid and the comparison source grid may be reversed. Then, the aggregation unit154obtains a total of differences in feature amount for each pixel and divides it by the number of pixels n in the grids to calculate a first aggregated value V1. The first aggregated value V1is replaced with zero and output when the grid of interest corresponds to a mask area. That is, the feature amount difference calculator152, the aggregation unit154, and the first addition unit156execute processing with a grid1as a comparison destination grid and a grid3as a comparison source grid, processing with the grid1as a comparison destination grid and a grid8as a comparison source grid, processing with the grid1as a comparison destination grid and a grid5as a comparison source grid, processing with the grid1as a comparison destination grid and a grid6as a comparison source grid, processing with the grid1as a comparison destination grid and a grid2as a comparison source grid, processing with the grid1as a comparison destination grid and a grid4as a comparison source grid, processing with the grid1as a comparison destination grid and a grid7as a comparison source grid, and processing with the grid1as a comparison destination grid and a grid9as a comparison source grid in parallel or sequentially.

A comparison destination grid and a comparison source grid may be selected in order in a combination shown inFIG.7.FIG.7is a diagram which shows another example of the rule for selecting a comparison destination grid and a comparison source grid. A combination of a comparison destination grid and a comparison source grid is not limited to a combination of the grid of interest and peripheral grids, and may include combinations of peripheral grids (especially combinations of an upper grid and a lower grid, a left grid and a right grid, an upper left grid and a lower right grid, and an upper right grid and a lower left grid).

More specifically, a method of calculating the differences in feature amount will be described. As the method of calculating the differences in feature amount, for example, the following patterns 1 to 4 can be considered. In the following description, identification numbers of pixels in each of a comparison destination grid and a comparison source grid are represented by i (i=1 to k; k is the number of pixels in each of the comparison destination grid and the comparison source grid).

(Pattern 1)

The feature amount difference calculator152calculates, for example, a difference ΔRi in luminance of an R component, a difference ΔGi in luminance of a G component, and a difference ΔBi in luminance of a B component between pixels at the same position in both a comparison destination grid and a comparison source grid (i=1 to k as described above). Then, it obtains each pixel feature amount Ppi=ΔRi2+ΔGi2+ΔBi2for each pixel, and calculates a maximum value or an average value of each pixel feature amount Ppi as the difference in feature amount between the comparison destination grid and the comparison source grid.

(Pattern 2)

The feature amount difference calculator152calculates, for example, a statistical value (refer to an average value, a median value, a mode value, or the like) Raa of the luminance of the R component, a statistical value (the same as above) Gaa of the luminance of the G component, and a statistical value (the same as above) Baa of the luminance of the B component of each pixel in the comparison destination grid, calculates a statistical value (the same as above) Rab of the luminance of the R component, a statistical value (the same as above) Gab of the luminance of the G component, and a statistical value (the same as above) Bab of the luminance of the B component of each pixel in the comparison source grid, and obtains differences ΔRa(=Raa-Rab), ΔGa(=Gaa-Gab), and ΔBa(=Baa-Bab) of these. Then, ΔRa2+ΔGa2+ΔBa2, which is a sum of squares of the differences in luminance, or Max(ΔRa2, ΔGa2, ΔBa2), which is a maximum value of the squares of the differences in luminance, as the difference in feature amount between the comparison destination grid and the comparison source grid.

(Pattern 3)

The feature amount difference calculator152calculates, for example, a first index value W1ai(=(R−B)/(R+G+B)) obtained by dividing a difference in luminance between the R component and the B component by a sum of the luminance of each of the R, G, and B components, and a second index value W2ai(=(R−G)/(R+G+B)) obtained by dividing a difference in luminance between the R component and the G component by the sum of the luminance of each of the R, G, and B components for each pixel i in the comparison destination grid. Moreover, the feature amount difference calculator152calculates, for example, a first index value W1bi(=(R−B)/(R+G+B)) obtained by dividing a difference in luminance between the R component and the B component by a sum of the luminance of each of the R, G, and B components, and a second index value W2bi(=(R−G)/(R+G+B)) obtained by dividing a difference in luminance between the R component and the G component by the sum of the luminance of each of the R, G, and B components for each pixel I in the comparison source grid. Next, the feature amount difference calculator152calculates each pixel feature amount Ppi=(W1ai-W1bi)2+(W2ai-W2bi)2. Then, the feature amount difference calculator152calculates a maximum value or an average value of each pixel feature amount Ppi as the difference in feature amount between the comparison destination grid and the comparison source grid. By combining the first index value and the second index value, a balance of the RGB components in each pixel can be expressed. In the same manner as described above, for example, the luminance of each of the R, G, and B components is defined as a size of a vector shifted by 120 degrees, and a vector sum may also be used in the same manner as in the combination of the first index value and the second index value.

(Pattern 4)

The feature amount difference calculator152calculates, for example, a statistical value (the same as above) Raa of the luminance of the R component, a statistical value (the same as above) Gaa of the luminance of the G component, and a statistical value (the same as above) Baa of the luminance of the B component of each pixel in the comparison destination grid, and calculates a statistical value (the same as above) Rab of the luminance of the R component, a statistical value (the same as above) Gab of the luminance of the G component, and a statistical value (the same as above) Bab of the luminance of the B component of each pixel in the comparison source grid. Next, the feature amount difference calculator152calculates a third index value3a(=(Raa−Baa)/(Raa+Gaa+Baa)) obtained by dividing a difference between the statistical value Raa of the luminance of the R component and the statistical value Baa of the luminance of the B component by a sum of the statistical values of the luminance of each of the R, G, B components, and a fourth index value W4a(=(Raa−Gaa)/(Raa+Gaa+Baa)) obtained by dividing a difference between the statistical value Raa of the luminance of the R component and the statistical value Gaa of the luminance of the G component by a sum of the statistical values of the luminance of each of the R, G, B components for the comparison destination grid. Similarly, the feature amount difference calculator152calculates a third index value3b(=(Rab−Bab)/(Rab+Gab+Bab)) obtained by dividing a difference between the statistical value Rab of the luminance of the R component and the statistical value Bab of the luminance of the B component by a sum of the statistical values of the luminance of each of the R, G, B components, and a fourth index value W4b(=(Rab−Gab)/(Rab+Gab+Bab)) obtained by dividing a difference between the statistical value Rab of the luminance of the R component and the statistical value Gab of the luminance of the G component by a sum of the statistical values of the luminance of each of the R, G, B components for the comparison source grid. Then, the feature amount difference calculator152obtains a difference ΔW3of the third index value W3aof the comparison destination grid and the third index value W3bof the comparison source grid and a difference ΔW4of the fourth index value W4aof the comparison destination grid and the fourth index value W4bof the comparison source grid, and calculates a sum ΔW32+ΔW42of the squares of these or maximum values Max (ΔW32, ΔW42) of the squares as the difference in feature amount between the comparison destination grid and the comparison source grid.

When an image to be processed is a black-and-white image, the feature amount difference calculator152may simply calculate a difference in luminance value as the difference in feature amount between the comparison destination grid and the comparison source grid, and, even when an image to be processed is an RGB image, it may convert the RGB image into a black-and-white image and calculate a difference in orbital value as the difference in feature amount between the comparison destination grid and the comparison source grid.

Returning toFIG.4, the first addition unit156calculates a second aggregated value V2by adding the first aggregated value V1obtained corresponding to the grid of interest. The second aggregated value V2is an example of the “aggregated value” in the claims. When processing of obtaining the second aggregated value V2is completed while the grid of interest is changed, data in which the second aggregated value V2is set for all the grids is generated for each partial area set.

When the data in which the second aggregated value V2is set for all the grids is generated, the second addition unit158adds these data between the partial area sets to generate an extraction target data PT.FIG.8is a diagram for describing processing of the first addition unit156and the second addition unit158. InFIG.8, the smallest rectangle is one pixel of the low-resolution image. Here, for simplification of description, it is assumed that there are three of the first partial area set PA1, the second partial area set PA2, and the third partial area set PA3, and their sizes are also considerably smaller than they actually are. In addition, it is assumed that the second aggregated value V2is normalized at any stage to be a value between zero and 1. In the shown example, the first partial area set PA1is a set of first grids composed of 16 pixels, the second partial area set PA2is a set of second grids composed of 9 pixels, and the third partial area set PA3is a set of third grids composed of four pixels. Since the second aggregated value V2is set for a grid in which a plurality of pixels are bundled, the second addition unit158expands the second aggregated value V2set in the grid to all the pixels in the grid, and then adds pixel values of the pixels between the partial area sets to generate the extraction target data PT. There may be a partial area set in which pixels and grids have a one-to-one correspondence, and, in this case, it is not necessary to expand the second aggregated value V2for each pixel. An example in which a size of a grid is 16 pixels, 9 pixels, or 4 pixels is shown, but when the pattern 2 or the pattern 4 described above is adopted as the method of calculating the difference in feature amount, it is possible to reduce a calculation load of the statistical values by setting the number of pixels on one side to a power of 2 such that the size of a grid is 4 pixels, 16 pixels, or 64 pixels.

The point of interest extractor160extracts a circumscribing rectangle that includes a set of parts of the data PT to be processed in which an added value of the aggregated values is equal to or greater than a threshold value.

As described above, the high-resolution processor170performs high-resolution processing on an area obtained by applying only the position of a point of interest to the captured image, and determines whether an object on a road is an object that the vehicle needs to avoid contact with.

A result of the determination by the high-resolution processor170is output to the traveling control device200and/or the notification device210. The traveling control device200performs automatic brake control, automatic steering control, and the like to avoid contact between an object determined to be a “falling object” (actually, an area on the image) and the vehicle. The notification device210outputs an alarm in various methods when time to collision (TTC) between the object determined to be a “falling object” (the same as above) and the vehicle is less than a threshold value.

According to the embodiment described above, it is possible to maintain a high detection accuracy while reducing a processing load by including the acquirer110that acquires a captured image of at least a road in the traveling direction of the vehicle, the low-resolution image generator120that generates a low-resolution image in which the image quality of the captured image is lowered, the grid definition unit140that defines one or more partial area sets, and the extractor150that derives an aggregated value obtained by aggregating differences in feature amount between partial areas included in each of one or more partial area sets and the peripheral partial areas, and extracts the point of interest based on the aggregated value.

When it is assumed that the processing performed by the feature amount difference calculator152or the aggregation unit154is executed for the captured image as it is, there is a concern that the processing load increases as the number of pixels increases, and an operation of the traveling control device200or the notification device210will not be in time for the approach of a falling object. Regarding this point, the object detection device100of the embodiment can detect an object while reducing the processing load by generating a low-resolution image and performing processing.

Furthermore, according to the embodiment, since the grid definition unit140defines a plurality of partial area sets so that the number of pixels in the grids differs between the plurality of partial area sets, and the extractor150extracts a point of interest by adding an aggregated value for each pixel between the partial area sets, it is possible to improve a robustness of detection performance against a variation in size of a falling object. This is because, when processing is simply performed on a low-resolution image, although there is a concern that the presence of a falling object may not be recognized due to a lowering of the image quality, according to the embodiment, it can be expected that a falling object will appear as a feature amount in a grid of any size using the device described above. From the description above, according to the object detection device100of the embodiment, it is possible to maintain a high detection accuracy while reducing the processing load.

Although a form for implementing the present invention has been described above using the embodiment, the present invention is not limited to the embodiment, and various modifications and substitutions can be made within a range not departing from the gist of the present invention.