INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING PROGRAM, AND INFORMATION PROCESSING METHOD

A processing load in a case where a plurality of different sensors is used can be reduced. An information processing apparatus according to an embodiment includes: a recognition processing unit (15, 40b) configured to perform recognition processing for recognizing a target object by adding, to an output of a first sensor (23), region information that is generated according to object likelihood detected in a process of object recognition processing based on an output of a second sensor (21) different from the first sensor.

FIELD

The present disclosure relates to an information processing apparatus, an information processing system, an information processing program, and an information processing method.

BACKGROUND

Technologies for detecting an object with a sensor such as an image sensor or a millimeter-wave radar are known. As sensors for detecting an object, there are sensors of various detection methods, and the sensors are suitable for different situations in some cases. Thus, technologies have been proposed for detecting an object by using, in combination, the sensors different in detection method.

CITATION LIST

Patent Literature

Patent Literature 1: WO 17/057056 A

SUMMARY

Technical Problem

In using, in combination, a plurality of sensors different in detection method, when detection processing is performed by using all the outputs of the sensors, the detection processing load may increase. In order to avoid the increase in detection processing load, it is possible to use a method in which a detection window is set for the output of the sensors and the range of the detection processing is limited. However, the method for setting the detection window has not been defined.

An object of the present disclosure is to provide an information processing apparatus, an information processing system, an information processing program, and an information processing method that are capable of reducing the processing load in a case where a plurality of different sensors is used.

Solution to Problem

For solving the problem described above, an information processing apparatus according to one aspect of the present disclosure has a recognition processing unit configured to perform recognition processing for recognizing a target object by adding, to an output of a first sensor, region information that is generated according to object likelihood detected in a process of object recognition processing based on an output of a second sensor different from the first sensor.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings. In the following embodiments, the same parts are denoted with the same reference numerals and repeated explanation of these parts is omitted.

Hereinafter, the embodiments of the present disclosure are described in the following order.

1. Technology applicable to each embodiment

1-1. Example of vehicle-mounted system

1-2. Outline of functions

1-3. Example of hardware configuration

2. Outline of embodiments of present disclosure

3. First Embodiment

3-1. Specific examples

4. Second Embodiment

8-1. First example

8-2. Second example

8-3. Third example

8-4. Fourth example

8-5. Fifth example

8-6. Sixth example

1. Technology Applicable to Each Embodiment

Prior to the description of each embodiment of the present disclosure, a technology applicable to each embodiment of the present disclosure is described for easy understanding.

1-1. Example of Vehicle-Mounted System

First, a vehicle-mounted system applicable to each embodiment of the present disclosure is schematically described.FIG.1is a block diagram illustrating an example of a schematic configuration of a vehicle control system which is an example of the vehicle-mounted system applicable to each embodiment according to the present disclosure.

A vehicle control system12000includes a plurality of electronic control units connected to one another via a communication network12001. In the example illustrated inFIG.1, the vehicle control system12000includes a drive system control unit12010, a body system control unit12020, a vehicle-exterior-information detection unit10, a vehicle-interior-information detection unit12040, and an integrated control unit12050. Further, as the functional configuration of the integrated control unit12050, a microcomputer12051, a sound/image output unit12052, and a vehicle-mounted network interface (I/F)12053are illustrated.

The drive system control unit12010controls the operation of devices related to the drive system of a vehicle in accordance with a variety of programs. For example, the drive system control unit12010functions as a control device for a driving force generation device, such as an internal-combustion engine and a driving motor, which generates a driving force of the vehicle, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

The body system control unit12020controls the operation of a variety of devices equipped in the vehicle body in accordance with a variety of programs. For example, the body system control unit12020functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps including a headlamp, a tail lamp, a brake lamp, a blinker, and a fog lamp. In such a case, the body system control unit12020receives an input of a radio wave sent from a mobile device functioning as a key or signals of the switches. The body system control unit12020receives the inputs of the radio wave or the signals to control a door lock device, the power window device, the lamps, and so on of the vehicle.

The vehicle-exterior-information detection unit10detects information regarding outside the vehicle on which the vehicle control system12000is mounted. For example, a data acquisition unit20is connected to the vehicle-exterior-information detection unit10. In the vehicle-exterior-information detection unit10, the data acquisition unit20includes a variety of sensors with which to monitor the situation outside the vehicle. For example, the data acquisition unit20may include an optical sensor that receives visible light or non-visible light such as an infrared ray and outputs an electrical signal based on the amount of light received, and the vehicle-exterior-information detection unit10receives an image captured by the optical sensor. Further, the data acquisition unit20may further include a sensor that monitors the external situation in another method such as a millimeter-wave radar, light detection and ranging or laser imaging detection and ranging (LiDAR), or an ultrasonic sensor.

The data acquisition unit20is provided in, for example, a front nose of a vehicle12100, a side mirror thereof, an upper part of a front glass inside the vehicle, or the like with a region ahead of the vehicle regarded as the data acquisition direction. The vehicle-exterior-information detection unit10may perform distance detection processing or detection processing of an object such as a person, a vehicle, an obstacle, a sign, or a character on the road surface on the basis of outputs of the sensors received from the data acquisition unit20.

The vehicle-interior-information detection unit12040detects information regarding inside the vehicle. For example, a driver state detection unit12041for detecting the state of the driver is connected to the vehicle-interior-information detection unit12040. The driver state detection unit12041includes, for example, a camera for capturing an image of the driver, and the vehicle-interior-information detection unit12040may calculate a degree of fatigue or a degree of concentration of the driver, or, alternatively, may judge whether or not the driver is dozing off on the basis of detection information inputted from the driver state detection unit12041.

The microcomputer12051can compute a control target value of the driving force generation device, the steering mechanism, or the braking device on the basis of vehicle-exterior-information and vehicle-interior information acquired by the vehicle-exterior-information detection unit10or the vehicle-interior-information detection unit12040and output a control command to the drive system control unit12010. For example, the microcomputer12051can perform a cooperative control intended to implement the functions of an advanced driver-assistance system (ADAS) including collision avoidance or shock mitigation for the vehicle, traveling after a leading vehicle based on a distance between vehicles, traveling while maintaining a vehicle speed, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, and the like.

Further, the microcomputer12051can perform a cooperative control intended to achieve automated driving that is autonomous traveling without an operation performed by a driver by controlling the driving force generation device, the steering mechanism, or the braking device on the basis of the information regarding the surroundings of the vehicle acquired by the vehicle-exterior-information detection unit10or the vehicle-interior-information detection unit12040.

The microcomputer12051can also output a control command to the body system control unit12020on the basis of the vehicle-exterior-information acquired by the vehicle-exterior-information detection unit10. For example, the microcomputer12051can perform a cooperative control intended to prevent glare, such as switching from a high beam to a low beam by controlling the headlamp depending on the position of a leading vehicle or an oncoming vehicle detected by the vehicle-exterior-information detection unit10.

The sound/image output unit12052sends, for a person on board in the vehicle or the outside the vehicle, an output signal of at least one of a sound and an image to an output device to which visual or audio information can be sent.FIG.1exemplifies, as the output device, an audio speaker12061, a display unit12062, and an instrument panel12063. The display unit12062may include, for example, at least one of an on-board display and a head-up display.

1-2. Outline of Functions

Next, an example of the functions of the vehicle-exterior-information detection unit10applicable to each embodiment of the present disclosure is schematically described.

FIG.2is a functional block diagram of an example for explaining the functions of the vehicle-exterior-information detection unit10in the vehicle control system12000ofFIG.1. InFIG.2, the data acquisition unit20includes a camera21and a millimeter-wave radar23. The vehicle-exterior-information detection unit10includes an information processing unit11. The information processing unit11includes an image processing unit12, a signal processing unit13, a geometric transformation unit14, and a recognition processing unit15.

The camera21includes an image sensor22. The image sensor22can be any type of image sensor such as a CMOS image sensor or a CCD image sensor. The camera21(image sensor22) captures an image of a region situated ahead of the vehicle on which the vehicle control system12000is mounted, and supplies the obtained image (hereinafter, referred to as a captured image) to the image processing unit12.

The millimeter-wave radar23senses the region situated ahead of the vehicle, and the sensed range and the sensed range of the camera21overlap at least partially. For example, the millimeter-wave radar23sends a transmission signal including a millimeter-wave to the front of the vehicle, and receives, using a reception antenna, a received signal that is a signal reflected off an object (reflector) present ahead of the vehicle. For example, a plurality of reception antennas is provided at predetermined intervals in the lateral direction (width direction) of the vehicle. Further, a plurality of reception antennas may also be provided in the height direction. The millimeter-wave radar23supplies the signal processing unit13with data (hereinafter, referred to as millimeter-wave data) that chronologically indicates the strength of a received signal received by each reception antenna.

Note that the transmission signal of the millimeter-wave radar23is scanned in a predetermined angular range, for example, in a two-dimensional plane to form a fan-like sensed range. This is scanned in the vertical direction to obtain a bird's-eye view having three-dimensional information.

The image processing unit12performs predetermined image processing on the captured image. For example, the image processing unit12performs thinning processing, filtering processing, or the like on pixels of the captured image in accordance with the size of an image that the recognition processing unit15can process, and reduces the number of pixels of the captured image (reduces the resolution). The image processing unit12supplies the captured image with resolution lowered (hereinafter, referred to as a low-resolution image) to the recognition processing unit15.

The signal processing unit13performs predetermined signal processing on the millimeter-wave data to generate a millimeter-wave image that is an image indicating the result of sensing performed by the millimeter-wave radar23. Note that the signal processing unit13generates, for example, a plural-channel (ch) millimeter-wave image including a signal strength image and a speed image. The signal strength image is a millimeter-wave image indicating the position of each object that is present ahead of the vehicle and the strength of a signal that is reflected from each object (received signal). The speed image is a millimeter-wave image indicating the position of each object that is present ahead of the vehicle and a relative speed of each object to the vehicle.

The geometric transformation unit14performs a geometric transformation on the millimeter-wave image to transform the millimeter-wave image into an image having the same coordinate system as that of the captured image. In other words, the geometric transformation unit14transforms the millimeter-wave image into an image viewed from the same viewpoint as the captured image (hereinafter, referred to as a geometrically transformed millimeter-wave image). More specifically, the geometric transformation unit14transforms the coordinate system of the signal strength image and the speed image from the coordinate system of the millimeter-wave image to the coordinate system of the captured image. Note that the signal strength image and the speed image that have been subjected to the geometric transformation are referred to as a geometrically transformed signal strength image and a geometrically transformed speed image, respectively. The geometric transformation unit14supplies the geometrically transformed signal strength image and the geometrically transformed speed image to the recognition processing unit15.

The recognition processing unit15uses a recognition model obtained in advance through machine learning to perform processing of recognizing a target object that is present ahead of the vehicle on the basis of the low-resolution image, the geometrically transformed signal strength image, and the geometrically transformed speed image. The recognition processing unit15supplies data indicating the recognition result of the target object to the integrated control unit12050via the communication network12001.

Note that the target object is an object to be recognized by the recognition processing unit15, and any object can be set to be the target object. However, it is desirable that an object that includes a portion having a high reflectance of a transmission signal of the millimeter-wave radar23is set to be the target object. Hereinafter, the case in which the target object is a vehicle is described as an appropriate example.

FIG.3illustrates an example of the configuration of the object recognition model40used in the recognition processing unit15.

The object recognition model40is a model obtained by machine learning. Specifically, the object recognition model40is a model obtained by deep learning which is a type of machine learning using a deep neural network. More specifically, the object recognition model40includes a single shot multi-box detector (SSD) which is one of the object recognition models using the deep neural network. The object recognition model40includes a feature-amount extraction unit44and a recognition unit45.

The feature-amount extraction unit44includes a feature extraction layer41ato a feature extraction layer41cthat are convolutional layers using a convolutional neural network, and an addition unit42. The feature extraction layer41aextracts a feature amount of a captured image Pa to generate a feature map that two-dimensionally represents the distribution of the feature amount (hereinafter, referred to as a captured image feature map). The feature extraction layer41asupplies the captured image feature map to the addition unit42.

The feature extraction layer41bextracts a feature amount of a geometrically transformed signal strength image Pb to generate a feature map that two-dimensionally represents the distribution of the feature amount (hereinafter, referred to as a signal strength image feature map). The feature extraction layer41bsupplies the signal strength image feature map to the addition unit42.

The feature extraction layer41cextracts a feature amount of a geometrically transformed speed image Pc to generate a feature map that two-dimensionally represents the distribution of the feature amount (hereinafter, referred to as a speed image feature map). The feature extraction layer41csupplies the speed image feature map to the addition unit42.

The addition unit42adds the captured image feature map, the signal strength image feature map, and the speed image feature map together to generate a combining feature map. The addition unit42supplies the combining feature map to the recognition unit45.

The recognition unit45includes a convolutional neural network. Specifically, the recognition unit45includes a convolutional layer43ato a convolutional layer43c.

The convolutional layer43aperforms a convolution operation on the combining feature map. The convolutional layer43aperforms processing of recognizing the target object on the basis of the combining feature map on which the convolution operation has been performed. The convolutional layer43asupplies the convolutional layer43bwith the combining feature map on which the convolution operation has been performed.

The convolutional layer43bperforms a convolution operation on the combining feature map provided by the convolutional layer43a. The convolutional layer43bperforms processing of recognizing the target object on the basis of the combining feature map on which the convolution operation has been performed. The convolutional layer43asupplies the convolutional layer43cwith the combining feature map on which the convolution operation has been performed.

The convolutional layer43cperforms a convolution operation on the combining feature map provided by the convolutional layer43b. The convolutional layer43bperforms processing of recognizing the target object on the basis of the combining feature map on which the convolution operation has been performed.

The object recognition model40outputs data indicating a result of the recognition of the target object that is performed by the convolutional layer43ato the convolutional layer43c.

Note that the size (the number of pixels) of the combining feature map decreases in order from the convolutional layer43a, and is the smallest in the convolutional layer43c. Further, as the size of the combining feature map increases, the recognition accuracy of a target object having a small size, as viewed from the vehicle (camera), increases, and as the size of the combining feature map decreases, the recognition accuracy of a target object having a large size, as viewed from the vehicle, increases. Thus, for example, in a case where the target object is a vehicle, a small vehicle in a distant location is easily recognized in the combining feature map having a large size, and a large vehicle nearby is easily recognized in the combining feature map having a small size.

FIG.4is a block diagram illustrating an example of the configuration of a learning system30. The learning system30performs learning processing on the object recognition model40ofFIG.3. The learning system30includes an input unit31, an image processing unit32, a correct-answer-data generation unit33, a signal processing unit34, a geometric transformation unit35, a training data generation unit36, and a learning unit37.

The input unit31includes various input devices, and is used for input of data necessary to generate training data, user operation, and so on. For example, in a case where a captured image is inputted, the input unit31supplies the captured image to the image processing unit32. For example, in a case where millimeter-wave data is inputted, the input unit31supplies the millimeter-wave data to the signal processing unit34. For example, the input unit31supplies the correct-answer-data generation unit33and the training data generation unit36with data indicating an instruction of a user that is inputted by an operation performed by the user.

The image processing unit32performs processing similar to the processing performed by the image processing unit12ofFIG.2. Specifically, the image processing unit32performs predetermined image processing on a captured image to generate a low-resolution image. The image processing unit32supplies the low-resolution image to the correct-answer-data generation unit33and the training data generation unit36.

The correct-answer-data generation unit33generates correct answer data on the basis of the low-resolution image. For example, the user designates a location of a vehicle in the low-resolution image through the input unit31. The correct-answer-data generation unit33generates correct answer data indicating the location of the vehicle in the low-resolution image on the basis of the location of the vehicle designated by the user. The correct-answer-data generation unit33supplies the correct answer data to the training data generation unit36.

The signal processing unit34performs processing similar to the processing performed by the signal processing unit13ofFIG.2. Specifically, the signal processing unit34performs predetermined signal processing on the millimeter-wave data to generate a signal strength image and a speed image. The signal processing unit34supplies the signal strength image and the speed image to the geometric transformation unit35.

The geometric transformation unit35performs processing similar to the processing performed by the geometric transformation unit14ofFIG.2. Specifically, the geometric transformation unit35performs a geometric transformation on the signal strength image and the speed image. The geometric transformation unit35supplies the geometrically transformed signal strength image and the geometrically transformed speed image that have been subjected to the geometric transformation to the training data generation unit36.

The training data generation unit36generates input data including the low-resolution image, the geometrically transformed signal strength image, and the geometrically transformed speed image, and training data including the correct answer data. The training data generation unit36supplies the training data to the learning unit37.

The learning unit37uses the training data to perform learning processing on the object recognition model40. The learning unit37outputs the object recognition model40that has learned.

Here, the learning processing on an object recognition model performed by the learning system30is described.

Note that, before the start of the processing, data used to generate training data is collected. For example, in a state where the vehicle is actually traveling, the camera21and the millimeter-wave radar23provided in the vehicle perform sensing with respect to a region situated ahead of the vehicle. Specifically, the camera21captures an image of the region situated ahead of the vehicle, and stores the captured image thus obtained into a storage unit. The millimeter-wave radar23detects an object present ahead of the vehicle, and stores the millimeter-wave data thus obtained in the storage unit. The training data is generated on the basis of the captured image and the millimeter-wave data accumulated in the storage unit.

First, the learning system30generates training data. For example, the user inputs, to the learning system30via the input unit31, the captured image and the millimeter-wave data that are acquired substantially simultaneously. In other words, the captured image and the millimeter-wave data obtained by performing sensing at substantially the same point in time are inputted to the learning system30. The captured image is supplied to the image processing unit32, and the millimeter-wave data is supplied to the signal processing unit34.

The image processing unit32performs image processing such as the thinning processing on the captured image to generate a low-resolution image. The image processing unit32supplies the low-resolution image to the correct-answer-data generation unit33and the training data generation unit36.

The signal processing unit34performs predetermined signal processing on the millimeter-wave data to estimate the position and speed of the object that has reflected the transmission signal ahead of the vehicle. The position of the object is represented by, for example, a distance from the vehicle to the object and a direction (angle) of the object with respect to an optical axis direction (traveling direction of the vehicle) of the millimeter-wave radar23. Note that the optical axis direction of the millimeter-wave radar23is equal to the center direction of the range in which the transmission signal is radiated, for example, in a case where the transmission signal is radially transmitted, and is equal to the center direction of the range in which the transmission signal is scanned in a case where the transmission signal is scanned. The speed of the object is represented by, for example, a relative speed of the object to the vehicle.

The signal processing unit34generates a signal strength image and a speed image on the basis of a result of the estimation of the position and speed of the object. The signal processing unit34supplies the signal strength image and the speed image to the geometric transformation unit35. Although not illustrated, the speed image is an image showing the position of the object present ahead of the vehicle and the distribution of the relative speed of each object in a bird's-eye view similarly to the signal strength image.

The geometric transformation unit35performs a geometric transformation on the signal strength image and the speed image, and transforms the signal strength image and the speed image into an image having the same coordinate system as that of the captured image, and thereby generates a geometrically transformed signal strength image and a geometrically transformed speed image. The geometric transformation unit35supplies the geometrically transformed signal strength image and the geometrically transformed speed image to the training data generation unit36.

In the geometrically transformed signal strength image, a portion having a higher signal strength is brighter, and a portion having a lower signal strength is darker. In the geometrically transformed speed image, a portion having a higher relative speed is brighter, a portion having a lower relative speed is darker, and a portion where the relative speed is undetectable (no object is present) is filled in black. As described above, the geometric transformation on the millimeter-wave image (the signal strength image and the speed image) represents not only the position of the object in the transverse direction and the depth direction but also the position of the object in the height direction.

However, the resolution of the millimeter-wave radar23in the height direction decreases as the distance increases. Thus, the height of an object that is far away is sometimes detected to be larger than the actual height.

In contrast, in the case of geometric transformation on the millimeter-wave image, the geometric transformation unit35limits the height of the object that is present a predetermined distance or more away. Specifically, in the case of geometric transformation on the millimeter-wave image, in a case where the height of the object that is present a predetermined distance or more away exceeds a predetermined upper limit value, the geometric transformation unit35limits the height of the object to the upper limit value and performs the geometric transformation. This prevents, for example, in a case where the target object is a vehicle, the occurrence of erroneous recognition due to the detection of the height of a vehicle in a distant location to be larger than the actual height.

The training data generation unit36generates input data including the captured image, the geometrically transformed signal strength image, and the geometrically transformed speed image, and training data including the correct answer data. The training data generation unit36supplies the training data thus generated to the learning unit37.

Next, the learning unit37causes the object recognition model40to perform learning. Specifically, the learning unit37inputs the input data included in the training data to the object recognition model40. The object recognition model40performs processing of recognizing the target object to output data indicating a result of the recognition. The learning unit37compares the result of the recognition of the object recognition model40with the correct answer data, and adjusts parameters and the like of the object recognition model40so as to reduce the error.

Next, the learning unit37determines whether or not the learning is to be continuously performed. For example, in a case where the learning performed by the object recognition model40has not come to an end, the learning unit37determines that the learning is to be continuously performed, and the processing returns to the learning data generation processing performed at the beginning. Thereafter, each processing described above is repeatedly executed until it is determined that the learning is to be terminated.

On the other hand, as a result of the determination by the learning unit37, for example, in a case where the learning by the object recognition model40has come to an end, the learning unit37determines that the learning is to be terminated, and the object recognition model learning processing is terminated. As described above, the object recognition model40that has performed learning is generated.

1-3. Example of Hardware Configuration

The description goes on to an example of the hardware configuration of the vehicle-exterior-information detection unit10applicable to each embodiment of the present disclosure.FIG.5is a block diagram illustrating an example of the hardware configuration of the vehicle-exterior-information detection unit10applicable to each embodiment. InFIG.5, the vehicle-exterior-information detection unit10includes a central processing unit (CPU)400, a read only memory (ROM)401, a random access memory (RAM)402, and interfaces (I/F)403,404, and405, which are connected to one another for communication via a bus410. Note that the vehicle-exterior-information detection unit10may further include a storage device such as a flash memory.

The CPU400controls the entire operation of the vehicle-exterior-information detection unit10using the RAM402as a work memory according to a program or data stored, in advance, in the ROM401. Here, the ROM401or the RAM402stores, in advance, the program and data for implementing the object recognition model40described with reference toFIGS.2to4. The program is executed by the CPU400, which constructs the object recognition model40in the vehicle-exterior-information detection unit10.

The interface403is an interface for connecting the camera21. The interface404is an interface for connecting the millimeter-wave radar23. The vehicle-exterior-information detection unit10controls the camera21and the millimeter-wave radar23via the interfaces403and404, and acquires captured image data (hereinafter, referred to as image data) captured by the camera21and millimeter-wave data acquired by the millimeter-wave radar23. The vehicle-exterior-information detection unit10executes processing of recognizing an object by applying, as the input data, the image data and the millimeter-wave data to the object recognition model40.

InFIG.5, the interface405is an interface for performing communication between the vehicle-exterior-information detection unit10and the communication network12001. The vehicle-exterior-information detection unit10sends information indicating the result of the object recognition outputted by the object recognition model40from the interface405to the communication network12001.

2. Outline of Embodiments of Present Disclosure

The description goes on to an outline of the embodiments of the present disclosure. In each embodiment of the present disclosure, a detection window for detecting the target object on the basis of an output of a first sensor for detecting the target object is set on the basis of an output of a second sensor for detecting the target object in a manner different from that of the first sensor, and the processing of recognizing the target object is performed on the basis of an output of a region corresponding to the detection window in the output of the second sensor.

FIG.6is a diagram schematically illustrating the object recognition model40according to an embodiment of the present disclosure. In an object recognition model40a, an image data100acquired from the camera21is inputted to a feature extraction layer110. Further, a millimeter-wave image data200based on the millimeter-wave image acquired from the millimeter-wave radar23is inputted to a feature extraction layer210.

The image data100inputted to the object recognition model40ais shaped into data including a feature amount of 1 ch or more in the image processing unit12, for example. The image data100is data in which features are extracted by the feature extraction layer110in the object recognition model40a, the size is changed as necessary, and ch of the feature amount is added. The image data100of which features are extracted by the feature extraction layer110is subjected to convolutional processing in an object recognition layer120, and a plurality of sets of object recognition layer data that is sequentially convolved is generated.

The object recognition model40agenerates an attention map130on the basis of the plurality of sets of object recognition layer data. The attention map130includes, for example, information indicating a detection window for limiting a target region for the object recognition with respect to a range indicated in the image data100. The attention map130thus generated is inputted to a multiplication unit220.

In contrast, the millimeter-wave image data200inputted to the object recognition model40ais shaped into data including a feature amount of 1 ch or more by the signal processing unit13and the geometric transformation unit14, for example. The millimeter-wave image data200is data in which features are extracted by the feature extraction layer210in the object recognition model40a, the size is changed as necessary (the size is set to be the same as that of the image data100, for example) and ch of the feature amount is added. The millimeter-wave image data200of each ch of which features are extracted by the feature extraction layer is inputted to the multiplication unit220, and is multiplied for each pixel with the attention map130. As a result, an area where the object recognition is performed is limited in the millimeter-wave image data200. Further, the output of the multiplication unit220is inputted to an addition unit221, and the output of the feature extraction layer210is added. The output of the addition unit221is inputted to the object recognition layer230and is subjected to the convolutional processing.

As described above, the object recognition processing is performed on the region limited by the attention map130, leading to the reduction in the processing amount of the object recognition processing.

Note that the processing speed can be increased by using data on a past frame101as the image data100.

3. First Embodiment

The description goes on to the first embodiment of the present disclosure.FIG.7is a diagram illustrating a configuration of an example of an object recognition model according to the first embodiment. Referring toFIG.7, in an object recognition model40b, processing in the feature extraction layers110and210and the object recognition layers120and230illustrated on the left side ofFIG.7is equivalent to that inFIG.6, and thus, description thereof is omitted herein.

FIG.7schematically illustrates, on the right side thereof, the object recognition layer230based on the millimeter-wave image data200and the object recognition layer120based on the image data100. The object recognition layer230includes sets of object recognition layer data2300,2301,2302,2303,2304,2305, and to2306that are sequentially convolved on the basis of the millimeter-wave image data200. Further, the object recognition layer120includes sets of object recognition layer data1200,1201,1202,1203,1204,1205, and1206that are sequentially convolved on the basis of the image data100.

Note that, in the following description, in a case where it is not necessary to particularly distinguish the sets of object recognition layer data1200to1206from one another, object recognition layer data120xis described as a representative. Similarly, in a case where it is not necessary to particularly distinguish the sets of object recognition layer data2300to2306from one another, object recognition layer data230xis described as a representative.

InFIG.7, specific examples of the object recognition layer data1200to1207are illustrated as layer (layer) images #0, #1, #2, #3, #4, #5, and #6 corresponding to the attention map. Although the details are described later, white portions of the layer images #1 and #2 of the layer images show detection windows.

That is, the object recognition layer120obtains object likelihood on the basis of the features of the layer images #0, #1, #2, #3, #4, #5, and #6, and determines a region having high object likelihood thus obtained. The object recognition layer120obtains, for the layer image #1 for example, object likelihood on the basis of the pixel information. Then, the object likelihood obtained is compared with a threshold, and a region in which the object likelihood is higher than the threshold is determined. In the example ofFIG.7, a region shown in white in the layer image #1 indicates a region having the object likelihood higher than the threshold. The object recognition layer120generates region information indicating the region. The region information includes information indicating a position in the layer image #1 and a value indicating the object likelihood at the position. The object recognition layer120sets a detection window on the basis of the region indicated in the region information and generates an attention map.

Here, the size of the sets of object recognition layer data1200to1206is sequentially reduced by convolution. For example, in the example ofFIG.7, the size of the layer image #0 (object recognition layer data1200) is set to ½ by convolution for one layer. For example, assuming that the size of the layer image #0 is 640 pixels×384 pixels, the size of the layer image #6 is 1 pixel×1 pixel by convolution (and shaping processing) of seven layers.

As described above, a layer image with a small number of convolutions and a large size can detect a smaller (distant) target object, and a layer image with a large number of convolutions and a small size can detect a larger (nearer) target object. The same applies to the sets of object recognition layer data2300to2306based on the millimeter-wave data.

A layer image with a large number of convolutions and a small number of pixels or a layer image with a small number of convolutions in which an object is recognized as a small object is not appropriate for use in the object recognition processing in some cases. Therefore, in the example ofFIG.7, the attention map may be generated using the number of layer images (for example, three layers of the layer images #1 to #3) according to the purpose instead of generating the attention map for all the seven layers.

The sets of object recognition layer data1200to1207are inputted to the corresponding combining units300. Further, the sets of object recognition layer data2300to2306based on the millimeter-wave image data200are inputted to the corresponding combining units300. The combining units300combine the sets of object recognition layer data1200to1207and the sets of object recognition layer data2300to2306thus inputted to generate combined object recognition layer data3100to3106.

FIG.8is a diagram illustrating a configuration of an example of the combining unit300according to the first embodiment. The combining unit300includes the multiplication unit220and the addition unit221. The multiplication unit220receives, at one input end, the object recognition layer data120xbased on the attention map based on the image data100. The multiplication unit220receives, at the other input end, the object recognition layer data230xbased on the millimeter-wave image data200. The multiplication unit220calculates, for each pixel, a product of the object recognition layer data120xinputted to one input end thereof and the object recognition layer data230xinputted to the other input end thereof. The calculation by the multiplication unit220emphasizes a region corresponding to the detection window in the millimeter-wave image data200(object recognition layer data230x).

The present invention is not limited thereto, and the object recognition model40amay reduce a region outside the detection window in the millimeter-wave image data200.

The result of multiplication by the multiplication unit220is inputted to one input end of the addition unit221. The addition unit221receives, at the other input end, the object recognition layer data230xbased on the millimeter-wave image data200. The addition unit221calculates a sum of matrices for the result of multiplication by the multiplication unit220inputted to one input end and the object recognition layer data230x.

As described above, the processing by the multiplication unit220and the addition unit221adds, to the millimeter-wave image data200by the millimeter-wave radar23as the first sensor, region information that is generated according to the object likelihood detected in the process of the object recognition processing based on the image data100by the camera21as the second sensor different from the first sensor.

Here, the addition unit221performs processing of adding the original image to the result of multiplication by the multiplication unit220. For example, in a case where the attention map is represented by a value of 0 or 1 for each pixel, for example, in a case where all the attention maps are 0 in a certain layer image, or in a region of 0 in the attention map, information is lost. Therefore, in the processing by a prediction unit150described later, the recognition processing on the region cannot be performed. In light of the above, the addition unit221adds the object recognition layer data230xbased on the millimeter-wave image data200to avoid a situation in which data is lost in the region.

Returning back toFIG.7, the combined object recognition layer data3100to3106outputted from the combining units300is inputted to the prediction unit150. The prediction unit150performs object recognition processing on the basis of the sets of combined object recognition layer data3100to3106thus inputted, and predicts a class or the like of the recognized object. The result of prediction by the prediction unit150is outputted from the vehicle-exterior-information detection unit10as data indicating the recognition result of the target object, and is conveyed to the integrated control unit12050via, for example, the communication network12001.

3-1. Specific Example

An attention map by the object recognition model40aaccording to the first embodiment is described more specifically with reference toFIGS.9and10.

FIG.9is a schematic diagram for explaining a first example of the attention map according to the object recognition model40aof the first embodiment.

FIG.9illustrates, on the left side, an example of original image data100a.FIG.9illustrates, on the right side, the object recognition layer data230x, the object recognition layer data230x, and the combined object recognition layer data310xfrom top to bottom. Further, from left to right, the object recognition layer data230x, the object recognition layer data230x, and the combined object recognition layer data310xare illustrated so as to correspond to the layer image #1 (object recognition layer data1201), the layer image #2 (object recognition layer data1202), and the layer image #3 (object recognition layer data1203).

Stated differently, the right diagram ofFIG.9illustrates, at the upper part, a feature map indicating the features of the millimeter-wave image data200, and illustrates, at the middle part, an attention map generated on the basis of the features of the image data100. In addition, the lower part of the right diagram ofFIG.9is the combined object recognition layer data310xobtained by combining the feature map based on the millimeter-wave image data200and the attention map based on the image data100by the combining unit300.

Hereinafter, the object recognition layer data230xcorresponding to the layer image #X is referred to as the object recognition layer data230xof the layer image #X. The combined object recognition layer data310xcorresponding to the layer image #X is referred to as the combined object recognition layer data310xof the layer image #X.

Referring toFIG.9, in the object recognition layer data2301of the layer image #1, of the object recognition layer data230x, an object-like recognition result is seen in a part shown in a region23110in the drawing. Further, the layer image #1 shows a state in which an attention map where the object likelihood of regions12110and12111are equal to or greater than the threshold and the regions12110and12111are set as the detection windows is generated. On the other hand, in combined object recognition layer data3101of the layer image #1, an object-like recognition result is seen in a region2301or corresponding to the region23110, and12110′ and12111′ corresponding to the regions12110and12111, respectively.

Similarly, in the layer image #2, in the object recognition layer data2302of the layer image #2, an object-like recognition result is seen in a part shown in a region23111, and the layer image #1 shows a state in which an attention map where the object likelihood of a region12113is equal to or greater than the threshold and the region12113is set as the detection window is generated. On the other hand, in combined object recognition layer data3102of the layer image #2, an object-like recognition result is seen in a region23011′ corresponding to the region23111and12113′ corresponding to the region12113.

As for the layer image #3, in the object recognition layer data2303of the layer image #3, an object-like recognition result is seen in a part shown in a region23112and, in the layer image #1, a region with the object likelihood equal to or greater than the threshold is not detected and no detection window is generated. In combined object recognition layer data3103of the layer image #3, an object-like recognition result is seen in a region23012′ corresponding to the region23112.

Further, in the regions12110and12111and the region12113, white and gray regions correspond to the detection windows. In such a case, for example, a region having a higher degree of white has higher object likelihood. As an example, in the region12113, a region having a high degree of white where a light gray region having a vertical rectangular shape and a dark gray region having a horizontal rectangular shape intersect is a region having the highest object likelihood in the region12113. As described above, the detection window is set, for example, on the basis of the region information including information indicating the corresponding position in the layer image and the value indicating the object likelihood.

As described above, in the layer images #1 and #2, without calculating the object likelihood for the object recognition layer data230xbased on the millimeter-wave image data200, it is possible to generate the combined object recognition layer data310xincluding the region of the detection window based on the image data100while emphasizing a region where the object-like recognition result is seen on the basis of the millimeter-wave image data200.

In addition, since the addition unit221adds the object recognition layer data230xbased on the millimeter-wave image data200, even in a case where no detection window is set in the layer image #2 as in the layer image #3, it is possible to emphasize a region where the object-like recognition result is seen on the basis of the millimeter-wave image data200.

FIG.10is a schematic diagram for explaining a second example of an attention map according to the object recognition model40aof the first embodiment. Since the meaning of each unit inFIG.10is similar to that inFIG.9described above, the description thereof is omitted herein.FIG.10illustrates, on the left side, an example of original image data100b.

Referring toFIG.10, in the object recognition layer data2301of the layer image #1, of the object recognition layer data230x, an object-like recognition result is seen in a part shown in a region23120in the drawing. Further, the layer image #1 shows a state in which an attention map where the object likelihood of regions12120and12121are equal to or greater than the threshold and the regions12120and12121are set as the detection windows is generated. On the other hand, in the combined object recognition layer data3101of the layer image #1, an object-like recognition result is seen in a region23020′ corresponding to the region23120, and12120′ and12121′ corresponding to the regions12120and12121, respectively.

Similarly, in the layer image #2, in the object recognition layer data2302of the layer image #2, an object-like recognition result is seen in a part shown in a region23121, and the layer image #2 shows a state in which an attention map where the object likelihood of a region12122is equal to or greater than the threshold and the region12122is set as the detection window is generated. On the other hand, in the combined object recognition layer data3102of the layer image #2, an object-like recognition result is seen in a region23021′ corresponding to the region23121and12122′ corresponding to the region12122.

In the layer image #3, in the object recognition layer data2303of the layer image #3, an object-like recognition result is seen in a part shown in a region23122, and the layer image #1 shows a state in which an attention map where the object likelihood of the region12123is equal to or greater than the threshold and the region12123is set as the detection window is generated. On the other hand, in the combined object recognition layer data3103of the layer image #3, an object-like recognition result is seen in a region23021′ corresponding to a region23123and12123′ corresponding to the region12123.

As with the first example described above, in the second example, in the layer images #1 to #3, without calculating the object likelihood for the object recognition layer data230xbased on the millimeter-wave image data200, it is possible to generate the combined object recognition layer data310xincluding the region of the detection window based on the image data100while emphasizing a region where the object-like recognition result is seen on the basis of the millimeter-wave image data200.

As described above, according to the first embodiment, even if the millimeter-wave image data200alone is a weak feature, it is possible to improve the performance of the object recognition by emphasizing the feature using the attention map based on the image data100captured by the camera21. In addition, this makes it possible to reduce the load related to the recognition processing in a case where a plurality of different sensors is used.

Note that, in the example ofFIG.7, the sets of combined object recognition layer data310xof the convolutional layers obtained by combining, by the combining unit300, the object recognition layer data120xand the object recognition layer data230xthat have convolutional layers corresponding to each other are inputted to the prediction unit150; however, this is not limited to this example. For example, the combined object recognition layer data310xobtained by combining, by the combining unit300, the object recognition layer data120xand the object recognition layer data230xthat have different convolutional layers (for example, the object recognition layer data1201and the object recognition layer data2302) can be inputted to the prediction unit150. In such a case, it is preferable to make the sizes of the object recognition layer data120xand the object recognition layer data230x, which are to be combined by the combining unit300, the same. Further, it is possible for the combining unit300to combine a part of the sets of object recognition layer data120xand the sets of object recognition layer data230xto generate the combined object recognition layer data310x. At this time, it is possible to select data in which the convolutional layers correspond to each other one by one from among the sets of object recognition layer data120xand the sets of object recognition layer data230xand combine the selected data in the combining unit300, or, alternatively, it is possible to select a plurality of sets of the respective data and combine the selected data in the combining unit300.

4. Second Embodiment

The description goes on to the second embodiment of the present disclosure. In the second embodiment, an example is described in which an attention map is generated in a method different from that of the first embodiment described above.FIG.11is a diagram illustrating a configuration of an example of an object recognition model according to the second embodiment.

InFIG.11, as described above, in an object recognition model40c, an object recognition layer120aperforms convolutional processing on the basis of the image data100to generate the sets of object recognition layer data1200to1206(not illustrated). Here, the object recognition layer120a, for example, doubles the size of the object recognition layer data1206having the deepest convolutional layer and the smallest size to generate an object recognition layer data1221for the next layer.

In such a case, since the newly generated object recognition layer data1221takes over the features of the object recognition layer data1206having the smallest size among the object recognition layers1200to1206, the features of the object recognition layer data1221are weak. Therefore, the object recognition layer120aconnects, to the object recognition layer data1206, the object recognition layer data1205that has the second deepest convolutional layer after the object recognition layer data1206and has a size, for example, twice the size of the object recognition layer data1206and generates the new object recognition layer data1221.

Next, similarly, the object recognition layer120a, for example, doubles the size of the object recognition layer data1221generated and connects the resultant to the corresponding object recognition layer data1205to generate new object recognition layer data1222. As described above, the object recognition layer120aaccording to the second embodiment repeats the processing of, for example, doubling the size of the generated object recognition layer data122xand combining the resultant and the corresponding object recognition layer data120xto newly generate object recognition layer data122x+1.

The object recognition layer120agenerates an attention map on the basis of the object recognition layer data1206,1221,1222,1223,1224,1225, and1226generated by sequentially doubling the size as described above. At this time, the object recognition layer data1226having the largest size is put into the layer image #0 to generate an attention map for the layer image #0. The object recognition layer data1225having the second largest size is put into the layer image #1 to generate an attention map for the layer image #1. Thereafter, the sets of object recognition layer data1224,1223,1222,1221, and1206are put, in order of decreasing size, into the layer images #2, #3, #4, #5, and #6 to generate attention maps for the layer images #2 to #6.

As described above, in the second embodiment, the object recognition layer120agenerates a new attention map by creating and putting the same by machine learning. As a result, it is possible to reduce false positive (FP) caused by a highly reflective object other than the recognition target, such as a guardrail or a curbstone, and to improve the performance of the object recognition by the millimeter-wave image data200alone. On the other hand, in the second embodiment, since the attention map is generated by connecting data to the object recognition layer data1206on which convolution has been performed up to a deep convolutional layer with respect to the image data100, the features of an object whose image is difficult to be caught by the camera21are weakened. For example, it is difficult to recognize an object hidden by water droplets, fog, or the like. In light of the above, it is preferable to switch, depending on the environment, between the method for generating an attention map according to the second embodiment and, for example, the method for generating an attention map according to the first embodiment.

The description goes on to the third embodiment of the present disclosure. In the third embodiment, an example is described in which the sets of object recognition layer data2300to2306based on the millimeter-wave image data200are multiplied by the attention maps (sets of object recognition layer data1200to1206) based on the image data100.FIG.12is a diagram illustrating a configuration of an example of an object recognition model according to the third embodiment.

In an object recognition model40dillustrated inFIG.12, the object recognition layer230generates the sets of object recognition layer data2300to2306on the basis of the millimeter-wave image data200in the same manner as that in the first embodiment. On the other hand, an object recognition layer120bgenerates the sets of object recognition layer data1200to1206and sets of object recognition layer data1200′ to1206′ on the basis of the image data100.

Here, the sets of object recognition layer data1200to1206are data in which parameters are adjusted so that the object recognition is performed by the image data100alone. On the other hand, the sets of object recognition layer data1200′ to1206′ are data in which parameters are adjusted so that the object recognition is performed using both the millimeter-wave image data200and the image data100. For example, in the learning system30described with reference toFIG.4, for identical image data100, learning for the object recognition with the image data100alone and learning for the object recognition with the image data100and the millimeter-wave image data200are executed, and the respective parameters are generated.

Similarly to the first embodiment, the combining units301combine the sets of object recognition layer data1200to1206and the set of object recognition layer data1200′ to1206′ generated in the object recognition layer120band the sets of object recognition layer data2300to2306generated in the object recognition layer230with corresponding sets of data.

FIG.13is a diagram illustrating a configuration of an example of the combining unit301according to the third embodiment. As illustrated inFIG.13, in the combining unit301, a concatenating unit222is added to the configuration of the multiplication unit220and the addition unit221of the combining unit300inFIG.8.

In the combining unit301, the multiplication unit220receives, at one input end, the object recognition layer data120xin which parameters have been adjusted so that the object recognition is performed by the image data100alone, and receives, at the other input end, the object recognition layer data230x. The multiplication unit220calculates, for each pixel, a product of the object recognition layer data120xinputted to one input end thereof and the object recognition layer data230xinputted to the other input end thereof. The result of multiplication by the multiplication unit220is inputted to one input end of the addition unit221. The addition unit221receives, at the other in put end, the object recognition layer data230x. The addition unit221calculates a sum of matrices for the result of multiplication by the multiplication unit220inputted to one input end and the object recognition layer data230x.

The output of the addition unit221is inputted to one input end of the concatenating unit222. The object recognition layer data120x′ in which parameters have been adjusted so that the object recognition is performed using the image data100and the millimeter-wave image data200is inputted to the other input end of the concatenating unit222. The concatenating unit222concatenates the output of the addition unit221and the object recognition layer data120x′.

In the concatenation processing, data of the output of the addition unit221and the object recognition layer data120x′ are listed, and the concatenation processing does not affect each of the output of the addition unit221and the object recognition layer data120x. As a result, the data outputted from the concatenating unit222is data including a feature amount obtained by adding the feature amount of the output of the addition unit221and the feature amount of the object recognition layer data120x.

The combining unit301performs the combining processing, so that an attention map showing the presence or absence of an object with the image data100alone can be generated and that the generated attention map can be multiplied by only the feature amount based on the millimeter-wave image data200. As a result, the feature amount based on the millimeter-wave image data200is limited, and FP can be reduced.

Thus, according to the object recognition model40dof the third embodiment, it is possible to generate an attention map on the basis of the image data100acquired by the camera21alone and perform the object recognition on the basis of the output obtained by combining the camera21and the millimeter-wave radar23.

The description goes on to the fourth embodiment of the present disclosure. In the fourth embodiment, an example is described in which concatenated data of the object recognition layer data120xbased on the image data100and the object recognition layer data230xbased on the millimeter-wave image data200is generated and the object recognition is performed using the concatenated data.

FIG.14is a diagram illustrating a configuration of an example of an object recognition model according to the fourth embodiment. In an object recognition model40eaccording to the fourth embodiment, the sets of concatenated data for performing the object recognition processing already include the object recognition layer data120xand the object recognition layer data230x. Therefore, it is not possible to set a detection window for the object recognition layer data230xbased on the millimeter-wave image data200in the sets of concatenated data. Thus, in the object recognition model40eaccording to the fourth embodiment, processing for reducing the region outside the detection window in the millimeter-wave image data200is performed before the concatenating unit222that concatenates the object recognition layer data120xand the object recognition layer data230x.

The description is provided more specifically. In the object recognition model40eillustrated inFIG.14, the sets of object recognition layer data2300to2306(not illustrated) generated in the object recognition layer230on the basis of the millimeter-wave image data200are inputted to the combining units300. On the other hand, an object recognition layer120cgenerates the sets of object recognition layer data1200to1206on the basis of the image data100, and generates an attention map by superimposing a predetermined number of sets of data of the object recognition layer data1200to1206thus generated. The attention map is inputted to the combining unit300.

Note that, in the example ofFIG.14, the object recognition layer120cgenerates the attention map by using image data123in which, among the sets of object recognition layer data1200to1206, three sets of object recognition layer data1200,1201, and1202in which the convolutional layers are sequentially adjacent are superimposed. This is not limited to the example, and for example, the object recognition layer120ccan generate the attention map by using the image data123in which all the sets of object recognition layer data1200to1206are superimposed. The present invention is not limited thereto, and the object recognition layer120cmay generate the attention map by using image data in which two or four or more sets of adjacent object recognition layer data120xare superimposed. Alternatively, the attention map can be generated by using the image data123in which the plurality of sets of object recognition layer data120xwith the convolutional layers intermittently selected are superimposed, instead of the plurality of sets of object recognition layer data120xwith the convolutional layers adjacent.

Similarly to the description usingFIG.8, the combining unit300obtains a product of the image data123and the sets of object recognition layer data2300to2306with the multiplication unit220, and the addition unit221adds the sets of object recognition layer data2300to2306to the obtained product. The respective sets of combined data obtained by combining the image data123and the sets of object recognition layer data2300to2306by the combining unit300are inputted to one input end of the concatenating unit222.

The sets of object recognition layer data1200to1206generated by the object recognition layer120con the basis of the image data100are inputted to the other input end of the concatenating unit222. The concatenating unit222concatenates the respective sets of combined data inputted to one input end and the sets of object recognition layer data1200to1206inputted to the other input end, and generates concatenated data2420,2421,2422,2423,2424,2425, and2426corresponding to the sets of object recognition layer data1200to12062.

The concatenated data2420to2426outputted from the concatenating unit222is inputted to the prediction unit150.

With such a configuration, it is possible to prevent the influence of the millimeter-wave image data200outside the detection window on the sets of concatenated data2420to2426for the prediction unit150to perform the object recognition. Thus, according to the object recognition model40eof the fourth embodiment, it is possible to generate an attention map on the basis of the image data100acquired by the camera21alone and perform the object recognition on the basis of the output obtained by combining the camera21and the millimeter-wave radar23.

The description goes on to the fifth embodiment of the present disclosure. The object recognition model according to the fifth embodiment is an example in which the image data100one frame before is used as the image data100for generating the attention map.

FIG.15is a diagram illustrating a configuration of an example of an object recognition model according to the fifth embodiment. Note that an object recognition model40fillustrated inFIG.15is an example in which the configuration of the fifth embodiment is applied to the object recognition model40d(seeFIG.12) according to the third embodiment.

In the object recognition model40fillustrated inFIG.15, an object recognition layer120dgenerates, in the same manner as that inFIG.12described above, the sets of object recognition layer data1200to1206on the basis of the image data100(referred to as the image data100of the current frame) acquired as the frame image data of a certain frame (referred to as the current frame) by the camera21in the object recognition layer120. Further, the object recognition layer230generates the sets of object recognition layer data2300to2306on the basis of the millimeter-wave image data200(referred to as the millimeter-wave image data200of the current frame) acquired by the millimeter-wave radar23corresponding to the current frame.

At this time, the sets of object recognition layer data1200to1206generated on the basis of the image data100of the current frame are stored in the memory420. For example, the memory420can be the RAM402illustrated inFIG.5. Here, it has been described that all the sets of object recognition layer data1200to1206are stored in the memory420; however, this is not limited to the example.

For example, only the object recognition layer data1200having the shallowest convolutional layer may be stored in the memory420.

On the other hand, the object recognition layer120dgenerates the attention map on the basis of the sets of object recognition layer data1200to1206that are generated on the basis of the image data100(referred to as the image data100of the past frame101) and stored in the memory420, the image data100being acquired in the past (for example, the immediately preceding frame) for the current frame by the camera21. Here, in a case where only the object recognition layer data1200having the shallowest convolutional layer is stored in the memory420, the convolutional processing can be sequentially performed on the object recognition layer data1200to generate the sets of object recognition layer data1201to1206.

The sets of object recognition layer data1200to1206and the sets of object recognition layer data2300to2306corresponding to the current frame are inputted to the corresponding combining units301. Further, the sets of object recognition layer data1200to1206generated on the basis of the image data100of the past frame101are inputted to the combining units301as the attention maps.

As described withFIG.13, the combining unit301obtains products of the sets of object recognition layer data1200to1206and the sets of object recognition layer data2300to2306with the multiplication unit220, and the addition unit221adds the sets of object recognition layer data2300to2306to the obtained result. The concatenating unit222concatenates the sets of object recognition layer data1200to1206generated on the basis of the image data100of the past frame101to each addition result of the addition unit221.

In this way, the attention map is generated using the data of the past frame101as the image data100, so that one or more convolutional processing in the object recognition layer120ccan be omitted, which improves the processing speed.

The description goes on to the sixth embodiment of the present disclosure. In the first to fifth embodiments described above, the data acquisition unit20includes the camera21and the millimeter-wave radar23as sensors; however, the combination of sensors included in the data acquisition unit20is not limited to this example. In the sixth embodiment, an example of another combination of sensors included in the data acquisition unit20is described.

8-1. First Example

FIG.16is a block diagram of an example illustrating the first example of a vehicle-exterior-information detection unit and a data acquisition unit according to the sixth embodiment. As illustrated inFIG.16, the first example is an example in which a data acquisition unit20aincludes the camera21and a LiDAR24as the sensors. The LiDAR24is a light reflection distance measuring sensor for measuring a distance in a LiDAR method that reflects light emitted from a light source in a target object and measures the distance, and the LiDAR24includes the light source and a light receiving unit.

A signal processing unit13agenerates, for example, three-dimensional group-of-points information on the basis of RAW data outputted from the LiDAR24. A geometric transformation unit14atransforms the three-dimensional group-of-points information generated by the signal processing unit13ainto an image viewed from the same viewpoint as the captured image by the camera21. More specifically, the geometric transformation unit14atransforms the coordinate system of the three-dimensional group-of-points information based on the RAW data outputted from the LiDAR24into the coordinate system of the captured image. The output data of the LiDAR24in which the coordinate system has been transformed into the coordinate system of the captured image by the geometric transformation unit14ais supplied to a recognition processing unit15a. The recognition processing unit15aperforms the object recognition processing using the output data of the LiDAR24in which the coordinate system has been transformed into the coordinate system of the captured image, instead of using the millimeter-wave image data200in the recognition processing unit15described above.

8-2. Second Example

FIG.17is a block diagram of an example illustrating the second example of a vehicle-exterior-information detection unit and a data acquisition unit according to the sixth embodiment. As illustrated inFIG.17, the second example is an example in which a data acquisition unit20bincludes the camera21and an ultrasonic sensor25as the sensors. The ultrasonic sensor25sends a sound wave (ultrasonic wave) in a frequency band higher than an audible frequency band and receives a reflected wave of the ultrasonic wave to measure the distance, and the ultrasonic sensor25includes, for example, a transmitting element for sending an ultrasonic wave and a receiving element for receiving the same. Transmission and reception of ultrasonic waves may be performed by one element. For example, the ultrasonic sensor25can obtain the three-dimensional group-of-points information by repeatedly transmitting and receiving an ultrasonic wave at a predetermined cycle while scanning the transmission direction of the ultrasonic wave.

A signal processing unit13bgenerates, for example, the three-dimensional group-of-points information on the basis of data outputted from the ultrasonic sensor25. A geometric transformation unit14btransforms the three-dimensional group-of-points information generated by the signal processing unit13binto an image viewed from the same viewpoint as the captured image by the camera21. More specifically, the geometric transformation unit14btransforms the coordinate system of the three-dimensional group-of-points information based on the data outputted from the ultrasonic sensor25into the coordinate system of the captured image. The output data of the ultrasonic sensor25in which the coordinate system has been transformed into the coordinate system of the captured image by the geometric transformation unit14bis supplied to a recognition processing unit15b. The recognition processing unit15bperforms the object recognition processing using the output data of the ultrasonic sensor25in which the coordinate system has been transformed into the coordinate system of the captured image, instead of using the millimeter-wave image data200in the recognition processing unit15described above.

8-3. Third Example

FIG.18is a block diagram of an example illustrating the third example of a vehicle-exterior-information detection unit and a data acquisition unit according to the sixth embodiment. As illustrated inFIG.18, the third example is an example in which a data acquisition unit20cincludes the camera21, the millimeter-wave radar23, and the LiDAR24as sensors.

In the vehicle-exterior-information detection unit10illustrated inFIG.18, the millimeter-wave data outputted from the millimeter-wave radar23is inputted to the signal processing unit13. The signal processing unit13performs processing similar to the processing described with reference toFIG.2on the inputted millimeter-wave data to generate a millimeter-wave image. The geometric transformation unit14performs a geometric transformation on the millimeter-wave image generated by the signal processing unit13to transform the millimeter-wave image into an image having the same coordinate system as that of the captured image. The image (referred to as a transformed millimeter-wave image) obtained by transforming the millimeter-wave image by the geometric transformation unit14is supplied to a recognition processing unit15c.

Further, in the vehicle-exterior-information detection unit10, the RAW data outputted from the output of the LiDAR24is inputted to a signal processing unit13c. The signal processing unit13cgenerates, for example, the three-dimensional group-of-points information on the basis of the RAW data inputted from the LiDAR24. A geometric transformation unit14ctransforms the three-dimensional group-of-points information generated by the signal processing unit13cinto an image viewed from the same viewpoint as the captured image by the camera21. The image (referred to as a transformed LiDAR image) obtained by transforming the three-dimensional group-of-points information by the geometric transformation unit14is supplied to the recognition processing unit15c.

The recognition processing unit15ccombines the transformed millimeter-wave image and the transformed LiDAR image inputted from each of the geometric transformation units14and14c, and performs the object recognition processing using the combined image instead of using the millimeter-wave image data200in the recognition processing unit15. Here, the recognition processing unit15ccan concatenate the transformed millimeter-wave image and the transformed LiDAR to integrate the transformed millimeter-wave image and the transformed LiDAR.

8-4. Fourth Example

FIG.19is a block diagram of an example illustrating the fourth example of a vehicle-exterior-information detection unit and a data acquisition unit according to the sixth embodiment. As illustrated inFIG.19, in the fourth example, the data acquisition unit20aincluding the camera21and the millimeter-wave radar23described with reference toFIG.16is applied. On the other hand, in the vehicle-exterior-information detection unit10, the image processing unit12and a geometric transformation unit14dare connected to the output of the camera21, and only the signal processing unit13is connected to the millimeter-wave radar23.

In the vehicle-exterior-information detection unit10, the image processing unit12performs predetermined image processing on the captured image outputted from the camera21. The captured image that has been subjected to the image processing by the image processing unit12is supplied to the geometric transformation unit14d. The geometric transformation unit14dtransforms the coordinate system of the captured image into the coordinate system of the millimeter-wave data outputted from the millimeter-wave radar23. The captured image (referred to as a transformed captured image) obtained by transforming into the coordinate system of the millimeter-wave data by the geometric transformation unit14dis supplied to a recognition processing unit15d.

On the other hand, in the vehicle-exterior-information detection unit10, the millimeter-wave data outputted from the millimeter-wave radar23is inputted to the signal processing unit13. The signal processing unit13performs predetermined signal processing on the inputted millimeter-wave data to generate a millimeter-wave image on the basis of the millimeter-wave data. The millimeter-wave image generated by the signal processing unit13is supplied to the recognition processing unit15d.

The recognition processing unit15dcan use the millimeter-wave image data based on the millimeter-wave image supplied by the signal processing unit13, for example, instead of using the image data100in the recognition processing unit15, and can use the transformed captured image supplied by the geometric transformation unit14dinstead of using the millimeter-wave image data200. For example, in a case where the performance of the millimeter-wave radar23is high and the performance of the camera21is low, the configuration according to the fourth example may be adopted.

8-5. Fifth Example

In the first to fourth examples of the sixth embodiment described above, the camera21and a sensor of a type different from that of the camera21are combined; however, this is not limited to the example. For example, as the fifth example of the sixth embodiment, a combination of cameras21having different characteristics can be applied. As an example, it is possible to apply a combination of the first camera21using a telephoto lens having a narrow angle of view and capable of imaging over a long distance and the second camera21using a wide angle lens having a wide angle of view and capable of imaging a wide range.

8-6. Sixth Example

The description goes on to the fifth example of the sixth embodiment. The fifth example is an example in which the configuration of the recognition processing unit15is switched according to conditions. Note that, for the sake of explanation, the recognition processing unit15(the object recognition model40a) according to the first embodiment is described below as an example.

As an example, the use/non-use of the attention map may be switched according to the weather or the scene. For example, at night and under rainy conditions, it may be difficult to recognize an object in an image captured by the camera21. In such a case, the object recognition is performed using only the output of the millimeter-wave radar23. Further, as another example, it is possible to change how to use the attention map in a case where one of the plurality of sensors included in the data acquisition unit20does not normally operate. For example, in a case where the normal image data100is not outputted due to a malfunction of the camera21or the like, the object recognition is performed at a recognition level similar to that in a case where the attention map is not used. As still another example, in a case where the data acquisition unit20includes three or more sensors, generating a plurality of attention maps is possible on the basis of outputs of the plurality of sensors. In such a case, a plurality of attention maps generated on the basis of the outputs of the sensors may be combined.

Further, the present technology may also be configured as below.

(1) An information processing apparatus comprising:

a recognition processing unit configured to perform recognition processing for recognizing a target object by adding, to an output of a first sensor, region information that is generated according to object likelihood detected in a process of object recognition processing based on an output of a second sensor different from the first sensor.

(2) The information processing apparatus according to the above (1), wherein

the recognition processing unit

uses an object recognition model obtained by machine learning to perform the recognition processing, and

the object recognition model generates the region information in one layer of a first convolutional layer generated on a basis of the output of the second sensor, and adds the region information generated to a layer, corresponding to the layer in which the region information has been generated, of a second convolutional layer generated on a basis of the output of the first sensor.

(3) The information processing apparatus according to the above (1), wherein

the recognition processing unit

uses an object recognition model obtained by machine learning to perform the recognition processing, and

the object recognition model generates the region information in a plurality of layers included in a first convolutional layer generated on a basis of the output of the second sensor, and adds the region information generated to each of a plurality of layers of a second convolutional layer, corresponding one-to-one to each of the plurality of layers in which the region information has been generated, generated on a basis of the output of the first sensor.

(4) The information processing apparatus according to the above (3), wherein

the recognition processing unit

generates the region information in each of a predetermined number of first convolutional layers of the first convolutional layer.

(5) The information processing apparatus according to any one of the above (1) to (4), wherein

the second sensor is an image sensor.

(6) The information processing apparatus according to the above (5), wherein

the first sensor is any one of a millimeter-wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor.

(7) The information processing apparatus according to the above (5), wherein

the first sensor

includes two or more sensors of the image sensor, a millimeter-wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor, and an output obtained by combining outputs of the two or more sensors is used as the output of the first sensor.

(8) The information processing apparatus according to any one of the above (1) to (4), wherein

the first sensor is an image sensor, and

the second sensor is any one of a millimeter-wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor.

(9) The information processing apparatus according to any one of the above (1) to (8), wherein

the recognition processing unit

emphasizes a region, of the output of the first sensor, corresponding to a region in which the object likelihood in the output of the second sensor is equal to or greater than a first threshold.

(10) The information processing apparatus according to any one of the above (1) to (9), wherein

the recognition processing unit

reduces a region, of the output of the first sensor, corresponding to a region in which the object likelihood in the output of the second sensor is less than a second threshold.

(11) The information processing apparatus according to any one of the above (1) to (10), wherein

the recognition processing unit

uses an output one frame before the second sensor to generate the region information.

(12) The information processing apparatus according to any one of the above (1) to (11), wherein

the recognition processing unit

concatenates the output of the second sensor to the region information.

(13) An information processing system comprising:

a first sensor;

a second sensor different from the first sensor; and

an information processing apparatus including a recognition processing unit configured to perform recognition processing for recognizing a target object by adding, to an output of the first sensor, region information that is generated according to object likelihood detected in a process of object recognition processing based on an output of the second sensor.

(14) An information processing program for causing a computer to execute processing comprising:

recognition processing step for performing recognition processing for recognizing a target object by adding, to an output of a first sensor, region information that is generated according to object likelihood detected in a process of object recognition processing based on an output of a second sensor different from the first sensor.

(15) An information processing method comprising:

executing, by a processor,

recognition processing step for performing recognition processing for recognizing a target object by adding, to an output of a first sensor, region information that is generated according to object likelihood detected in a process of object recognition processing based on an output of a second sensor different from the first sensor.

REFERENCE SIGNS LIST