Patent Description:
It has been proposed, in the past, that location information regarding an obstacle that is detected by a millimeter-wave radar be superimposed to be displayed on a captured image using a projection transformation performed with respect to a radar plane and a captured-image plane (for example, refer to Patent Literature <NUM>).

Patent Literature <NUM>: <CIT> <NPL> discloses a robust CNN-based sensor fusion architecture referred to as deep gated information fusion network (DGFN) with a camera- Lidar sensor fusion method performing 2D object detection.

<NPL> discloses a 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization.

<NPL> discloses a road detection framework induced by the inverse depth of LiDAR's point cloud including a fusion of a <NUM>-D LiDAR and a monocular camera, where the <NUM>-D point cloud of LiDAR is projected onto the camera's image frame, to exploit both range and colour information.

<NUM><NUM><NUM> However, Patent Literature <NUM> does not discuss improving the accuracy in recognizing a target object such as a vehicle using a camera and a millimeter-wave radar.

<NUM><NUM><NUM><NUM> The present technology has been made in view of the circumstances described above, and is intended to improve the accuracy in recognizing a target object.

An information processing apparatus according to claim <NUM>.

An information processing method according to claim <NUM>.

A mobile-object control apparatus according to a second aspect of the present technology includes a geometric transformation section that transforms at least one of a captured image or a sensor image to match coordinate systems of the captured image and the sensor image, the captured image being obtained by an image sensor that captures an image of surroundings of a mobile object, the sensor image indicating a sensing result of a sensor of which a sensing range at least partially overlaps a sensing range of the image sensor; an object recognition section that performs processing of recognizing a target object on the basis of the captured image and sensor image of which the coordinate systems have been matched to each other; and a movement controller that controls movement of the mobile object on the basis of a result of the recognition of the target object.

A mobile-object control apparatus according to a third aspect of the present technology includes an image sensor; a sensor of which a sensing range at least partially overlaps a sensing range of the image sensor; a geometric transformation section that transforms at least one of a captured image or a sensor image to match coordinate systems of the captured image and the sensor image, the captured image being obtained by the image sensor, the sensor image indicating a sensing result of the sensor; an object recognition section that performs processing of recognizing a target object on the basis of the captured image and sensor image of which the coordinate systems have been matched to each other; and a movement controller that controls movement on the basis of a result of the recognition of the target object.

In the first aspect of the present technology, at least one of a captured image or a sensor image is transformed to match coordinate systems of the captured image and the sensor image, the captured image being obtained by the image sensor, the sensor image indicating a sensing result of the sensor of which a sensing range at least partially overlaps a sensing range of the image sensor; and processing of recognizing a target object is performed on the basis of the captured image and sensor image of which the coordinate systems have been matched to each other.

In the second aspect of the present technology, at least one of a captured image or a sensor image is transformed to match coordinate systems of the captured image and the sensor image, the captured image being obtained by the image sensor that captures an image of surroundings of a mobile object, the sensor image indicating a sensing result of the sensor of which a sensing range at least partially overlaps a sensing range of the image sensor; processing of recognizing a target object is performed on the basis of the captured image and sensor image of which the coordinate systems have been matched to each other; and movement of the mobile object is controlled on the basis of a result of the recognition of the target object.

In the third aspect of the present technology, at least one of a captured image or a sensor image is transformed to match coordinate systems of the captured image and the sensor image, the captured image being obtained by the image sensor, the sensor image indicating a sensing result of the sensor of which a sensing range at least partially overlaps a sensing range of the image sensor; processing of recognizing a target object is performed on the basis of the captured image and sensor image of which the coordinate systems have been matched to each other; and movement is controlled on the basis of a result of the recognition of the target object.

<NUM><NUM><NUM><NUM> Embodiments for carrying out the present technology are in the claims.

First, the present technology is described with reference to <FIG>.

<NUM><NUM><NUM><NUM> <FIG> is a block diagram illustrating an example of a schematic functional configuration of a vehicle control system <NUM> that is an example of a mobile-object control system to which the present technology is applicable.

<NUM><NUM><NUM><NUM> Note that, when a vehicle <NUM> provided with the vehicle control system <NUM> is to be distinguished from other vehicles, the vehicle provided with the vehicle control system <NUM> will be hereinafter referred to as an own automobile or an own vehicle.

The vehicle control system <NUM> includes an input section <NUM>, a data acquisition section <NUM>, a communication section <NUM>, in-vehicle equipment <NUM>, an output controller <NUM>, an output section <NUM>, a drivetrain controller <NUM>, a drivetrain system <NUM>, a body-related controller <NUM>, a body-related system <NUM>, a storage <NUM>, and an automated driving controller <NUM>. The input section <NUM>, the data acquisition section <NUM>, the communication section <NUM>, the output controller <NUM>, the drivetrain controller <NUM>, the body-related controller <NUM>, the storage <NUM>, and the automated driving controller <NUM> are connected to each other through a communication network <NUM>. For example, the communication network <NUM> includes a bus or a vehicle-mounted communication network compliant with any standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), or FlexRay (registered trademark). Note that the respective structural elements of the vehicle control system <NUM> may be directly connected to each other without using the communication network <NUM>.

Note that the description of the communication network <NUM> will be omitted below when the respective structural elements of the vehicle control system <NUM> communicate with each other through the communication network <NUM>. For example, when the input section <NUM> and the automated driving controller <NUM> communicate with each other through the communication network <NUM>, it will be simply stated that the input section <NUM> and the automated driving controller <NUM> communicate with each other.

The input section <NUM> includes an apparatus used by a person on board to input various pieces of data, instructions, and the like. For example, the input section <NUM> includes an operation device such as a touch panel, a button, a microphone, a switch, and a lever; an operation device with which input can be performed by a method other than a manual operation, such as sound or a gesture; and the like. Alternatively, for example, the input section <NUM> may be externally connected equipment such as a remote-control apparatus using infrared or another radio wave, or mobile equipment or wearable equipment compatible with an operation of the vehicle control system <NUM>. The input section <NUM> generates an input signal on the basis of data, an instruction, or the like input by a person on board, and supplies the generated input signal to the respective structural elements of the vehicle control system <NUM>.

The data acquisition section <NUM> includes various sensors and the like for acquiring data used for a process performed by the vehicle control system <NUM>, and supplies the acquired data to the respective structural elements of the vehicle control system <NUM>.

For example, the data acquisition section <NUM> includes various sensors used to detect, for example, a state of the own automobile. Specifically, for example, the data acquisition section <NUM> includes a gyroscope; an acceleration sensor; an inertial measurement unit (IMU); and a sensor or the like used to detect an amount of operation of an accelerator pedal, an amount of operation of a brake pedal, a steering angle of a steering wheel, the number of revolutions of an engine, the number of revolutions of a motor, a speed of wheel rotation, or the like.

Further, for example, the data acquisition section <NUM> includes various sensors used to detect information regarding the outside of the own automobile. Specifically, for example, the data acquisition section <NUM> includes an image-capturing apparatus such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. Furthermore, for example, the data acquisition section <NUM> includes an environment sensor used to detect weather, a meteorological phenomenon, or the like, and a surrounding-information detection sensor used to detect an object around the own automobile. For example, the environment sensor includes a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and the like. The surrounding-information detection sensor includes an ultrasonic sensor, a radar, LiDAR (light detection and ranging, laser imaging detection and ranging), a sonar, and the like.

Moreover, for example, the data acquisition section <NUM> includes various sensors used to detect the current location of the own automobile. Specifically, for example, the data acquisition section <NUM> includes, for example, a global navigation satellite system (GNSS) receiver that receives a GNSS signal from a GNSS satellite.

Further, for example, the data acquisition section <NUM> includes various sensors used to detect information regarding the inside of a vehicle. Specifically, for example, the data acquisition section <NUM> includes an image-capturing apparatus that captures an image of a driver, a biological sensor that detects biological information of the driver, a microphone that collects sound in the interior of a vehicle, and the like. For example, the biological sensor is provided to a seat surface, the steering wheel, or the like, and detects biological information of a person on board sitting on a seat, or a driver holding the steering wheel.

The communication section <NUM> communicates with the in-vehicle equipment <NUM> as well as various pieces of vehicle-exterior equipment, a server, a base station, and the like, transmits data supplied by the respective structural elements of the vehicle control system <NUM>, and supplies the received data to the respective structural elements of the vehicle control system <NUM>. Note that a communication protocol supported by the communication section <NUM> is not particularly limited. It is also possible for the communication section <NUM> to support a plurality of types of communication protocols.

For example, the communication section <NUM> wirelessly communicates with the in-vehicle equipment <NUM> using a wireless LAN, Bluetooth (registered trademark), near-field communication (NFC), a wireless USB (WUSB), or the like. Further, for example, the communication section <NUM> communicates with the in-vehicle equipment <NUM> by wire using a universal serial bus (USB), a high-definition multimedia interface (HDMI) (registered trademark), a mobile high-definition link (MHL), or the like through a connection terminal (not illustrated) (and a cable if necessary).

Further, for example, the communication section <NUM> communicates with equipment (for example, an application server or a control server) situated in an external network (for example, the Internet, a cloud network, or a carrier-specific network) through a base station or an access point. Furthermore, for example, the communication section <NUM> communicates with a terminal (for example, a terminal of a pedestrian or a store, or a machine-type communication (MTC) terminal) situated near the own automobile, using a peer-to-peer (P2P) technology. Moreover, for example, the communication section <NUM> performs V2X communication such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication between the own automobile and a home, and vehicle-to-pedestrian communication. Further, for example, the communication section <NUM> includes a beacon receiver, receives a radio wave or an electromagnetic wave transmitted from, for example, a radio station installed on a road, and acquires information regarding, for example, the current location, traffic congestion, traffic regulation, or a necessary time.

Examples of the in-vehicle equipment <NUM> include mobile equipment or wearable equipment of a person on board, information equipment that is brought in or attached to the own automobile, and a navigation apparatus that searches for a route to any destination.

The output controller <NUM> controls output of various pieces of information to a person on board of the own automobile or to the outside of the own automobile. For example, the output controller <NUM> generates an output signal that includes at least one of visual information (such as image data) or audio information (such as sound data), supplies the output signal to the output section <NUM>, and thereby controls output of the visual information and the audio information from the output section <NUM>. Specifically, for example, the output controller <NUM> combines pieces of data of images captured by different image-capturing apparatuses of the data acquisition section <NUM>, generates a bird's-eye image, a panoramic image, or the like, and supplies an output signal including the generated image to the output section <NUM>. Further, for example, the output controller <NUM> generates sound data including, for example, a warning beep or a warning message alerting a danger such as collision, contact, or entrance into a dangerous zone, and supplies an output signal including the generated sound data to the output section <NUM>.

The output section <NUM> includes an apparatus capable of outputting the visual information or the audio information to a person on board of the own automobile or to the outside of the own automobile. For example, the output section <NUM> includes a display apparatus, an instrument panel, an audio speaker, headphones, a wearable device such as an eyeglass-type display used to be worn on the person on board, a projector, a lamp, and the like. Instead of an apparatus including a commonly used display, the display apparatus included in the output section <NUM> may be an apparatus, such as a head-up display, a transparent display, or an apparatus including an augmented reality (AR) display function, that displays the visual information in the field of view of a driver.

The drivetrain controller <NUM> generates various control signals, supplies them to the drivetrain system <NUM>, and thereby controls the drivetrain system <NUM>. Further, the drivetrain controller <NUM> supplies the control signals to the structural elements other than the drivetrain system <NUM> as necessary to, for example, notify them of a state of controlling the drivetrain system <NUM>.

The drivetrain system <NUM> includes various apparatuses related to the drivetrain of the own automobile. For example, the drivetrain system <NUM> includes a driving force generation apparatus, such as an internal-combustion engine and a driving motor, that generates driving force, a driving force transmitting mechanism used to transmit the driving force to wheels, a steering mechanism that adjusts the steering angle, a braking apparatus that generates a braking force, an antilock braking system (ABS), an electronic stability control (ESC) system, an electric power steering apparatus, and the like.

The body-related controller <NUM> generates various control signals, supplies them to the body-related system <NUM>, and thereby controls the body-related system <NUM>. Further, the body-related controller <NUM> supplies the control signals to the structural elements other than the body-related system <NUM> as necessary to, for example, notify them of a state of controlling the body-related system <NUM>.

The body-related system <NUM> includes various body-related apparatuses provided to a vehicle body. For example, the body-related system <NUM> includes a keyless entry system, a smart key system, a power window apparatus, a power seat, a steering wheel, an air conditioner, various lamps (such as a headlamp, a tail lamp, a brake lamp, a blinker, and a fog lamp), and the like.

For example, the storage <NUM> includes a read only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, and the like. The storage <NUM> stores therein various programs, data, and the like that are used by the respective structural elements of the vehicle control system <NUM>. For example, the storage <NUM> stores therein map data such as a three-dimensional high-accuracy map, a global map, and a local map. The high-accuracy map is a dynamic map or the like. The global map is less accurate and covers a wider area than the high-accuracy map. The local map includes information regarding the surroundings of the own automobile.

The automated driving controller <NUM> performs control related to automated driving such as autonomous traveling or a driving assistance. Specifically, for example, the automated driving controller <NUM> performs a cooperative control intended to implement a function of an advanced driver-assistance system (ADAS) including collision avoidance or shock mitigation for the own automobile, traveling after a leading vehicle based on a distance between vehicles, traveling while maintaining a vehicle speed, a warning of collision of the own automobile, a warning of deviation of the own automobile from a lane, and the like. Further, for example, the automated driving controller <NUM> performs a cooperative control intended to achieve, for example, automated driving that is autonomous traveling without an operation performed by a driver. The automated driving controller <NUM> includes a detector <NUM>, a self-location estimator <NUM>, a state analyzer <NUM>, a planning section <NUM>, and a movement controller <NUM>.

The detector <NUM> detects various pieces of information necessary to control automated driving. The detector <NUM> includes a vehicle-exterior-information detector <NUM>, a vehicle-interior-information detector <NUM>, and a vehicle state detector <NUM>.

The vehicle-exterior-information detector <NUM> performs a process of detecting information regarding the outside of the own automobile on the basis of data or a signal from each structural element of the vehicle control system <NUM>. For example, the vehicle-exterior-information detector <NUM> performs processes of detecting, recognizing, and tracking an object around the own automobile, and a process of detecting a distance to the object. Examples of the detection-target object include a vehicle, a person, an obstacle, a structure, a road, a traffic light, a traffic sign, and a road sign. Further, for example, the vehicle-exterior-information detector <NUM> performs a process of detecting an environment surrounding the own automobile. Examples of the detection-target surrounding environment include weather, temperature, humidity, brightness, and a road surface condition. The vehicle-exterior-information detector <NUM> supplies data indicating a result of the detection process to, for example, the self-location estimator <NUM>; a map analyzer <NUM>, a traffic-rule recognition section <NUM>, and a state recognition section <NUM> of the state analyzer <NUM>; and an emergency event avoiding section <NUM> of the movement controller <NUM>.

The vehicle-interior-information detector <NUM> performs a process of detecting information regarding the inside of a vehicle on the basis of data or a signal from each structural element of the vehicle control system <NUM>. For example, the vehicle-interior-information detector <NUM> performs processes of authenticating and recognizing a driver, a process of detecting a state of the driver, a process of detecting a person on board, and a process of detecting a vehicle interior environment. Examples of the detection-target state of a driver include a physical condition, a degree of arousal, a degree of concentration, a degree of fatigue, and a direction of a line of sight. Examples of the detection-target vehicle interior environment include temperature, humidity, brightness, and odor. The vehicle-interior-information detector <NUM> supplies data indicating a result of the detection process to, for example, the state recognition section <NUM> of the state analyzer <NUM> and the emergency event avoiding section <NUM> of the movement controller <NUM>.

The vehicle state detector <NUM> performs a process of detecting a state of the own automobile on the basis of data or a signal from each structural element of the vehicle control system <NUM>. Examples of the detection-target state of the own automobile include speed, acceleration, a steering angle, the presence or absence of anomaly and its details, a driving operation state, a position and an inclination of a power seat, a state of a door lock, and states of other pieces of vehicle-mounted equipment. The vehicle state detector <NUM> supplies data indicating a result of the detection process to, for example, the state recognition section <NUM> of the state analyzer <NUM> and the emergency event avoiding section <NUM> of the movement controller <NUM>.

The self-location estimator <NUM> performs a process of estimating a location, a posture, and the like of the own automobile on the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the vehicle-exterior-information detector <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>. Further, the self-location estimator <NUM> generates, as necessary, a local map (hereinafter referred to as a self-location estimation map) used to estimate a self-location. For example, the self-location estimation map is a high-accuracy map using a technology such as simultaneous localization and mapping (SLAM). The self-location estimator <NUM> supplies data indicating a result of the estimation process to, for example, the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>. Further, the self-location estimator <NUM> stores the self-location estimation map in the storage <NUM>.

The state analyzer <NUM> performs a process of analyzing states of the own automobile and its surroundings. The state analyzer <NUM> includes the map analyzer <NUM>, the traffic-rule recognition section <NUM>, the state recognition section <NUM>, and a state prediction section <NUM>.

Using, as necessary, data or signals from the respective structural elements of the vehicle control system <NUM>, such as the self-location estimator <NUM> and the vehicle-exterior-information detector <NUM>, the map analyzer <NUM> performs a process of analyzing various maps stored in the storage <NUM>, and constructs a map including information necessary for an automated driving process. The map analyzer <NUM> supplies the constructed map to, for example, the traffic-rule recognition section <NUM>, the state recognition section <NUM>, and the state prediction section <NUM>, as well as a route planning section <NUM>, a behavior planning section <NUM>, and a movement planning section <NUM> of the planning section <NUM>.

The traffic-rule recognition section <NUM> performs a process of recognizing traffic rules around the own automobile on the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the self-location estimator <NUM>, the vehicle-exterior-information detector <NUM>, and the map analyzer <NUM>. The recognition process makes it possible to recognize a location and a state of a traffic light around the own automobile, the details of traffic control performed around the own automobile, and a travelable lane. The traffic-rule recognition section <NUM> supplies data indicating a result of the recognition process to, for example, the state prediction section <NUM>.

The state recognition section <NUM> performs a process of recognizing a state related to the own automobile on the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the self-location estimator <NUM>, the vehicle-exterior-information detector <NUM>, the vehicle-interior-information detector <NUM>, the vehicle state detector <NUM>, and the map analyzer <NUM>. For example, the state recognition section <NUM> performs a process of recognizing a state of the own automobile, a state of the surroundings of the own automobile, a state of a driver of the own automobile, and the like. Further, the state recognition section <NUM> generates, as necessary, a local map (hereinafter referred to as a state recognition map) used to recognize the state of the surroundings of the own automobile. The state recognition map is, for example, an occupancy grid map.

Examples of the recognition-target state of the own automobile include a location, a posture, and movement (such as speed, acceleration, and a movement direction) of the own automobile, as well as the presence or absence of anomaly and its details. Examples of the recognition-target state of the surroundings of the own automobile include the type and a location of a stationary object around the own automobile; the type, a location, and movement (such as speed, acceleration, and a movement direction) of a moving object around the own automobile; a structure of a road around the own automobile and a condition of the surface of the road; and weather, temperature, humidity, and brightness around the own automobile. Examples of the recognition-target state of a driver include a physical condition, a degree of arousal, a degree of concentration, a degree of fatigue, movement of a line of sight, and a driving operation.

The state recognition section <NUM> supplies data indicating a result of the recognition process (including a state recognition map as necessary) to, for example, the self-location estimator <NUM> and the state prediction section <NUM>. Further, the state recognition section <NUM> stores the state-recognition map in the storage <NUM>.

The state prediction section <NUM> performs a process of predicting a state related to the own automobile on the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM>. For example, the state prediction section <NUM> performs a process of predicting a state of the own automobile, a state of the surroundings of the own automobile, a state of a driver, and the like.

Examples of the prediction-target state of the own automobile include the behavior of the own automobile, the occurrence of anomaly in the own automobile, and a travelable distance of the own automobile. Examples of the prediction-target state of the surroundings of the own automobile include the behavior of a moving object, a change in a state of a traffic light, and a change in environment such as weather around the own automobile. Examples of the prediction-target state of a driver include the behavior and the physical condition of the driver.

The state prediction section <NUM> supplies data indicating a result of the prediction process to, for example, the route planning section <NUM>, the behavior planning section <NUM>, and the movement planning section <NUM> of the planning section <NUM> together with the data from the traffic-rule recognition section <NUM> and the state recognition section <NUM>.

The route planning section <NUM> plans a route to a destination on the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the map analyzer <NUM> and the state prediction section <NUM>. For example, the route planning section <NUM> sets a route from the current location to a specified destination on the basis of a global map. Further, for example, the route planning section <NUM> changes a route as appropriate on the basis of the states of, for example, traffic congestion, an accident, traffic regulation, and a construction, as well as the physical condition of a driver. The route planning section <NUM> supplies data indicating the planned route to, for example, the behavior planning section <NUM>.

On the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the map analyzer <NUM> and the state prediction section <NUM>, the behavior planning section <NUM> plans the behavior of the own automobile in order for the own automobile to travel safely on the route planned by the route planning section <NUM> within a time planned by the route planning section <NUM>. For example, the behavior planning section <NUM> makes plans about, for example, a start to move, a stop, a travel direction (such as a forward movement, a backward movement, a left turn, a right turn, and a change in direction), a lane for traveling, a traveling speed, and passing. The behavior planning section <NUM> supplies data indicating the planned behavior of the own automobile to, for example, the movement planning section <NUM>.

On the basis of data or signals from the respective structural elements of the vehicle control system <NUM>, such as the map analyzer <NUM> and the state prediction section <NUM>, the movement planning section <NUM> plans movement of the own automobile in order to achieve the behavior planned by the behavior planning section <NUM>. For example, the movement planning section <NUM> makes plans about, for example, acceleration, deceleration, and a traveling course. The movement planning section <NUM> supplies data indicating the planned movement of the own automobile to, for example, an acceleration/deceleration controller <NUM> and a direction controller <NUM> of the movement controller <NUM>.

The movement controller <NUM> controls movement of the own automobile. The movement controller <NUM> includes the emergency event avoiding section <NUM>, the acceleration/deceleration controller <NUM>, and the direction controller <NUM>.

On the basis of a result of the detections performed by the vehicle-exterior-information detector <NUM>, the vehicle-interior-information detector <NUM>, and the vehicle state detector <NUM>, the emergency event avoiding section <NUM> performs a process of detecting emergency events such as collision, contact, entrance into a dangerous zone, something unusual in a driver, and anomaly in the vehicle. When the emergency event avoiding section <NUM> detects the occurrence of an emergency event, the emergency event avoiding section <NUM> plans movement of the own automobile such as a sudden stop or a quick turning for avoiding the emergency event. The emergency event avoiding section <NUM> supplies data indicating the planned movement of the own automobile to, for example, the acceleration/deceleration controller <NUM> and the direction controller <NUM>.

The acceleration/deceleration controller <NUM> controls acceleration/deceleration to achieve the movement of the own automobile planned by the movement planning section <NUM> or the emergency event avoiding section <NUM>. For example, the acceleration/deceleration controller <NUM> computes a control target value for a driving force generation apparatus or a braking apparatus to achieve the planned acceleration, the planned deceleration, or the planned sudden stop, and supplies a control instruction indicating the computed control target value to the drivetrain controller <NUM>.

The direction controller <NUM> controls a direction to achieve the movement of the own automobile planned by the movement planning section <NUM> or the emergency event avoiding section <NUM>. For example, the direction controller <NUM> computes a control target value for a steering mechanism to achieve the traveling course planned by the movement planning section <NUM> or the quick turning planned by the emergency event avoiding section <NUM>, and supplies a control instruction indicating the computed control target value to the drivetrain controller <NUM>.

<FIG> illustrates portions of examples of configurations of a data acquisition section 102A that is a first embodiment of the data acquisition section <NUM> in the vehicle control system <NUM> of <FIG>, and a vehicle-exterior-information detector 141A in the vehicle control system <NUM> of <FIG>.

<NUM><NUM><NUM><NUM> The data acquisition section 102A includes a camera <NUM> and a millimeter-wave radar <NUM>. The vehicle-exterior-information detector 141A includes an information processor <NUM>. The information processor <NUM> includes an image processor <NUM>, a signal processor <NUM>, a geometric transformation section <NUM>, and an object recognition section <NUM>.

<NUM><NUM><NUM><NUM> The camera <NUM> includes an image sensor 201A. Any type of image sensor such as a CMOS image sensor or a CCD image sensor can be used as the image sensor 201A. The camera <NUM> (the image sensor 201A) captures an image of a region situated ahead of the vehicle <NUM>, and supplies the obtained image (hereinafter referred to as a captured image) to the image processor <NUM>.

<NUM><NUM><NUM><NUM> The millimeter-wave radar <NUM> performs sensing with respect to the region situated ahead of the vehicle <NUM>, and sensing ranges of the millimeter-wave radar <NUM> and the camera <NUM> at least partially overlap. For example, the millimeter-wave radar <NUM> transmits a transmission signal including a millimeter wave in a forward direction of the vehicle <NUM>, and receives, using a reception antenna, a reception signal that is a signal reflected off an object (a reflector) situated ahead of the vehicle <NUM>. For example, a plurality of reception antennas is arranged at specified intervals in a lateral direction (a width direction) of the vehicle <NUM>. Further, a plurality of reception antennas may also be arranged in the height direction. The millimeter-wave radar <NUM> supplies the signal processor <NUM> with data (hereinafter referred to as millimeter-wave data) that chronologically indicates the intensity of a reception signal received using each reception antenna.

<NUM><NUM><NUM><NUM> The image processor <NUM> performs specified image processing on a captured image. For example, the image processor <NUM> performs processing of reduction in number or filtering processing with respect to a pixel in the captured image according to the image size for which the object recognition section <NUM> can perform processing. The image processor <NUM> reduces the number of pixels in the captured image (reduces the resolution). The image processor <NUM> supplies the captured image with a reduced resolution (hereinafter referred to as a low-resolution image) to the object recognition section <NUM>.

<NUM><NUM><NUM><NUM> The signal processor <NUM> performs specified signal processing on millimeter-wave data to generate a millimeter-wave image that is an image indicating a result of sensing performed by the millimeter-wave radar <NUM>. Note that the signal processor <NUM> generates two types of millimeter-wave images that are, a signal-intensity image and a speed image. The signal-intensity image is a millimeter-wave image indicating a location of each object situated ahead of the vehicle <NUM> and the intensity of a signal reflected off the object (a reception signal). The speed image is a millimeter-wave image indicating a location of each object situated ahead of the vehicle <NUM> and a relative speed of the object with respect to the vehicle <NUM>. The signal processor <NUM> supplies the signal-intensity image and the speed image to the geometric transformation section <NUM>.

<NUM><NUM><NUM><NUM> The geometric transformation section <NUM> performs a geometric transformation on a millimeter-wave image to transform the millimeter-wave image into an image of which a coordinate system is identical to the coordinate system of a captured image. In other words, the geometric transformation section <NUM> transforms a millimeter-wave image into an image (hereinafter referred to as a geometrically transformed millimeter-wave image) obtained as viewed from the same viewpoint as a captured image. More specifically, the geometric transformation section <NUM> transforms the coordinate system of a signal-intensity image and a speed image from the coordinate system of a millimeter-wave image into the coordinate system of a captured image. Note that the signal-intensity image and the speed image on which a geometric transformation has been performed are respectively referred to as a geometrically transformed signal-intensity image and a geometrically transformed speed image. The geometric transformation section <NUM> supplies the geometrically transformed signal-intensity image and the geometrically transformed speed image to the object recognition section <NUM>.

<NUM><NUM><NUM><NUM> The object recognition section <NUM> performs processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of the low-resolution image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image. The object recognition section <NUM> supplies data indicating a result of recognizing the target object to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>. The data indicating a result of recognizing a target object includes, for example, the location and the size of a target object in a captured image, and the type of object.

<NUM><NUM><NUM><NUM> Note that the target object is an object to be recognized by the object recognition section <NUM>, and any object may be set to be the target object. However, it is favorable that an object that includes a portion having a high reflectivity of a transmission signal of the millimeter-wave radar <NUM> be set to be a target object. The case in which the target object is a vehicle is appropriately described below as an example.

<FIG> illustrates an example of a configuration of an object recognition model <NUM> used for the object recognition section <NUM>.

<NUM><NUM><NUM><NUM> The object recognition model <NUM> is a model obtained by machine learning. Specifically, the object recognition model <NUM> is a model obtained by deep learning that is a type of machine learning and uses a deep neural network. More specifically, the object recognition model <NUM> is made up of Single Shot MultiBox Detector (SSD), which is one of the object recognition models using a deep neural network. The object recognition model <NUM> includes a feature-amount extraction section <NUM> and a recognition section <NUM>.

<NUM><NUM><NUM><NUM> The feature-amount extraction section <NUM> includes VGG16 271a to VGG16 271c, which are convolutional layers using a convolutional neural network, and an adder <NUM>.

<NUM><NUM><NUM><NUM> The VGG16 271a extracts a feature amount of a captured image Pa, and generates a feature map (hereinafter referred to as a captured-image feature map) two-dimensionally representing a distribution of the feature amount. The VGG16 271a supplies the captured-image feature map to the adder <NUM>.

<NUM><NUM><NUM><NUM> The VGG16 271b extracts a feature amount of a geometrically transformed signal-intensity image Pb, and generates a feature map (hereinafter referred to as a signal-intensity-image feature map) two-dimensionally representing a distribution of the feature amount. The VGG16 271b supplies the signal-intensity-image feature map to the adder <NUM>.

<NUM><NUM><NUM><NUM> The VGG16 271c extracts a feature amount of a geometrically transformed speed image Pc, and generates a feature map (hereinafter referred to as a speed-image feature map) two-dimensionally representing a distribution of the feature amount. The VGG16 271c supplies the speed-image feature map to the adder <NUM>.

<NUM><NUM><NUM><NUM> The adder <NUM> adds the captured-image feature map, the signal-intensity-image feature map, and the speed-image feature map to generate a combining feature map. The adder <NUM> supplies the combining feature map to the recognition section <NUM>.

<NUM><NUM><NUM><NUM> The recognition section <NUM> includes a convolutional neural network. Specifically, the recognition section <NUM> includes convolutional layers 273a to 273c.

<NUM><NUM><NUM><NUM> The convolutional layer 273a performs a convolution operation on the combining feature map. The convolutional layer 273a performs processing of recognizing a target object on the basis of the combining feature map on which the convolution operation has been performed. The convolutional layer 273a supplies the convolutional layer 273b with the combining feature map on which the convolution operation has been performed.

<NUM><NUM><NUM><NUM> The convolutional layer 273b performs a convolution operation on the combining feature map supplied by the convolutional layer 273a. The convolutional layer 273b performs processing of recognizing the target object on the basis of the combining feature map on which the convolution operation has been performed. The convolutional layer 273a supplies the convolutional layer 273c with the combining feature map on which the convolution operation has been performed.

<NUM><NUM><NUM><NUM> The convolutional layer 273c performs a convolution operation on the combining feature map supplied by the convolutional layer 273b. The convolutional layer 273b performs processing of recognizing the target object on the basis of the combining feature map on which the convolution operation has been performed.

<NUM><NUM><NUM><NUM> The object recognition model <NUM> outputs data indicating a result of the recognition of the target object that is performed by the convolutional layers 273a to 273c.

<NUM><NUM><NUM><NUM> Note that, in order from the convolutional layer 273a, the size (the number of pixels) of a combining feature map becomes smaller, and is smallest in the convolutional layer 273c. Further, if the combining feature map has a larger size, a target object having a small size, as viewed from the vehicle <NUM>, is recognized with a higher degree of accuracy, and if the combining feature map has a smaller size, a target object having a large size, as viewed from the vehicle <NUM>, is recognized with a higher degree of accuracy. Thus, for example, when the target object is a vehicle, a small distant vehicle is easily recognized in a combining feature map having a large size, and a large nearby vehicle is easily recognized in a combining feature map having a small size.

<FIG> is a block diagram illustrating an example of a configuration of a learning system <NUM>.

<NUM><NUM><NUM><NUM> The learning system <NUM> performs learning processing on the object recognition model <NUM> of <FIG>. The learning system <NUM> includes an input section <NUM>, an image processor <NUM>, a correct-answer-data generator <NUM>, a signal processor <NUM>, a geometric transformation section <NUM>, a training data generator <NUM>, and a learning section <NUM>.

<NUM><NUM><NUM><NUM> The input section <NUM> includes various input devices, and is used for, for example, input of data necessary to generate training data, and an operation performed by a user. For example, the input section <NUM> supplies a captured image to the image processor <NUM> when the captured image is input. For example, the input section <NUM> supplies millimeter-wave data to the signal processor <NUM> when the millimeter-wave data is input. For example, the input section <NUM> supplies the correct-answer-data generator <NUM> and the training data generator <NUM> with data indicating an instruction of a user that is input by an operation performed by the user.

<NUM><NUM><NUM><NUM> The image processor <NUM> performs processing similar to the processing performed by the image processor <NUM> of <FIG>. In other words, the image processor <NUM> performs specified image processing on a captured image to generate a low-resolution image. The image processor <NUM> supplies the low-resolution image to the correct-answer-data generator <NUM> and the training data generator <NUM>.

<NUM><NUM><NUM><NUM> The correct-answer-data generator <NUM> generates correct answer data on the basis of the low-resolution image. For example, a user specifies a location of a vehicle in the low-resolution image through the input section <NUM>. The correct-answer-data generator <NUM> generates correct answer data indicating the location of the vehicle in the low-resolution image on the basis of the location of the vehicle that is specified by the user. The correct-answer-data generator <NUM> supplies the correct answer data to the training data generator <NUM>.

<NUM><NUM><NUM><NUM> The signal processor <NUM> performs processing similar to the processing performed by the signal processor <NUM> of <FIG>. In other words, the signal processor <NUM> performs specified signal processing on millimeter-wave data to generate a signal-intensity image and a speed image. The signal processor <NUM> supplies the signal-intensity image and the speed image to the geometric transformation section <NUM>.

<NUM><NUM><NUM><NUM> The geometric transformation section <NUM> performs processing similar to the processing performed by the geometric transformation section <NUM> of <FIG>. In other words, the geometric transformation section <NUM> performs a geometric transformation on the signal-intensity image and the speed image. The geometric transformation section <NUM> supplies the training data generator <NUM> with a geometrically transformed signal-intensity image and a geometrically transformed speed image that are obtained by performing the geometric transformation.

<NUM><NUM><NUM><NUM> The training data generator <NUM> generates training data that includes input data and correct answer data, the input data including the low-resolution image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image. The training data generator <NUM> supplies the training data to the learning section <NUM>.

<NUM><NUM><NUM><NUM> The learning section <NUM> performs learning processing on the object recognition model <NUM> using the training data. The learning section <NUM> outputs the object recognition model <NUM> that has performed learning.

Next, learning processing on an object recognition model that is performed by the learning system <NUM> is described with reference to a flowchart of <FIG>.

<NUM><NUM><NUM><NUM> Note that data used to generate training data is collected before this processing is started. For example, in a state in which the vehicle <NUM> is actually traveling, the camera <NUM> and the millimeter-wave radar <NUM> provided to the vehicle <NUM> perform sensing with respect to a region situated ahead of the vehicle <NUM>. Specifically, the camera <NUM> captures an image of the region situated ahead of the vehicle <NUM>, and stores an obtained captured image in the storage <NUM>. The millimeter-wave radar <NUM> detects an object situated ahead of the vehicle <NUM>, and stores obtained millimeter-wave data in the storage <NUM>. The training data is generated on the basis of the captured image and millimeter-wave data accumulated in the storage <NUM>.

<NUM><NUM><NUM><NUM> In Step S1, the learning system <NUM> generates training data.

<NUM><NUM><NUM><NUM> For example, a user inputs, to the learning system <NUM> and through the input section <NUM>, a captured image and millimeter-wave data that are acquired at substantially the same time. In other words, the captured image and millimeter-wave data obtained by performing sensing at substantially the same point in time are input to the learning system <NUM>. The captured image is supplied to the image processor <NUM>, and the millimeter-wave data is supplied to the signal processor <NUM>.

<NUM><NUM><NUM><NUM> The image processor <NUM> performs image processing, such as processing of reduction in number, with respect to the captured image, and generates a low-resolution image. The image processor <NUM> supplies the low-resolution images to the correct-answer-data generator <NUM> and the training data generator <NUM>.

<NUM><NUM><NUM><NUM> <FIG> illustrates an example of a low-resolution image.

<NUM><NUM><NUM><NUM> The correct-answer-data generator <NUM> generates correct answer data indicating a location of a target object in a low-resolution image on the basis of the location of the target object that is specified by a user through the input section <NUM>. The correct-answer-data generator <NUM> supplies the correct answer data to the training data generator <NUM>.

<NUM><NUM><NUM><NUM> <FIG> illustrates an example of correct answer data generated with respect to the low-resolution image of <FIG>. A region boxed in white indicates a location of a vehicle that is the target object.

<NUM><NUM><NUM><NUM> The signal processor <NUM> performs specified signal processing on millimeter-wave data to estimate a location and a speed of an object off which a transmission signal has been reflected in a region situated ahead of the vehicle <NUM>. The location of the object is represented by, for example, a distance from the vehicle <NUM> to the object, and a direction (an angle) of the object with respect to an optical-axis direction of the millimeter-wave radar <NUM> (a traveling direction of the vehicle <NUM>). Note that, for example, when a transmission signal is radially transmitted, the optical-axis direction of the millimeter-wave radar <NUM> is the same as a direction of the center of a range in which the radial transmission is performed, and when scanning is performed with the transmission signal, the optical-axis direction of the millimeter-wave radar <NUM> is the same as a direction of the center of a range in which the scanning is performed. The speed of the object is represented by, for example, a relative speed of the object with respect to the vehicle <NUM>.

<NUM><NUM><NUM><NUM> The signal processor <NUM> generates a signal-intensity image and a speed image on the basis of a result of estimating the location and the speed of the object. The signal processor <NUM> supplies the signal-intensity image and the speed image to the geometric transformation section <NUM>.

<NUM><NUM><NUM><NUM> <FIG> illustrates an example of a signal-intensity image. An x-axis of the signal-intensity image represents the lateral direction (the width direction of the vehicle <NUM>), and a y-axis of the signal-intensity image represents the optical-axis direction of the millimeter-wave radar <NUM> (the traveling direction of the vehicle <NUM>, the depth direction). In the signal-intensity image, a location of an object situated ahead of the vehicle <NUM>, and a distribution of the reflection intensity of each object, that is, a distribution of the intensity of a reception signal reflected off the object situated ahead of the vehicle <NUM> are given with a bird's-eye view.

Note that the speed image is an image in which the location of the object situated ahead of the vehicle <NUM>, and the distribution of the relative speed of the object are given with a bird's-eye view, as in the case of the signal-intensity image, although an illustration thereof is omitted.

<NUM><NUM><NUM><NUM> The geometric transformation section <NUM> performs a geometric transformation on a signal-intensity image and a speed image, and transforms the signal-intensity image and the speed image into images of which a coordinate system is identical to the coordinate system of a captured image to generate a geometrically transformed signal-intensity image and a geometrically transformed speed image. The geometric transformation section <NUM> supplies the geometrically transformed signal-intensity image and the geometrically transformed speed image to the training data generator <NUM>.

<NUM><NUM><NUM><NUM> <FIG> illustrates examples of a geometrically transformed signal-intensity image and a geometrically transformed speed image. A of <FIG> illustrates the example of a geometrically transformed signal-intensity image, and B of <FIG> illustrates the example of a geometrically transformed speed image. Note that the geometrically transformed signal-intensity image and the geometrically transformed speed image in <FIG> are generated on the basis of millimeter-wave data acquired at substantially the same time as a captured image from which the low-resolution image of <FIG> is generated.

<NUM><NUM><NUM><NUM> In the geometrically transformed signal-intensity image, a portion with a higher signal intensity is brighter, and a portion with a lower signal intensity is darker. In the geometrically transformed speed image, a portion in which a higher relative speed is exhibited is brighter, a portion in which a lower relative speed is exhibited is darker, and a portion in which the relative speed is not detectable (there exists no object) is in solid black.

<NUM><NUM><NUM><NUM> As described above, when a geometric transformation is performed on a millimeter-wave image (a signal-intensity image and a speed image), not only the location of an object in the lateral direction and the depth direction, but also the location of the object in the height direction is given.

<NUM><NUM><NUM><NUM> However, with respect to the millimeter-wave radar <NUM>, the resolution in the height direction becomes low as a distance is increased. Thus, the height of a distant object may be detected to be higher than its actual height.

<NUM><NUM><NUM><NUM> On the other hand, the geometric transformation section <NUM> restricts the height of an object situated at a distance not less than a specified distance when the geometric transformation section <NUM> performs a geometric transformation on a millimeter-wave image. Specifically, when an object situated at a distance not less than a specified distance has a height exhibiting a value greater than a specified upper limit in the geometric transformation on a millimeter-wave image, the geometric transformation section <NUM> restricts the height of the object to the upper limit to perform the geometric transformation. This prevents false recognition performed due to the height of a distant vehicle being detected to be higher than its actual height when the target object is, for example, a vehicle.

<NUM><NUM><NUM><NUM> The training data generator <NUM> generates training data that includes input data and correct answer data, the input data including a captured image, a geometrically transformed signal-intensity image, and a geometrically transformed speed image. The training data generator <NUM> supplies the generated training data to the learning section <NUM>.

<NUM><NUM><NUM><NUM> In Step S2, the learning section <NUM> causes the object recognition model <NUM> to perform learning. Specifically, the learning section <NUM> inputs the input data included in the training data to the object recognition model <NUM>. The object recognition model <NUM> performs processing of recognizing a target object, and outputs data indicating a result of the recognition. The learning section <NUM> compares the result of the recognition performed by the object recognition model <NUM> with the correct answer data, and adjusts, for example, a parameter of the object recognition model <NUM> such that the error is reduced.

<NUM><NUM><NUM><NUM> In Step S3, the learning section <NUM> determines whether the learning is to be continuously performed. For example, when the learning performed by the object recognition model <NUM> has not come to an end, the learning section <NUM> determines that the learning is to be continuously performed, and the process returns to Step S1.

<NUM><NUM><NUM><NUM> Thereafter, the processes of Steps S1 to S3 are repeatedly performed until it is determined, in Step S3, that the learning is to be terminated.

<NUM><NUM><NUM><NUM> On the other hand, the learning section <NUM> determines, in Step S3, that the learning performed by the object recognition model <NUM> is to be terminated when, for example, the learning has come to an end, and the learning processing performed on the object recognition model is terminated.

<NUM><NUM><NUM><NUM> As described above, the object recognition model <NUM> that has performed learning is generated.

<NUM><NUM><NUM><NUM> Note that <FIG> illustrates an example of a result of recognition performed by the object recognition model <NUM> that has performed learning only using millimeter-wave data without using a captured image.

<NUM><NUM><NUM><NUM> A of <FIG> illustrates an example of a geometrically transformed signal-intensity image generated on the basis of millimeter-wave data.

<NUM> B of <FIG> illustrates an example of a result of recognition performed by the object recognition model <NUM>. Specifically, the viewpoint-transformed intensity image of A of <FIG> is superimposed on a captured image that is acquired at substantially the same time as the millimeter-wave data from which the viewpoint-transformed intensity image of A of <FIG> is generated, and a boxed region indicates a location in which a vehicle of the target object is recognized.

<NUM> As illustrated in this example, the object recognition model <NUM> also makes it possible to recognize a vehicle of the target object with a degree of accuracy not less than a specified degree of accuracy when only millimeter-wave data (a geometrically transformed signal-intensity image and a geometrically transformed speed image) is used.

Next, target-object recognition processing performed by the vehicle <NUM> is described with reference to a flowchart of <FIG>.

<NUM> This processing is started when, for example, an operation for activating the vehicle <NUM> to start driving is performed, that is, when, for example, an ignition switch, a power switch, a start switch, or the like of the vehicle <NUM> is turned on. Further, this processing is terminated when, for example, an operation for terminating the driving of the vehicle <NUM> is performed, that is, when, for example, the ignition switch, the power switch, the start switch, or the like of the vehicle <NUM> is turned off.

<NUM> In Step S101, the camera <NUM> and the millimeter-wave radar <NUM> perform sensing with respect to a region situated ahead of the vehicle <NUM>.

<NUM> Specifically, the camera <NUM> captures an image of a region situated ahead of the vehicle <NUM>, and supplies the obtained captured image to the image processor <NUM>.

<NUM> The millimeter-wave radar <NUM> transmits a transmission signal in a forward direction of the vehicle <NUM>, and receives, using a plurality of reception antennas, reception signals that are signals reflected off an object situated ahead of the vehicle <NUM>. The millimeter-wave radar <NUM> supplies the signal processor <NUM> with millimeter-wave data that chronologically indicates the intensity of the reception signal received using each reception antenna.

<NUM> In Step S102, the image processor <NUM> performs preprocessing on the captured image. Specifically, the image processor <NUM> performs, for example, processing of reduction in number with respect to the captured image to generate a low-resolution image, and supplies the low-resolution image to the object recognition section <NUM>.

<NUM> In Step S103, the signal processor <NUM> generates a millimeter-wave image. Specifically, the signal processor <NUM> performs processing similar to the processing performed by the signal processor <NUM> in Step S1 of <FIG> to generate a signal-intensity image and a speed image on the basis of the millimeter-wave data. The signal processor <NUM> supplies the signal-intensity image and the speed image to the geometric transformation section <NUM>.

<NUM> In Step S104, the geometric transformation section <NUM> performs a geometric transformation on the millimeter-wave image. Specifically, the geometric transformation section <NUM> performs processing similar to the processing performed by the geometric transformation section <NUM> in Step S1 of <FIG> to transform the signal-intensity image and the speed image into a geometrically transformed signal-intensity image and a geometrically transformed speed image. The geometric transformation section <NUM> supplies the geometrically transformed signal-intensity image and the geometrically transformed speed image to the object recognition section <NUM>.

<NUM> In Step S105, the object recognition section <NUM> performs processing of recognizing a target object on the basis of the low-resolution image, and the millimeter-wave image on which the geometric transformation has been performed. Specifically, the object recognition section <NUM> inputs, to the object recognition model <NUM>, input data that includes the low-resolution image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image. The object recognition model <NUM> performs processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of the input data.

<NUM> <FIG> illustrates an example of a recognition result when the target object is a vehicle. A of <FIG> illustrates an example of a captured image. B of <FIG> illustrates an example of a result of recognizing a vehicle. In B of <FIG>, a region in which the vehicle is recognized is boxed.

<NUM> The object recognition section <NUM> supplies data indicating a result of recognizing the target object to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>.

<NUM> On the basis of, for example, the result of recognizing the target object, the self-location estimator <NUM> performs a process of estimating a location, a posture, and the like of the vehicle <NUM>.

<NUM> On the basis of, for example, the result of recognizing the target object, the map analyzer <NUM> performs a process of analyzing various maps stored in the storage <NUM>, and constructs a map including information necessary for an automated driving process.

<NUM> On the basis of, for example, the result of recognizing the target object, the traffic-rule recognition section <NUM> performs a process of recognizing traffic rules around the vehicle <NUM>.

<NUM> On the basis of, for example, the result of recognizing the target object, the state recognition section <NUM> performs a process of recognizing a state of the surroundings of the vehicle <NUM>.

When the emergency event avoiding section <NUM> detects the occurrence of an emergency event on the basis of, for example, the result of recognizing the target object, the emergency event avoiding section <NUM> plans movement of the vehicle <NUM> such as a sudden stop or a quick turning for avoiding the emergency event. <NUM><NUM>.

<NUM> Thereafter, the process returns to Step S101, and the processes of and after Step S101 are performed.

<NUM> The accuracy in recognizing a target object situated ahead of the vehicle <NUM> can be improved as described above.

<NUM> Specifically, when the processing of recognizing a target object is performed only using a captured image, the accuracy in recognizing a target object is reduced due to bad weather (such as rain or fog), at night, or in poor visual conditions due to, for example, an obstacle. On the other hand, when millimeter-wave radar is used, the accuracy in recognizing a target object is hardly reduced due to bad weather, at night, or in poor visual conditions due to, for example, an obstacle. Thus, the camera <NUM> and the millimeter-wave radar <NUM> (a captured image and millimeter-wave data) are fused to perform processing of recognizing a target object, and this makes it possible to compensate for a drawback caused when only a captured image is used. This results in improving the accuracy in recognition.

<NUM> Further, as illustrated in A of <FIG>, a captured image is represented by a coordinate system defined by an x-axis and a z-axis. The x-axis represents the lateral direction (the width direction of the vehicle <NUM>), and the z-axis represents the height direction. On the other hand, as illustrated in B of <FIG>, the millimeter-wave image is represented by a coordinate system defined by an x-axis and a y-axis. The x-axis is similar to the x-axis of the coordinate system of the captured image. Note that the x-axis extends in line with a direction in which a transmission signal of the millimeter-wave radar <NUM> is spread out in a planar manner. The y-axis represents the optical-axis direction of the millimeter-wave radar <NUM> (the traveling direction of the vehicle <NUM>, the depth direction).

<NUM> When there is a difference in coordinate system between a captured image and a millimeter-wave image, as described above, this results in being difficult to understand a correlation between the captured image and the millimeter-wave image. For example, it is difficult to match each pixel of a captured image with a reflection point in a millimeter-wave image (a point at which the intensity of a reception signal is high). Thus, when the object recognition model <NUM> is caused to perform learning by deep learning using the captured image and the millimeter-wave image of A of <FIG>, the difficulty in learning will be increased, and this may result in a reduction in the accuracy in learning.

<NUM> On the other hand, according to the present technology, a geometric transformation is performed on a millimeter-wave image (a signal-intensity image and a speed image) to obtain an image (a geometrically transformed signal-intensity image and a geometrically transformed speed image) of which a coordinate system has been matched to the coordinate system of a captured image, and the object recognition model <NUM> is caused to perform learning using the obtained image. This results in facilitating matching of each pixel of the captured image with a reflection point in the millimeter-wave image, and in improving the accuracy in learning. Further, in actual processing of recognizing a vehicle, the use of a geometrically transformed signal-intensity image and a geometrically transformed speed image results in improving the accuracy in recognizing a target object.

Next, the present technology is described with reference to <FIG>.

<FIG> illustrates an example of a configuration of a vehicle-exterior-information detector 141B that is a second embodiment of the vehicle-exterior-information detector <NUM> of the vehicle control system <NUM> of <FIG>. Note that a portion in the figure that corresponds to a portion in <FIG> is denoted by the same reference numeral as <FIG>, and a description thereof is omitted as appropriate.

<NUM> The vehicle-exterior-information detector 141B includes an information processor <NUM>. The information processor <NUM> is similar to the information processor <NUM> of <FIG> in including the signal processor <NUM> and the geometric transformation section <NUM>. On the other hand, the information processor <NUM> is different from the information processor <NUM> in including an image processor <NUM> and an object recognition section <NUM> instead of the image processor <NUM> and the object recognition section <NUM>, and in that a combiner <NUM> is added. The object recognition section <NUM> includes an object recognition section 431a and an object recognition section 431b.

<NUM> As in the case of the image processor <NUM>, the image processor <NUM> generates a low-resolution image on the basis of a captured image. The image processor <NUM> supplies the low-resolution image to the object recognition section 431a.

<NUM> Further, the image processor <NUM> cuts out a portion of the captured image according to the image size for which the object recognition section 431b can perform processing. The image processor <NUM> supplies the image cut out of the captured image (hereinafter referred to as a crop image) to the object recognition section 431b.

<NUM> As in the case of the object recognition section <NUM> of <FIG>, the object recognition model <NUM> of <FIG> is used for the object recognition section 431a and the object recognition section 431b.

As in the case of the object recognition section <NUM> of <FIG>, <NUM> the object recognition section 431a performs processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of the low-resolution image, a geometrically transformed signal-intensity image, and a geometrically transformed speed image. The object recognition section 431a supplies the combiner <NUM> with data indicating a result of the processing of recognizing the target object.

<NUM> The object recognition section 431b performs processing of recognizing the target object situated ahead of the vehicle <NUM> on the basis of the crop image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image. The object recognition section 431b supplies the combiner <NUM> with data indicating a result of the processing of recognizing the target object.

<NUM> Note that learning processing is performed with respect to each of the object recognition model <NUM> used for the object recognition section 431a and the object recognition model <NUM> used for the object recognition section 431b using different training data, although a detailed description thereof is omitted. Specifically, the object recognition model <NUM> used for the object recognition section 431a is caused to perform learning using training data that includes input data and correct answer data, the input data including the low-resolution image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image, the correct answer data being generated on the basis of the low-resolution image. On the other hand, the object recognition model <NUM> used for the object recognition section 431b is caused to perform learning using training data that includes input data and correct answer data, the input data including the crop image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image, the correct answer data being generated on the basis of the crop image.

<NUM> The combiner <NUM> combines the result of recognition of the target object that is performed by the object recognition section 431a, and the result of recognition of the target object that is performed by the object recognition section 431b. The combiner <NUM> supplies data indicating a result of recognizing the target object that is obtained by the combining to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>.

Next, the target-object recognition processing is described with reference to a flowchart of <FIG>.

<NUM> In Step S201, sensing is performed on a region situated ahead of the vehicle <NUM>, as in the process of Step S101 of <FIG>.

<NUM> In Step S202, the image processor <NUM> performs preprocessing on a captured image. Specifically, the image processor <NUM> performs processing similar to the processing performed in Step S102 of <FIG> to generate a low-resolution image on the basis of the captured image. The image processor <NUM> supplies the low-resolution image to the object recognition section 431a.

<NUM> Further, the image processor <NUM> detects, for example, a vanishing point of a road in the captured image. From the captured image, the image processor <NUM> cuts out an image in a rectangular region that has a specified size and is centered at the vanishing point. The image processor <NUM> supplies a crop image obtained by the cutout to the object recognition section 431b.

<NUM> <FIG> illustrates an example of a relationship between a captured image and a crop image. Specifically, A of <FIG> illustrates an example of a captured image. Further, a rectangular region, in the captured image, that is boxed with a dotted line in B of <FIG> is cut out as a crop image, the rectangular region having a specified size and being centered at a vanishing point of a road.

<NUM> In Step S203, a millimeter-wave image, that is, a signal-intensity image and a speed image, is generated, as in the process of Step S103 of <FIG>.

<NUM> In Step S204, a geometric transformation is performed on the millimeter-wave images, as in the process of Step S104 of <FIG>. This results in generating a geometrically transformed signal-intensity image and a geometrically transformed speed image. The geometric transformation section <NUM> supplies the geometrically transformed signal-intensity image and the geometrically transformed speed image to the object recognition section 431a and the object recognition section 431b.

<NUM> In Step S205, the object recognition section 431a performs processing of recognizing a target object on the basis of a low-resolution image, and the millimeter-wave image on which the geometric transformation has been performed. Specifically, <NUM>7the object recognition section 431a performs processing similar to the processing performed in Step S105 of <FIG> to perform processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of the low-resolution image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image. The object recognition section 431a supplies the combiner <NUM> with data indicating a result of the processing of recognizing the target object.

<NUM> In Step S206, the object recognition section 431b performs processing of recognizing a vehicle on the basis of the crop image, and the millimeter-wave image on which the geometric transformation has been performed. Specifically, the object recognition section <NUM> inputs, to the object recognition model <NUM>, input data that includes the crop image, the geometrically transformed signal-intensity image, and the geometrically transformed speed image. The object recognition model <NUM> performs processing of recognizing the target object situated ahead of the vehicle <NUM> on the basis of the input data. The object recognition section 431b supplies the combiner <NUM> with data indicating a result of recognizing the target object.

<NUM> In Step S207, the combiner <NUM> combines the results of recognizing the target object. Specifically, the combiner <NUM> combines the result of the recognition of the target object that is performed by the object recognition section 431a and the result of the recognition of the target object that is performed by the object recognition section 431b. The combiner <NUM> supplies data indicating a result of recognizing the target object that is obtained by the combining to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>.

When the emergency event avoiding section <NUM> detects the occurrence of an emergency event on the basis of, for example, the result of recognizing the target object, the emergency event avoiding section <NUM> plans movement of the vehicle <NUM> such as a sudden stop or a quick turning for avoiding the emergency event. <NUM><NUM><NUM>.

<NUM> Thereafter, the process returns to Step S201, and the processes of and after Step S201 are performed.

<NUM> The accuracy in recognizing a target object situated ahead of the vehicle <NUM> can be improved as described above. Specifically, the use of a low-resolution image instead of a captured image results in a reduction in the accuracy in recognizing a distant target object in particular. However, when the processing of recognizing a target object is performed using a high-resolution crop image obtained by cutting out an image of a region around a vanishing point of a road, this makes it possible to improve the accuracy in recognizing a distant vehicle situated around the vanishing point in the case in which the target object is a vehicle.

<FIG> illustrates examples of configurations of a data acquisition section 102B that is a second embodiment of the data acquisition section <NUM> in the vehicle control system <NUM> of <FIG>, and a vehicle-exterior-information detector 141C of the vehicle-exterior-information detector <NUM> in the vehicle control system <NUM> of <FIG>. Note that a portion in the figure that corresponds to a portion in <FIG> is denoted by the same reference numeral as <FIG>, and a description thereof is omitted as appropriate.

<NUM> The data acquisition section 102B is similar to the data acquisition section 102A of <FIG> in including the camera <NUM> and the millimeter-wave radar <NUM>, and is different from the data acquisition section 102A of <FIG> in including LiDAR <NUM>.

<NUM> The vehicle-exterior-information detector 141C includes an information processor <NUM>. The information processor <NUM> is similar to the information processor <NUM> of <FIG> in including the image processor <NUM>, the signal processor <NUM>, and the geometric transformation section <NUM>. On the other hand, the information processor <NUM> is different from the information processor <NUM> in including an object recognition section <NUM> instead of the object recognition section <NUM>, and in that a signal processor <NUM> and a geometric transformation section <NUM> are added.

<NUM> The LiDAR <NUM> performs sensing with respect to a region situated ahead of the vehicle <NUM>, and sensing ranges of the LiDAR <NUM> and the camera <NUM> at least partially overlap. For example, the LiDAR <NUM> performs scanning with a laser pulse in the lateral direction and in the height direction with respect to the region situated ahead of the vehicle <NUM>, and receives reflected light that is a reflection of the laser pulse. The LiDAR <NUM> calculates a distance to an object situated ahead of the vehicle <NUM> on the basis of the time taken to receive the reflected light, and on the basis of a result of the calculation, the LiDAR <NUM> generates three-dimensional group-of-points data (point cloud) that indicates a shape and a location of the object situated ahead of the vehicle <NUM>. The LiDAR <NUM> supplies the group-of-points data to the signal processor <NUM>.

<NUM> The signal processor <NUM> performs specified signal processing (for example, interpolation processing or processing of reduction in number) with respect to the group-of-points data, and supplies the geometric transformation section <NUM> with the group-of-points data on which the signal processing has been performed.

<NUM> The geometric transformation section <NUM> performs a geometric transformation on the group-of-points data to generate a two-dimensional image (hereinafter referred to as two-dimensional group-of-points data) of which a coordinate system is identical to the coordinate system of a captured image. The geometric transformation section <NUM> supplies the two-dimensional group-of-points data to the object recognition section <NUM>.

<NUM> The object recognition section <NUM> performs processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of a low-resolution image, a geometrically transformed signal-intensity image, a geometrically transformed speed image, and the two-dimensional group-of-points data. The object recognition section <NUM> supplies data indicating a result of recognizing the target object to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>. The data indicating a result of recognizing a target object includes, for example, the location and the speed of a target object in a three-dimensional space, in addition to the location and the size of the target object in a captured image, and the type of object.

<NUM> Note that an object recognition model having a configuration similar to the configuration of, for example, the object recognition model <NUM> of <FIG> is used for the object recognition section <NUM>, although a detailed description thereof is omitted. However, one VGG16 is added for two-dimensional group-of-points data. Then, the object recognition model for the object recognition section <NUM> is caused to perform learning using training data that includes input data and correct answer data, the input data including a low-resolution image, a geometrically transformed signal-intensity image, a geometrically transformed speed image, and two-dimensional group-of-points data, the correct answer data being generated on the basis of the low-resolution image.

<NUM> As described above, the addition of the LiDAR <NUM> results in further improving the accuracy in recognizing a target object.

<FIG> illustrates an example of a configuration of a vehicle-exterior-information detector 141D of the vehicle-exterior-information detector <NUM> in the vehicle control system <NUM> of <FIG>. Note that a portion in the figure that corresponds to a portion in <FIG> is denoted by the same reference numeral as <FIG>, and a description thereof is omitted as appropriate.

<NUM> The vehicle-exterior-information detector 141D includes an information processor <NUM>. The information processor <NUM> is similar to the information processor <NUM> of <FIG> in including the image processor <NUM> and the signal processor <NUM>. On the other hand, the information processor <NUM> is different from the information processor <NUM> in including an object recognition section <NUM> instead of the object recognition section <NUM>, and in that a geometric transformation section <NUM> is added and the geometric transformation section <NUM> has been removed.

<NUM> The geometric transformation section <NUM> performs a geometric transformation on a low-resolution image supplied by the image processor <NUM> to transform the low-resolution image into an image (hereinafter referred to as a geometrically transformed low-resolution image) of which a coordinate system is identical to the coordinate system of a millimeter-wave image output by the signal processor <NUM>. For example, the geometric transformation section <NUM> transforms the low-resolution image into an image of a bird's-eye view. The geometric transformation section <NUM> supplies the geometrically transformed low-resolution image to the object recognition section <NUM>.

<NUM> The object recognition section <NUM> performs processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of the geometrically transformed low-resolution image, and a signal-intensity image and a speed image that are supplied by the signal processor <NUM>. The object recognition section <NUM> supplies data indicating a result of recognizing the target object to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>.

<NUM> As described above, a captured image may be transformed into an image of which a coordinate system is identical to the coordinate system of the millimeter-wave image to perform the processing of recognizing an object.

<FIG> illustrates an example of a configuration of a vehicle-exterior-information detector 141E of the vehicle-exterior-information detector <NUM> in the vehicle control system <NUM> of <FIG>. Note that a portion in the figure that corresponds to a portion in <FIG> is denoted by the same reference numeral as <FIG>, and a description thereof is omitted as appropriate.

<NUM> The vehicle-exterior-information detector 141E includes an information processor <NUM>. The information processor <NUM> is similar to the information processor <NUM> of <FIG> in including the image processor <NUM>, the signal processor <NUM>, and the signal processor <NUM>. On the other hand, the information processor <NUM> is different from the information processor <NUM> in including a geometric transformation section <NUM> and an object recognition section <NUM> instead of the geometric transformation section <NUM> and the object recognition section <NUM>, and in that a geometric transformation section <NUM> is added and the geometric transformation section <NUM> has been removed.

<NUM> The geometric transformation section <NUM> performs a geometric transformation on a low-resolution image supplied by the image processor <NUM> to transform the low-resolution image into three-dimensional group-of-points data (hereinafter referred to as image data of group of points) of which a coordinate system is identical to the coordinate system of group-of-points data output by the signal processor <NUM>. The geometric transformation section <NUM> supplies the image data of group of points to the object recognition section <NUM>.

<NUM> The geometric transformation section <NUM> performs a geometric transformation on a signal-intensity image and a speed image that are supplied by the signal processor <NUM> to transform the signal-intensity image and the speed image into pieces of three-dimensional group-of-points data (hereinafter referred to as signal-intensity data of group of points and speed data of group of points) of which a coordinate system is identical to the coordinate system of the group-of-points data output by the signal processor <NUM>. The geometric transformation section <NUM> supplies the signal-intensity data of group of points and the speed data of group of points to the object recognition section <NUM>.

<NUM> The object recognition section <NUM> performs processing of recognizing a target object situated ahead of the vehicle <NUM> on the basis of the image data of group of points, the signal-intensity data of group of points, the speed data of group of points, and group-of-points data that is generated by sensing performed by the LiDAR <NUM> and supplied by the signal processor <NUM>. The object recognition section <NUM> supplies data indicating a result of recognizing the target object to, for example, the self-location estimator <NUM>; the map analyzer <NUM>, the traffic-rule recognition section <NUM>, and the state recognition section <NUM> of the state analyzer <NUM>; and the emergency event avoiding section <NUM> of the movement controller <NUM>.

As described above, a captured image and a millimeter-wave image may be transformed into pieces of group-of-points data to perform the processing of recognizing an object.

<FIG> illustrates examples of configurations of a data acquisition section 102C of the data acquisition section <NUM> in the vehicle control system <NUM> of <FIG>, and a vehicle-exterior-information detector 141F of the vehicle-exterior-information detector <NUM> in the vehicle control system <NUM> of <FIG>. Note that a portion in the figure that corresponds to a portion in <FIG> is denoted by the same reference numeral as <FIG>, and a description thereof is omitted as appropriate.

<NUM> The data acquisition section 102C is similar to the data acquisition section 102A of <FIG> in including the camera <NUM>, and is different from the data acquisition section 102A in including a millimeter-wave radar <NUM> instead of the millimeter-wave radar <NUM>.

<NUM> The vehicle-exterior-information detector 141F includes an information processor <NUM>. The information processor <NUM> is similar to the information processor <NUM> of <FIG> in including the image processor <NUM>, the geometric transformation section <NUM>, and the object recognition section <NUM>, and is different from the information processor <NUM> in that the signal processor <NUM> has been removed.

<NUM> The millimeter-wave radar <NUM> includes a signal processor <NUM> that includes a function equivalent to the function of the signal processor <NUM>. The signal processor <NUM> performs specified signal processing on millimeter-wave data to generate two types of millimeter-wave images that are a signal-intensity image and a speed image that indicate a result of sensing performed by the millimeter-wave radar <NUM>. The signal processor <NUM> supplies the signal-intensity image and the speed image to the geometric transformation section <NUM>.

<NUM> As described above, millimeter-wave data may be transformed into millimeter-wave images in the millimeter-wave radar <NUM>.

<FIG> illustrates examples of configurations of a data acquisition section 102D of the data acquisition section <NUM> in the vehicle control system <NUM> of <FIG>, and a vehicle-exterior-information detector <NUM> of the vehicle-exterior-information detector <NUM> in the vehicle control system <NUM> of <FIG>. Note that a portion in the figure that corresponds to a portion in <FIG> is denoted by the same reference numeral as <FIG>, and a description thereof is omitted as appropriate.

The data acquisition section 102D is similar to the data acquisition section 102C of <FIG> in including the camera <NUM>, and is different from the data acquisition section 102C in including a millimeter-wave radar <NUM> instead of the millimeter-wave radar <NUM>.

<NUM> The vehicle-exterior-information detector <NUM> includes an information processor <NUM>. The information processor <NUM> is similar to the information processor <NUM> of <FIG> in including the image processor <NUM> and the object recognition section <NUM>, and is different from the information processor <NUM> in that the geometric transformation section <NUM> has been removed.

<NUM> In addition to the signal-processor <NUM>, the millimeter-wave radar <NUM> includes a geometric transformation section <NUM> that includes a function equivalent to the function of the geometric transformation section <NUM>.

<NUM> The geometric transformation section <NUM> transforms the coordinate system of a signal-intensity image and a speed image from the coordinate system of a millimeter-wave image to the coordinate system of a captured image, and supplies the object recognition section <NUM> with a geometrically transformed signal-intensity image and a geometrically transformed speed image that are obtained by the geometric transformation.

<NUM> As described above, millimeter-wave data may be transformed into millimeter-wave images and a geometric transformation may be performed on the millimeter-wave images in the millimeter-wave radar <NUM>.

Modifications of the present technology described above are described below.

<NUM> The example in which a vehicle is a recognition target has been primarily described above. However, as described above, any object other than a vehicle may be a recognition target. For example, it is sufficient if leaning processing is performed on the object recognition model <NUM> using training data that includes correct answer data indicating a location of a target object to be recognized.

<NUM> Further, the present technology is also applicable to the case of recognizing a plurality of types of objects. For example, it is sufficient if leaning processing is performed on the object recognition model <NUM> using training data that includes correct answer data indicating a location and a label (the type of target object) of each target object.

<NUM> Furthermore, when a captured image has a size for which the object recognition section <NUM> can satisfactorily perform processing, the captured image may be directly input to the object recognition model <NUM> to perform processing of recognizing a target object.

<NUM> The example of recognizing a target object situated ahead of the vehicle <NUM> has been described above. However, the present technology is also applicable to the case of recognizing a target object situated around the vehicle <NUM> in another direction, as viewed from the vehicle <NUM>.

<NUM> Further, the present technology is also applicable to the case of recognizing a target object around a mobile object other than a vehicle. For example, it is conceivable that the present technology could be applied to a mobile object such as a motorcycle, a bicycle, personal mobility, an airplane, a ship, construction machinery, and agricultural machinery (a tractor). Further, examples of the mobile object to which the present technology is applicable also include a mobile object, such as a drone and a robot, that is remotely operated by a user without the user getting on the mobile object.

<NUM> Furthermore, the present technology is also applicable to the case of performing processing of recognizing a target object at a fixed place such as a monitoring system.

<NUM> Moreover, the object recognition model <NUM> of <FIG> is merely an example, and a model other than the object recognition model <NUM> that is generated by machine learning may also be used.

<NUM> Further, the present technology is also applicable to the case of performing processing of recognizing a target object by a camera (an image sensor) and LiDAR being used in combination.

<NUM> Furthermore, the present technology is also appliable to the case of using a sensor that detects an object and is other than a millimeter-wave radar and LiDAR.

<NUM> Further, for example, coordinate systems of all of the images may be transformed such that the coordinate systems of the respective images are matched to a new coordinate system that is different from the coordinate systems of the respective images.

The series of processes described above can be performed using hardware or software. When the series of processes is performed using software, a program included in the software is installed on a computer. Here, examples of the computer include a computer incorporated into dedicated hardware, and a computer such as a general-purpose personal computer that is capable of performing various functions by various programs being installed thereon.

<FIG> is a block diagram of an example of a configuration of hardware of a computer that performs the series of processes described above using a program.

In the computer <NUM>, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, and a random access memory (RAM) <NUM> are connected to one another through a bus <NUM>.

Further, an input/output interface <NUM> is connected to the bus <NUM>. An input section <NUM>, an output section <NUM>, a recording section <NUM>, a communication section <NUM>, and a drive <NUM> are connected to the input/output interface <NUM>.

The input section <NUM> includes, for example, an input switch, a button, a microphone, and an imaging element. The output section <NUM> includes, for example, a display and a speaker. The recording section <NUM> includes, for example, a hard disk and a nonvolatile memory. The communication section <NUM> includes, for example, a network interface. The drive <NUM> drives a removable medium <NUM> such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer <NUM> having the configuration described above, the series of processes described above is performed by the CPU <NUM> loading, for example, a program recorded in the recording section <NUM> into the RAM <NUM> and executing the program via the input/output interface <NUM> and the bus <NUM>.

For example, the program executed by the computer <NUM> (the CPU <NUM>) can be provided by being recorded in the removable medium <NUM> serving as, for example, a package medium. Further, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer <NUM>, the program can be installed on the recording section <NUM> via the input/output interface <NUM> by the removable medium <NUM> being mounted on the drive <NUM>. Further, the program can be received by the communication section <NUM> via the wired or wireless transmission medium to be installed on the recording section <NUM>. Moreover, the program can be installed in advance on the ROM <NUM> or the recording section <NUM>.

Note that the program executed by the computer may be a program in which processes are chronologically performed in the order of the description herein, or may be a program in which processes are performed in parallel or a process is performed at a necessary timing such as a timing of calling.

Further, the system as used herein refers to a collection of a plurality of components (such as apparatuses and modules (parts)) and it does not matter whether all of the components are in a single housing. Thus, a plurality of apparatuses accommodated in separate housings and connected to one another via a network, and a single apparatus in which a plurality of modules is accommodated in a single housing are both systems.

Furthermore, the embodiment of the present technology is not limited to the examples described above, and various modifications may be made thereto without departing from the scope of the claims.

For example, the present technology may also have a configuration of cloud computing in which a single function is shared to be cooperatively processed by a plurality of apparatuses via a network.

Further, the respective steps described using the flowcharts described above may be shared to be performed by a plurality of apparatuses, in addition to being performed by a single apparatus.

Moreover, when a single step includes a plurality of processes, the plurality of processes included in the single step may be shared to be performed by a plurality of apparatuses, in addition to being performed by a single apparatus.

Note that the effects described herein are not limitative but are merely illustrative, and other effects may be provided.

Claim 1:
An information processing apparatus to improve the accuracy in recognizing a target object, comprising:
a data acquisition section (<NUM>) comprising a camera (<NUM>) with an image sensor (201A) having a sensing range and adapted to obtain a captured image (Pa) with a coordinate system , and a millimeter-wave radar sensor (<NUM>) having a sensing range at least partially overlapping the sensing range of the image sensor and adapted to obtain an image indicating the sensing result of the sensor;
an image processor (<NUM>) receiving the captured image and adapted to provide a captured image (Pa) with reduced resolution;
a signal processor (<NUM>) receiving the image from the millimeter-wave radar sensor (<NUM>) and adapted to provide a radar sensor image comprising a signal-intensity image (Pb) indicating a location of each object and the intensity of a signal reflected off the object, and a speed image (Pc) indicating a location of each object and a relative speed of the object relative to the sensor;
a geometric transformation section (<NUM>) adapted to receive the radar sensor image and to transform the radar sensor image into a geometrically transformed radar sensor image of which a coordinate system is matched to the coordinate system of the captured image ; and
an object recognition section (<NUM>) adapted to perform processing of recognizing a target object on a basis of the captured image with reduced resolution and geometrically transformed radar sensor image of which the coordinate systems have been matched to each other.