The present technology relates to an information processing apparatus, an information processing method, and a program capable of obtaining a distance to an object more accurately. An extraction unit extracts, on the basis of an object recognised in an imaged image obtained by a camera, sensor data corresponding to an object region including an object in the imaged image among sensor data obtained by a rangefinding sensor. The present technology can be applied to an evaluation apparatus for distance information, for example.

TECHNICAL FIELD

The present technology relates to an information processing apparatus, an information processing method, and a program, and more particularly, relates to an information processing apparatus, an information processing method, and a program capable of more accurately obtaining a distance to an object.

BACKGROUND ART

Patent Document 1 discloses a technology for generating rangefinding information for an object on the basis of a rangefinding point in a rangefinding point arrangement region set in an object region in distance measurement using a stereo image.

CITATION LIST

Patent Document

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

However, there is a possibility that the accurate distance to an object cannot be obtained depending on the state of the object recognized in the image only by using the rangefinding point set in the object region.

The present technology has been made in view of such a situation, and makes it possible to more accurately obtain the distance to an object.

Solutions to Problems

An information processing apparatus of the present technology is an information processing apparatus including an extraction unit that extracts, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image among the sensor data obtained by a rangefinding sensor.

An information processing method of the present technology is an information processing method in which an information processing apparatus extracts, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image among the sensor data obtained by a rangefinding sensor.

A program of the present technology is a program for causing a computer to execute processing of extracting, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image among the sensor data obtained by a rangefinding sensor.

In the present technology, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image is extracted among the sensor data obtained by a rangefinding sensor.

MODE FOR CARRYING OUT THE INVENTION

Modes for carrying out the present technology (hereinafter, embodiments) will be described below. Note that the description will be given in the following order.

1. Configuration example of vehicle control system

2. Evaluation of distance information of recognition system

3. Configuration and operation of evaluation apparatus

4. Modification of point cloud data extraction

5. Configuration and operation of information processing apparatus

6. Configuration example of computer

1. Configuration Example of Vehicle Control System

FIG.1is a block diagram illustrating a configuration example of a vehicle control system11, which is an example of a mobile apparatus control system to which the present technology is applied.

The vehicle control system11is provided in a vehicle1and performs processing related to travel assistance and automated driving of the vehicle1.

The vehicle control system11includes a processor21, a communication unit22, a map information accumulation unit23, a global navigation satellite system (GLASS) reception unit24, an external recognition sensor25, an in-vehicle sensor26, a vehicle sensor27, a recording unit28, a travel assistance/automated driving control unit29, a driver monitoring system (DMS)30, a human machine interface (HMI)31, and a vehicle control unit32.

The processor21, the communication unit22, the map information accumulation unit23, the GNSS reception unit24, the external recognition sensor25, the in-vehicle sensor26, the vehicle sensor27, the recording unit28, the travel assistance/automated driving control. unit29, the driver monitoring system (DMS)30, the human machine interface (HMI)31, and the vehicle control unit32are connected to one another via a communication network41. The communication network41includes, for example, a vehicle-mounted communication network conforming to a discretionary standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark), a bus, and the like. Note that there is a case where each unit of the vehicle control system11is directly connected by, for example, near field communication (NFC), Bluetooth (registered trademark), and the like without via the communication network41.

Note that hereinafter, in a case where each unit of the vehicle control system11performs communication via the communication network41, description of the communication network41will be omitted. For example, in a case where the processor21and the communication unit22perform communication via the communication network41, it is simply described that the processor21and the communication unit22perform communication.

The processor21includes various processors such as a central processing unit (CPU), a micro processing unit (MPU), and an electronic control unit (ECU). The processor21controls the entire vehicle control system11.

The communication unit22communicates with various equipment inside and outside the vehicle, other vehicles, servers, base stations, and the like, and transmits and receives various data. As the communication with the outside of the vehicle, for example, the communication unit22receives, from the outside, a program for updating software for controlling the operation of the vehicle control system11, map information, traffic information, information around the vehicle1, and the like. For example, the communication unit22transmits, to the outside, information regarding the vehicle1(for example, data indicating the state of the vehicle1, a recognition result by a recognition unit73, and the like), information around the vehicle1, and the like. For example, the communication unit22performs communication corresponding to a vehicle emergency call system such as an eCall.

Note that the communication method of the communication unit22is not particularly limited. Furthermore, a plurality of communication methods may be used.

As communication with the inside of the vehicle, for example, the communication unit22performs wireless communication with in-vehicle equipment by a communication method such as wireless LAN, Bluetooth, NFC, or wireless USB (WUSB). For example, the communication unit22performs wired communication with in-vehicle equipment by a communication method such as a universal serial bus (USB), a high-definition multimedia interface (HDMI, registered trademark), or a mobile high-definition link (MHL) via a connection terminal (and, if necessary, a cable) not illustrated.

Here, the in-vehicle equipment is, for example, equipment that is not connected to the communication network41in the vehicle. For example, mobile equipment or wearable equipment carried by a passenger such as a driver, information equipment brought into the vehicle and temporarily installed, the like are assumed.

For example, the communication unit22communicates with a server and the like existing on an external network (for example, the Internet, a cloud network, or a company-specific network) via a base station or an access point by a wireless communication method such as the fourth generation mobile communication system (4G), the fifth generation mobile communication system (5G), long term evolution (LTE), or dedicated short range communications (DSRC).

For example, the communication unit22communicates with a terminal (for example, a terminal of a pedestrian or a store, or a machine type communication (MTC) terminal) existing in the vicinity of The subject vehicle using a peer to peer (P2P) technology. For example, the communication unit22performs V2X communication. The V2X communication is, for example, vehicle to vehicle communication with another vehicle, vehicle to infrastructure communication with a roadside device and the like, vehicle to home communication, vehicle to pedestrian communication with a terminal and the like carried by a pedestrian, and the like.

For example, the communication unit22receives an electromagnetic wave transmitted by a vehicle information and communication system (VICS, registered trademark) such as a radio wave beacon, an optical beacon, or FM multiplex broadcasting.

The map information accumulation unit23accumulates a map acquired from the outside and a map created by the vehicle1. For example, the map information accumulation unit23accumulates a three-dimensional highly accurate map, a global map having lower accuracy than the highly accurate map and covering a wide area, and the like.

The highly accurate map is, for example, a dynamic map, a point cloud map, a vector map (also referred to as advanced driver assistance system (ADAS) map), and the like. The dynamic map is, for example, a map including four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided from an external server and the like. The point cloud map is a map including point clouds (point cloud data). The vector map is a map in which information such as a lane and a position of a traffic signal associated with the point cloud map. The point cloud map and the vector map may be provided from, for example, an external server and the like, or may be created by the vehicle1as a map for performing matching with a local map described later on the basis of a sensing result by a radar52, a LiDAR53, and the like, and may be accumulated in the map information accumulation unit23. Furthermore, in a case where a highly accurate map is provided from an external server and the like, in order to reduce the communication capacity, map data of, for example, several hundred meters square regarding a planned path on which the vehicle1travels from now on is acquired from the server and the like.

The GNSS reception unit24receives a GNSS signal from a GNSS satellite, and supplies the GNSS signal to the travel assistance/automated driving control unit29.

The external recognition sensor25includes various sensors used for recognition of a situation outside the vehicle1, and supplies sensor data from each sensor to each unit of the vehicle control system11. The type and number of sensors included in the external recognition sensor25are discretionary.

For example, the external recognition sensor25includes a camera51, the radar52, the light detection and ranging, laser imaging detection and ranging (LiDAR)53, and an ultrasonic sensor54. The number of the camera51, the radar52, the LiDAR53, and the ultrasonic sensor54is discretionary, and an example of a sensing region of each sensor will be described later.

Note that as the camera51, for example, a camera of a discretionary imaging method such as a time of flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera is used as necessary.

Furthermore, for example, the external recognition sensor25includes an environment sensor for detecting weather, meteorological phenomenon, brightness, and the like. The environment sensor includes, for example, a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, an illuminance sensor, and the like.

Moreover, for example, the external recognition sensor25includes a microphone used for detection of sound around the vehicle1, a position of a sound source, and the like.

The in-vehicle sensor26includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system11. The type and number of sensors included in the in-vehicle sensor26are discretionary.

For example, the in-vehicle sensor26includes a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, a biological sensor, and the like. As the camera, for example, a camera of any imaging method such as a ToF camera, a stereo camera, a monocular camera, an infrared camera, and the like can be used. The biological sensor is provided, for example, in a seat, a steering wheel, and the like, and detects various kinds of biological information of a passenger such as a driver.

The vehicle sensor27includes various sensors for detecting the state of the vehicle1, and supplies sensor data from each sensor to each unit of the vehicle control system11. The type and number of sensors included in the vehicle sensor27are discretionary.

For example, the vehicle sensor27includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU). For example, the vehicle sensor27includes a steering angle sensor that detects a steering angle of a steering wheel, a yaw rate sensor, an accelerator sensor that detects an operation amount of an accelerator pedal, and a brake sensor that detects an operation amount of a brake pedal. For example, the vehicle sensor27includes a rotation sensor that detects the rotation speed of the engine or the motor, an air pressure sensor that detects the air pressure of the tire, a slip rate sensor that detects the slip rate of the tire, and a wheel speed sensor that detects the rotation speed of the wheel. For example, the vehicle sensor27includes a battery sensor that detects a remaining amount and temperature of the battery, and an impact sensor that detects an external impact.

The recording unit28includes, for example, a read only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, and the like. The recording unit28records various programs, data, and the like used by each unit of the vehicle control system11. For example, the recording unit28records a rosbag file including a message transmitted and received by a robot operating system (ROS) in which an application program related to automated driving operates. For example, the recording unit28includes an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and records information of the vehicle1before and after an event such as an accident.

The travel assistance/automated driving control unit29controls travel assistance and automated driving of The vehicle1. For example, the travel assistance/automated driving control unit29includes an analysis unit61, a behavior planning unit62, and an operation control unit63.

The analysis unit61performs analysis processing of the situation of the vehicle1and the surroundings. The analysis unit61includes a self-position estimation unit71, a sensor fusion unit72, and a recognition unit73.

The self-position estimation unit71estimates the self-position of the vehicle1on the basis of the sensor data from the external recognition sensor25and the highly accurate map accumulated in the map information accumulation unit23, For example, the self-position estimation unit71generates a local map on the basis of sensor data from the external recognition sensor25, and estimates the self-position of the vehicle1by matching the local map with the highly accurate map. The position of the vehicle1is based on, for example, the center of a rear wheel pair axle.

The local map is, for example, a three-dimensional highly accurate map created using a technology such as simultaneous localization and mapping (SLAM), an occupancy grid map, and the like. The three-dimensional highly accurate map is, for example, the above-described point cloud map and the like. The occupancy grid map is a map in which a three-dimensional or two-dimensional space around the vehicle1is divided into grids of a predetermined size to indicate an occupancy state of an object in units of grids. The occupancy state of an object is indicated by, for example, the presence or absence or existence probability of the object. The local map is also used for detection processing and recognition processing of a situation outside the vehicle1by the recognition unit73, for example.

Note that the self-position estimation unit71may estimate the self-position of the vehicle1on the basis of a GLASS signal and sensor data from the vehicle sensor27.

The sensor fusion unit72performs sensor fusion processing of obtaining new information by combining a plurality of different types of sensor data (for example, image data supplied from the camera51and sensor data supplied from the radar52). Methods for combining different types of sensor data include integration, fusion, association, and the like.

The recognition unit73performs detection processing and recognition processing of the situation outside the vehicle1.

For example, the recognition unit73performs detection processing and recognition processing of the situation outside the vehicle1on the basis of information from the external recognition sensor25, information from the self-position estimation unit71, information from the sensor fusion unit72, and the like.

Specifically, for example, the recognition unit73performs detection processing, recognition processing, and the like of an object around the vehicle1. The detection processing of an object is, for example, processing of detecting the presence or absence, size, shape, position, motion, and the like of the object. The recognition processing of an object is, for example, processing of recognizing an attribute such as a type of the object or identifying a specific object. However, the detection processing and the recognition processing are not necessarily clearly divided, and may overlap.

For example, the recognition unit73detects an object around the vehicle1by performing clustering for classifying point clouds based on sensor data such as LiDAR or radar for each cluster of point clouds. Therefore, the presence or absence, size, shape, and position of the object around the vehicle1are detected.

For example, the recognition unit73detects the motion of the object around the vehicle1by performing tracking that follows the motion of the cluster of the point cloud classified by clustering. Therefore, the speed and the traveling direction (movement vector) of the object around the vehicle1are detected.

For example, the recognition unit73recognizes the type of the object around the vehicle1by performing object recognition processing such as semantic segmentation on the image data supplied from the camera51.

Note that as the object to be detected or recognized, for example, a vehicle, a human, a bicycle, an obstacle, a structure, a road, a traffic light, a traffic sign, a road sign, and the like are assumed.

For example, the recognition unit73performs recognition processing of traffic rules around the vehicle1on the basis of the map accumulated in the map information accumulation unit23, the estimation result of the self-position, and the recognition result of the object around the vehicle1. By this processing, for example, the position and the state of a traffic signal, the content of a traffic sign and a road sign, the content of traffic regulation, a travelable lane, and the like are recognized.

For example, the recognition unit73performs recognition processing of the environment around the vehicle1. As the surrounding environment to be recognized, for example, weather, temperature, humidity, brightness, a state of a road surface, and the like are assumed.

The behavior planning unit62creates a behavior plan of the vehicle1. For example, the behavior planning unit62creates a behavior plan by performing processing of path planning and path following.

Note that the global path planning is processing of planning a rough path from the start to the goal. This path planning includes processing of local path planning that is called a trajectory planning and that enables safe and smooth traveling in the vicinity of the vehicle1in consideration of the motion characteristics of the vehicle1in the path planned by the path plan.

The path following is processing of planning an operation for safely and accurately traveling a path planned by a path planning within a planned time. For example, the target speed and the target angular velocity of the vehicle1are calculated.

The operation control unit63controls the operation of the vehicle1in order to achieve the behavior plan created by the behavior planning unit62.

For example, the operation control unit63controls a steering control unit81, a brake control unit82, and a drive control unit83to perform acceleration/deceleration control and direction control such that the vehicle1travels on the trajectory calculated by the trajectory plan. For example, the operation control unit63performs cooperative control for the purpose of implementing the functions of the ADAS such as collision avoidance or impact mitigation, follow-up traveling, vehicle speed maintaining traveling, collision warning of the subject vehicle, lane departure warning of the subject vehicle, and the like. For example, the operation control unit63performs cooperative control for the purpose of automated driving and the like in which the vehicle autonomously travels without depending on the operation of the driver.

The DMS30performs authentication processing of a driver, recognition processing of a driver state, and the like on the basis of sensor data from the in-vehicle sensor26, input data input to the HMI31, and the like. As the state of the driver to be recognized, for example, a physical condition, an arousal level, a concentration level, a fatigue level, a line-of-sight direction, a drunkenness level, a driving operation, a posture, and the like are assumed.

Note that the DMS30may perform authentication processing of a passenger other than the driver and recognition processing of the state of the passenger. Furthermore, for example, the DFS30may perform recognition processing of the situation inside the vehicle on the basis of sensor data from the in-vehicle sensor26. As the situation inside the vehicle to be recognized, for example, temperature, humidity, brightness, odor, and the like are assumed.

The HMI31is used for inputting various data, instructions, and the like, generates an input signal on the basis of the input data, instructions, and the like, and supplies the input signal to each unit of the vehicle control system11. For example, the HMI31includes an operation device such as a touchscreen, a button, a microphone, a switch, and a lever, an operation device that enables inputting by a method other than manual operation by voice, gesture, and the like. Note that the HMI31may be, for example, a remote control apparatus using infrared rays or other radio waves, or external connection equipment such as mobile equipment or wearable equipment compatible with the operation of the vehicle control system11.

Furthermore, the HMI31performs output control of controlling generation and output of visual information, auditory information, and tactile information to the passenger or the outside of the vehicle, as well as output content, output timing, an output method, and the like. The visual information is, for example, information indicated by an image or light such as an operation screen, a state display of the vehicle1, a warning display, or a monitor image indicating the situation around the vehicle1. The auditory information is, for example, information indicated by voice such as guidance, a warning sound, a warning message, and the like. The tactile information is, for example, information given to the tactile sense of the passenger by force, vibration, motion, and the like.

As a device that outputs visual information, for example, a display device, a projector, a navigation apparatus, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, and the like are assumed. The display device may be an apparatus that displays visual information in the field of view of the passenger, such as a head-up display, a transmissive display, a wearable device having an augmented reality (AR) function, and the like, in addition co an apparatus having a normal display.

As a device that outputs auditory information, for example, an audio speaker, a headphone, an earphone, and the like are assumed.

As a device that outputs tactile information, for example, a haptics element using haptics technology and the like are assumed. The haptics element is provided on, for example, the steering wheel, the seat, and the like.

The vehicle control unit32controls each unit of the vehicle1. The vehicle control unit32includes the steering control unit81, the brake control unit82, the drive control unit83, a body system control unit84, a light control unit85, and a horn control unit86.

The steering control unit81performs detection, control, and the like of the state of a steering system of the vehicle1. The steering system includes, for example, a steering mechanism including a steering wheel and the like, an electric power steering, and the like. The steering control unit81includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, and the like.

The brake control unit82detects and controls the state of a brake system of the vehicle1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), and the like. The brake control unit82includes, for example, a control unit such as an ECU that controls the brake system, an actuator that drives the brake system, and the like.

The drive control unit83detects and controls the state of a drive system of the vehicle1. The drive system includes, for example, an accelerator pedal, a driving force generation apparatus for generating a driving force such as an internal combustion engine, a driving motor, and the like, a driving force transmission mechanism for transmitting the driving force to the wheels, and the like. The drive control unit83includes, for example, a control unit such as an ECU that controls the drive system, an actuator that drives the drive system, and the like.

The body system control unit84detects and controls the state of a body system of the vehicle1. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning apparatus, an airbag, a seat belt, a shift lever, and the like. The body system control unit84includes, for example, a control unit such as an ECU that controls the body system, an actuator that drives the body system, and the like.

The light control unit85detects and controls states of various lights of the vehicle1. As the lights to be controlled, for example, a headlight, a backlight, a fog light, a turn signal, a brake light, a projection, a display of a bumper, and the like are assumed. The light control unit85includes a control unit such as an ECU that controls light, an actuator that drives light, and the like.

The horn control unit86detects and controls the state of a car horn of the vehicle1. The horn control unit86includes, for example, a control unit such as an ECU that controls the car horn, an actuator that drives the car horn, and the like.

FIG.2is a view illustrating an example of a sensing region by the camera51, the radar52, the LiDAR53, and the ultrasonic sensor54of the external recognition sensor25inFIG.1.

A sensing region101F and a sensing region101B illustrate examples of the sensing region of the ultrasonic sensor54. The sensing region101F covers the periphery of the front end of the vehicle1. The sensing region101B covers The periphery of the rear end of the vehicle1.

The sensing results in the sensing region101F and the sensing region101B are used, for example, for parking assistance of the vehicle1.

Sensing regions102F to102B illustrate examples of sensing regions of the radar52for a short distance or a middle distance. The sensing region102F covers a position farther than the sensing region101F in front of the vehicle1. The sensing region102B covers a position farther than the sensing region101B behind the vehicle1. The sensing region102L covers the periphery behind the left side surface of the vehicle1. The sensing region102R covers the periphery behind the right side surface of the vehicle1.

The sensing result in the sensing region102F is used, for example, for detection of a vehicle, a pedestrian, and the like existing in front of the vehicle1. The sensing result in the sensing region102B is used, for example, for a collision prevention function and the like behind the vehicle1. The sensing results in the sensing region102L and the sensing region102R are used, for example, for detection of an object in a blind spot on the side of the vehicle1.

Sensing regions103F to103B illustrate examples of sensing regions by the camera51. The sensing region.103F covers a position farther than the sensing region102F in front of the vehicle1. The sensing region103B covers a position farther than the sensing region102B behind the vehicle1. The sensing region103L covers the periphery of the left side surface of the vehicle1. The sensing region103R covers the periphery of the right side surface of the vehicle1.

The sensing result in the sensing region103F is used, for example, for recognition of a traffic light and a traffic sign, a lane departure prevention assist system, and the like. The sensing result in the sensing region103B is used, for example, for parking assistance, a surround view system, and the like. The sensing results in the sensing region103L and the sensing region103R are used, for example, in a surround view system and the like.

A sensing region104illustrates an example of a sensing region of the LiDAR53. The sensing region104covers a position farther than the sensing region103F in front of the vehicle1. On the other hand, the sensing region104has a narrower range in the left-right direction than that of the sensing region103F.

The sensing result in the sensing region104is used, for example, for emergency, braking, collision avoidance, pedestrian detection, and the like.

A sensing region105illustrates an example of the sensing region of a radar52for a long range. The sensing region105covers a position farther than the sensing region104in front of the vehicle1. On the other hand, the sensing region105has a narrower range in the left-right direction than that of the sensing region104.

The sensing result in the sensing region105is used, for example, for adaptive cruise control (ACC).

Note that the sensing region of each sensor may have various configurations other than those inFIG.2.

Specifically, the ultrasonic sensor54may also sense the side of the vehicle1, or the LiDAR53may sense behind the vehicle1.

2. Evaluation of Distance Information of Recognition System

For example, as illustrated inFIG.3, as a method of evaluating distance information output by a recognition system210that recognizes an object around the vehicle1by performing the sensor fusion processing described above, it is conceivable to compare and evaluate the point cloud data of a LiDAR220as a correct value. However, in a case where a user U visually compares the distance information of the recognition system210with the LiDAR point cloud data frame by frame, it takes a huge amount of time.

Therefore, in the following, a configuration in which the distance information of the recognition system and the LiDAR point cloud data are automatically compared will be described.

3. Configuration and Operation of Evaluation Apparatus

(Configuration of Evaluation Apparatus)

FIG.4is a block diagram illustrating the configuration of an evaluation apparatus that evaluates distance information of the recognition system as described above.

FIG.4illustrates a recognition system320and an evaluation apparatus340.

The recognition system320recognizes an object around the vehicle1on the basis of an imaged image obtained by the camera311and millimeter wave data obtained by the millimeter wave radar312. The camera311and the millimeter wave radar312correspond to the camera51and the radar52inFIG.1, respectively.

The recognition system320includes a sensor fusion unit321and a recognition unit322.

The sensor fusion unit321corresponds to the sensor fusion unit72inFIG.1, and performs sensor fusion processing using the imaged image from the camera311and the millimeter wave data from the millimeter wave radar312.

The recognition unit322corresponds to the recognition unit73inFIG.1, and performs recognition processing (detection processing) of an object around the vehicle1on the basis of a processing result of the sensor fusion processing by the sensor fusion unit321.

The recognition result of the object around the vehicle1is output by the sensor fusion processing by the sensor fusion unit321and the recognition processing by the recognition unit322.

The recognition result of the object obtained while the vehicle1is traveling is recorded as a data log and input to the evaluation apparatus340. Note that the recognition result of the object includes distance information indicating the distance to the object around the vehicle1, object information indicating the type and attribute of the object, speed information indicating the speed of the object, and the like.

Similarly, while the vehicle1is traveling, point cloud data is obtained by a LiDAR331serving as a rangefinding sensor in the present embodiment, and moreover, various vehicle information regarding the vehicle1is obtained via a CAN332. The LiDAR331and the CAN332correspond to the LiDAR53and the communication network41inFIG.1, respectively. The point cloud data and vehicle information obtained while the vehicle1is traveling are also recorded as a data log and input to the evaluation apparatus340.

The evaluation apparatus340includes a conversion unit341, an extraction unit342, and a comparison unit343.

The conversion unit341converts the point cloud data that is the data in an xyz three-dimensional coordinate system obtained by the LiDAR331into a camera coordinate system of the camera311, and supplies the converted point cloud data to the extraction unit342.

By using the recognition result from the recognition system320and the point cloud data from the conversion unit341, the extraction unit342extracts, among point cloud data, the point cloud data corresponding to an object region including the object in the imaged image on the basis of the object recognized in the imaged image. In other words, the extraction unit342performs clustering on the point cloud data corresponding to the recognized object among the point cloud data.

Specifically, the extraction unit342associates the imaged image including a rectangular frame indicating the object region of the recognized object supplied from the recognition system320as the recognition result with the point cloud data from the conversion unit341, and extracts the point cloud data existing in the rectangular frame. At this time, the extraction unit342sets an extraction condition of the point cloud data on the basis of the recognized object, and extracts the point cloud data existing in the rectangular frame on the basis of the extraction condition. The extracted point cloud data is supplied to the comparison unit343as point cloud data corresponding to the object that is the evaluation target for the distance information.

With the point cloud data from the extraction unit342as a correct value, the comparison unit343compares the point cloud data with the distance information included in the recognition result from the recognition system320. Specifically, it is determined whether or not a difference between the distance information from the recognition system320and a correct value (point cloud data) falls within a predetermined reference value. The comparison result is output as an evaluation result of the distance information from the recognition system320. Note that the accuracy of the correct value can be further enhanced by using the mode of the point cloud data existing in the rectangular frame as the point cloud data used as the correct value.

Conventionally, for example, as illustrated in the upper part ofFIG.5, it has been visually confirmed as to which point cloud data371of point cloud data371obtained by LIDAR corresponds to a rectangular frame361F indicating the vehicle recognized in an imaged image360.

On the other hand, according to the evaluation apparatus340, as illustrated in the lower part ofFIG.5, the point cloud data371corresponding to the rectangular frame361F indicating the vehicle recognized in the imaged image360is extracted from the point cloud data371obtained by the LiDAR. Therefore, it is possible to narrow down the point cloud data corresponding to the evaluation target, and it becomes possible to perform comparison between the distance information of the recognition system and the LiDAR point cloud data accurately with a low load.

Example of Extraction of Point Cloud Data

As described above, the extraction unit342can set the extraction condition (clustering condition) of the point cloud data on the basis of the recognized object, for example, according to the state of the recognized object.

As illustrated in the upper left side ofFIG.6, in a case where another vehicle412exists closer to the subject vehicle than to a vehicle411that is an evaluation target in an imaged image410, a rectangular frame411F for the vehicle411overlaps with a rectangular frame412F for the other vehicle412. In a case where point cloud data existing in the rectangular frame411F is extracted in this state, point cloud data that does not correspond to the evaluation target is extracted as illustrated in a bird's-eye view on the upper right side ofFIG.6. In the bird's-eye view as in the upper right side ofFIG.6, the point cloud data on the three-dimensional coordinates obtained by the LiDAR331is illustrated together with the corresponding object.

Therefore, as illustrated in the lower left side ofFIG.6, by masking the region corresponding to the rectangular frame412F for the other vehicle412, the extraction unit342excludes the point cloud data corresponding to the region overlapping the rectangular frame412F in the rectangular frame411F from the extraction target. Therefore, as illustrated in the bird's-eye view on the right side of the lower part ofFIG.6, only the point cloud data corresponding to the evaluation target can be extracted.

Note that the rectangular frame is defined by, for example, the width and height of a rectangular frame with the coordinates of the upper left vertex of the rectangular frame as a reference point, and whether or not the rectangular frames overlap each other is determined on the basis of the reference point, the width, and the height of each rectangular frame.

As illustrated in the upper left side ofFIG.7, in a case where an obstacle422such as a utility pole exists behind the vehicle421that is the evaluation target in an imaged image420a,when point cloud data existing in a rectangular frame421F of the vehicle421is extracted, point cloud data that does not correspond to the evaluation target is extracted as a bird's-eye view in the upper right side ofFIG.7.

Similarly, as illustrated in the lower left side ofFIG.7, in a case where an obstacle423such as a utility pole exists closer to the subject vehicle than to the vehicle421that is the evaluation target in an imaged image420b,when point cloud data existing in a rectangular frame421F of the vehicle421is extracted, point cloud data that does not correspond to the evaluation target is extracted as a bird's-eye view in the lower right side ofFIG.7.

On the other hand, as illustrated on the left side ofFIG.8, the extraction unit342extracts the point cloud data in which the distance to the evaluation target is within a predetermined range by excluding, from the extraction target, the point cloud data in which the distance to the object that is the evaluation target (recognized object) is larger than a predetermined distance threshold. Note that the distance to the evaluation target is acquired from distance information included in the recognition result output by the recognition system320.

At this time, the extraction unit342sets the distance threshold according to the object that is the evaluation target (the type of the object). The distance threshold is set to a larger value as the moving speed of the object that is the evaluation target is higher, for example. Note that the type of the object that is the evaluation target is also acquired from the object information included in the recognition result output by the recognition system320.

For example, in a case where the evaluation target is a vehicle, by setting the distance threshold to 1.5 m, point cloud data in which the distance to the vehicle is larger than 1.5 m is excluded from the extraction target. Furthermore, in a case where the evaluation target is a motorcycle, by setting the distance threshold to 1 m, point cloud data in which the distance to the motorcycle is larger than 1 m is excluded from the extraction target. Moreover, in a case where the evaluation target is a bicycle or a pedestrian, by setting the distance threshold to 50 cm, point cloud data in which the distance to the bicycle or the pedestrian is larger than 50 cm is excluded from the extraction target.

Note that the extraction unit342may change the set distance threshold according to the moving speed (vehicle speed) of the vehicle1on which the camera311and the millimeter wave radar312are mounted. In general, the inter-vehicle distance between vehicles increases during high-speed traveling, and the inter-vehicle distance decreases during low-speed traveling. Therefore, when the vehicle1is traveling at a high speed, the distance threshold is changed to a larger value. For example, in a case where the vehicle1is traveling at 40 km/h or higher, when the evaluation target is a vehicle, the distance threshold is changed from 1.5 m to 3 m. In a case where the vehicle1is traveling at 40 km/h or higher, when the evaluation target is a motorcycle, the distance threshold is changed from 1 in to 2 m. Note that the vehicle speed of the vehicle1is acquired from vehicle information obtained via the CAN332.

Moreover, as illustrated on the right side ofFIG.8, the extraction unit342extracts the point cloud data in which the difference in speed from the evaluation target is within a predetermined range by excluding, from the extraction target, the point cloud data in which the difference between the speed of the object (recognized object) that is the evaluation target and the speed calculated on the basis of time-series change in the point cloud data is larger than a predetermined speed threshold. The speed of point cloud data is calculated by a change in the position of the point cloud data in time series. The speed of the evaluation target is acquired from speed information included in the recognition result output by the recognition system320.

In the example on the right side ofFIG.8, the point cloud data at a speed of 0 km/h existing behind the object that is the evaluation target and the point cloud data at a speed of 0 km/h exists closer to the subject vehicle than to the object that is the evaluation target are excluded from the extraction target, and the point cloud data at a speed of 15 km/h existing in the vicinity of the object that is the evaluation target is extracted.

The extraction unit342can also chance the extraction region of point cloud data according to the distance to the object that is the evaluation target, in other words, the size of the object region in the imaged image.

For example, as illustrated inFIG.9, in an imaged image440, a rectangular frame441F for a vehicle441positioned at a long distance becomes small, and a rectangular frame442F for a vehicle442positioned at a short distance becomes large. In this case, in the rectangular frame441F, the number of point cloud data corresponding to the vehicle441is small. On the other hand, in the rectangular frame442F, although the number of point cloud data corresponding to the vehicle442is large, many point cloud data corresponding to the background and the road surface are included.

Therefore, in a case where the rectangular frame is larger than a predetermined area, the extraction unit342sets only the point cloud data corresponding to the vicinity of the center of the rectangular frame as the extraction target, and in a case where the rectangular frame is smaller than the predetermined area, the extraction unit sets the point cloud data corresponding to the entire rectangular frame as the extraction target.

That is, as illustrated inFIG.10, in the rectangular frame441F having a small area, the point cloud data corresponding to the entire rectangular frame441F is extracted. On the other hand, in the rectangular frame442F having a large area, only the point cloud data corresponding to a region C442F near the center of the rectangular frame442F is extracted. Therefore, point cloud data corresponding to the background and the road surface can be excluded from the extraction target.

Furthermore, also in a case where the evaluation target is a bicycle, a pedestrian, a motorcycle, and the like, the rectangular frame for these includes many point cloud data corresponding to the background and the road surface. Therefore, in a case where the type of the object acquired from the object information included in the recognition result output by the recognition system320is a bicycle, a pedestrian, a motorcycle, and the like, only the point cloud data corresponding to the vicinity of the center of the rectangular frame may be set as the extraction target.

As described above, by setting the extraction condition (clustering condition) of the point cloud data on the basis of the object that is the evaluation target, it is possible to more reliably extract the point cloud data corresponding to the object that is the evaluation target.

(Evaluation Processing of Distance Information)

Here, evaluation processing of distance information by the evaluation apparatus340will be described with reference to the flowchart ofFIG.11.

In step S1, the extraction unit342acquires the recognition result of the object recognized in an imaged image film the recognition system320.

In step S2, the conversion unit341performs coordinate conversion on the point cloud data obtained by the LiDAR331.

In step S3, the extraction unit342sets, on the basis of the object, an extraction condition of the point cloud data corresponding to the object region of the object recognized in the imaged image by the recognition system320among the point cloud data converted into the camera coordinate system.

In step S4, the extraction unit342extracts the point cloud data corresponding to the object region for the recognized object on the basis of the set extraction condition.

In step S6, with the point cloud data extracted by the extraction unit342as a correct value, the comparison unit343compares the point cloud data with the distance information included in the recognition result from the recognition system320. The comparison result is output as an evaluation result of the distance information from the recognition system320.

According to the above processing, in the evaluation of the distance information from the recognition system320, it is possible to narrow down the point cloud data corresponding to the evaluation target, and it becomes possible to perform comparison between the distance information of the recognition system and the LiDAR point cloud data accurately with a low load.

(Extraction Condition Setting Processing of Point Cloud Data)

Next, extraction condition setting processing of point cloud data executed in step S3of the evaluation processing of distance information described above will be described with reference toFIGS.12and13. This processing is started in a state where the point cloud data corresponding to the object region of the recognized object (object that is the evaluation target) in the point cloud data is specified.

In step S11, the extraction unit342determines whether or not the object region of the recognized object (object that is the evaluation target) overlaps another object region for another object.

In a case where it is determined that the object region overlaps with another object region, the process proceeds to step S12, and the extraction unit342excludes, from the extraction target, the point cloud data corresponding to the region overlapping with another object region as described with reference toFIG.6. Thereafter, the process proceeds to step S13.

On the other hand, in a case where it is determined than the object region does not overlap with another object region, step S12is skipped, and the process proceeds to step S13.

In step S13, the extraction unit342determines whether or not the object region is larger than a predetermined area.

In a case where it is determined that the object region is larger than the predetermined area, the process proceeds to step S14, and the extraction unit342sets the point cloud data near the center of the object region as the extraction target as described with reference toFIGS.9and10. Thereafter, the process proceeds to step S15.

On the other hand, in a case where it is determined that the object region is not larger than the predetermined area, that is, when the object region is smaller than the predetermined area, step S14is skipped, and the process proceeds to step315.

In step S15, the extraction unit342determines whether or not a speed difference from the recognized object is larger than a speed threshold for each of the point cloud data corresponding to the object region.

In a case where it is determined that the speed difference from the recognized object is larger than the speed threshold, the process proceeds to step S16, and the extraction unit342excludes the corresponding point cloud data from the extraction target as described with reference toFIG.8. Thereafter, the process proceeds to step S17inFIG.13.

On the other hand, in a case where it is determined than the speed difference from the recognized object is larger than the speed threshold, that is, an a case where the speed difference from the recognized object is smaller than the speed threshold, step S16is skipped, and the process proceeds to step S17.

In step S17, the extraction unit342sets the distance threshold according to the recognized object (the type of the object) acquired from the object information included in the recognition result.

Next, in step S18, the extraction unit342changes the set distance threshold according to the vehicle speed of the vehicle1acquired from the vehicle information.

Then, in step S19, the extraction unit342determines whether or not the distance to the recognized object is larger than the distance threshold for each of the point cloud data corresponding to the object region.

In a case where is determined that the distance to the recognized object is larger than the distance threshold, the process proceeds to step S20, and the extraction unit342excludes the corresponding point cloud data from the extraction target as described with reference toFIG.8. The extraction condition setting processing of the point cloud data ends.

On the other hand, in a case where it is determined that the distance to the recognized object is larger than the distance threshold, that is, in a case where the distance to the recognized object is smaller than the distance threshold, step S20is skipped, and the extraction condition setting processing of the point cloud data ends.

According to the above processing, since the extraction condition (clustering condition) of the point cloud data is set according to the state of the object that is the evaluation target, it is possible to more reliably extract the point cloud data corresponding to the object that is the evaluation target. As a result, it is possible to evaluate distance information more accurately, and eventually it becomes possible to obtain the distance to the object more accurately.

4. Modification of Point Cloud Data Extraction

Hereinafter, a modification of point cloud data extraction will be described.

Normally, in a case where the vehicle travels forward at a certain speed, the appearance of an object moving at a speed different from that of the vehicle among objects around the vehicle changes. In this case, the point cloud data or corresponding to the object also changes according to the change in the appearance of the object around the vehicle.

For example, as illustrated inFIG.14, it is assumed that a vehicle511traveling in a lane adjacent to a lane in which the subject vehicle travels is recognized in imaged images510aand510bimaged while the subject vehicle is traveling on a road having two lanes on each side. In the imaged image510a,the vehicle511travels in the vicinity of the subject vehicle in the adjacent lane, and in the imaged image510b,the vehicle511travels in the adjacent lane at a position away, forward from the subject vehicle.

In a case where the vehicle511is traveling in the vicinity of the subject vehicle as in the imaged image510a,as the point cloud data corresponding to a rectangular region511Fa for the vehicle511, not only the point cloud data of the rear surface of the vehicle511but also many point cloud data of the side surface of the vehicle511are extracted.

On the other hand, in a case where the vehicle511is traveling away from the subject vehicle as in the imaged image510b, only the point cloud data of the rear surface of the vehicle511is extracted as the point cloud data corresponding to a rectangular region511Fb for the vehicle511.

In a case where the point cloud data of the side surface of the vehicle511is included in the extracted point cloud data as in the imaged image510a,there is a possibility that an accurate distance to the vehicle511cannot be obtained.

Therefore, in a case where the vehicle511is traveling in the vicinity of the subject vehicle, only the point cloud data of the rear surface of the vehicle511is the extraction target, and the point cloud data of the side surface of the vehicle511is excluded from the extraction target.

For example, in the extraction condition setting processing of point cloud data, the processing illustrated in the flowchart ofFIG.15is executed.

In step S31, the extraction unit342determines whether or not the point cloud data is in a predetermined positional relationship.

In a case where it is determined that the point cloud data is in the predetermined positional relationship, the process proceeds to step S32, and the extraction unit342sets only the point cloud data corresponding to a part of the object region as the extraction target.

Specifically, in a case where a region of an adjacent lane in the vicinity of the subject vehicle is set, and point cloud data corresponding to the object region is arranged so as to indicate an object having a size of, for example, 5 m in the depth direction and 3 m in the horizontal direction in the region of the adjacent lane, it is regarded that the vehicle is traveling in the vicinity of the subject vehicle, and only the point cloud data corresponding to the horizontal direction (point cloud data of the vehicle rear surface) is extracted.

On the other hand, in a case where it is determined that the point cloud data is not in the predetermined positional relationship, step S32skipped, and the point cloud data corresponding to the entire object regions is set as the extraction target.

As described above, in a case where the vehicle is traveling in the vicinity of the subject vehicle, only the point cloud data of the rear surface of the vehicle can be set as the extraction target.

Note that in a case where, other than this, general clustering processing of the point cloud data corresponding to the object region is executed and the point cloud data continuous in an L shape in the depth direction and the horizontal direction is extracted, it is regarded that the vehicle is traveling in the vicinity of the subject vehicle, and only the point cloud data of the rear surface of the vehicle may be extracted. Furthermore, in a case where the variance of the distance indicated by the point cloud data corresponding to the object region is larger than a predetermined threshold, it is regarded that the vehicle is traveling in the vicinity of the subject vehicle, and only the point cloud data of the rear surface of the vehicle may be extracted.

Normally, as illustrated inFIG.16, for example, the point cloud data of LiDAR becomes denser as it is closer to the road surface and becomes sparse as it is farther from the road surface in an imaged image520. In the example ofFIG.16, the distance information of a traffic sign521existing at a position away from the road surface is generated on the basis of the point cloud data corresponding to its rectangular frame521F. However, the number of point cloud data corresponding to the object such as the traffic sign521or the traffic light not illustrated existing at a position away from the road surface is smaller than that of other objects existing at the position close to the road surface, and there is a possibility chat the reliability of the point cloud data becomes low.

Therefore, for an object existing at a position away from the road surface, the number of point cloud data corresponding to the object is increased by using a plurality of frames of point cloud data.

For example, in the extraction condition setting processing of point cloud data, the processing illustrated in the flowchart ofFIG.17is executed.

In step S51, the extraction unit342determines whether or not the object region of the recognized object exists higher than a predetermined height in the imaged image. The height mentioned here refers to a distance from the lower end to the upper end direction of the imaged image.

In a case where it is determined that the object region exists higher than the predetermined height in the imaged image, the process proceeds to step S52, and the extraction unit342sets the point cloud data of the plurality of frames corresponding to the object region as the extraction target.

For example, as illustrated inFIG.18, point cloud data531(t) obtained at time t, point cloud data531(t-1) obtained at time t-1, which is one frame before time t, and point cloud data531(t-2) obtained at time t-2, which is two frames before time t, are superimposed on an imaged image520(t) at current time t. Then, among the point cloud data531(t),531(t-1), and531(t-2), the point cloud data corresponding to the object region of the imaged image520(t) is set as the extraction target. Note that in a case where the subject vehicle is traveling at a high speed, the distance to the recognized object becomes closer by the time of the elapsed frame. Therefore, in the point cloud data531(t-1) and531(t-2), the distance information of the point cloud data corresponding to the object region is different from that of the point cloud data531(t). Therefore, the distance information of the point cloud data531(t-1) and531(t-2) is corrected on the basis of the distance traveled by the subject vehicle at the time of the elapsed frame.

On the other hand, in a case where it is determined that the object region does not exist higher than the predetermined height in the imaged image, step S52is skipped, and the point cloud data of one frame at the current time corresponding to the object region is set as the extraction target.

As described above, for an object existing at a position away from the road surface, the number of point cloud data corresponding to the object is increased by using a plurality of frames of point cloud data, and a decrease in the reliability of the point cloud data can be avoided.

For example, as illustrated inFIG.19, in a case where a signpost542is positioned above a vehicle541traveling in front of the subject vehicle in an imaged image540, the signpost542is sometimes included in a rectangular frame541F for the vehicle541. In this case, as the point cloud data corresponding to the rectangular frame541F, in addition to the point cloud data corresponding to the vehicle541, the point cloud data corresponding to the signpost542is also extracted.

In this case, since the vehicle541moves at a predetermined speed while the signpost542does not move, the point cloud data for the object that does not move is excluded from the extraction target.

For example, in the extraction condition setting processing of point cloud data, the processing illustrated in the flowchart ofFIG.20is executed.

In step S71, the extraction unit342determines whether or not the speed difference calculated on the basis of the time-series change of the point cloud data is larger than a predetermined threshold between the upper part and the lower part of the object region for the object recognized in the imaged image.

Here, it is determined whether or not the speed calculated on the basis of the point cloud data in the upper part of the object region is substantially 0, and moreover, a difference between the speed calculated on the basis of the point cloud data in the upper part of the object region and the speed calculated on the basis of the point cloud data in the lower part of the object region is obtained.

In a case where it is determined that the speed difference between the upper part and the lower part of the object region is larger than the predetermined threshold, the process proceeds to step S72, and the extraction unit342excludes the point cloud data corresponding to the upper part of the object region from the extraction target.

On the other hand, in a case where it is determined that the speed difference between the upper part and the lower part of the object region is not larger than the predetermined threshold, step S72is skipped, and the point cloud data corresponding to the entire object regions is set as the extraction target.

As described above, the point cloud data for an object that does not move such as a signpost or a signboard above the vehicle can be excluded from the extraction target.

In general, since LiDAR is susceptible to rain, fog, and dust, in rainy weather, the rangefinding performance of LiDAR deteriorates, and the reliability of the point cloud data extracted corresponding to the object region also decreases.

Therefore, by using the point cloud data of a plurality of frames depending on the weather, the point cloud data extracted corresponding to the object region is increased, and a decrease in the reliability of the point cloud data is avoided.

For example, in the extraction condition setting processing of point cloud data, the processing illustrated in the flowchart ofFIG.21is executed.

In step S91, the extraction unit342determines whether or not the weather is rainy.

For example, as the vehicle information obtained via the CAN332, the extraction unit342determines whether or not it rains on the basis of detection information from a raindrop sensor that detects raindrops in a detection area of the front windshield. Furthermore, the extraction unit342may determine whether or not it is rainy on the basis of the operation state of the wiper. The wiper may operate on the basis of detection information from the raindrop sensor, or may operate in response to an operation of the driver.

In a case where it is determined that the weather is rainy, the process proceeds to step S92, and the extraction unit342sets the point cloud data of a plurality of frames corresponding to the object region as the extraction target as described with reference toFIG.18.

On the other hand, in a case where it is determined that the weather is not rainy, step S92is skipped, and the point cloud data of one frame at the current time corresponding to the object region is set as the extraction target.

As described above, in rainy weather, by using the point cloud data of a plurality of frames, it is possible to increase point cloud data extracted corresponding to the object region and to avoid a decrease in the reliability of the point cloud data.

5. Configuration and Operation of Information Processing Apparatus

In the above, an example in which the present technology is applied to an evaluation apparatus that compares distance information of the recognition system with Point cloud data of the LiDAR in a so-called off-board manner has been described.

The present technology is not limited to this, and can also be applied to a configuration in which object recognition is performed in real time (on-board) in a traveling vehicle.

(Configuration of Information Processing Apparatus)

FIG.22is a block diagram illustrating the configuration of an information processing apparatus600that performs on-board object recognition.

FIG.22illustrates a first information processing unit620and a second information processing unit640constituting the information processing apparatus600. For example, the information processing apparatus600is configured as a part of the analysis unit61inFIG.1, and recognizes an object around the vehicle1by performing sensor fusion processing.

The first information processing unit620recognizes the object around the vehicle1on the basis of an imaged image obtained by the camera311and millimeter wave data obtained by the millimeter wave radar312.

The first information processing unit620includes a sensor fusion unit621and a recognition unit622. The sensor fusion unit621and the recognition unit622have functions similar to those of the sensor fusion unit321and the recognition unit322inFIG.4.

The second information processing unit640includes a conversion unit641, an extraction unit642, and a correction unit643. The conversion unit641and the extraction unit642have functions similar to those of the conversion unit341and the extraction unit342inFIG.4.

The correction unit643corrects distance information included in a recognition result from the first information processing unit620on the basis of point cloud data from the extraction unit642. The corrected distance information is output as a rangefinding result of the object that becomes the recognition target. Note that the accuracy of the corrected distance information can be further enhanced by using the mode value of the point cloud data existing in the rectangular frame as the point cloud data used for correction.

(Rangefinding Processing of Object)

Next, rangefinding processing of an object by the information processing apparatus600will be described with reference to the flowchart inFIG.23. The processing inFIG.23is executed on-board at a traveling vehicle.

In step S101, the extraction unit642acquires the recognition result of the object recognized in the imaged image from the first information processing unit620.

In step S102, the conversion unit641performs coordinate conversion on the point cloud data obtained by the LiDAR331.

In step S103, the extraction unit642sets, on the basis of the object, an extraction condition of the point cloud data corresponding to the object region of the object recognized in the imaged image by the first information processing unit20among the point cloud data converted into the camera coordinate system.

Specifically, the extraction condition setting processing of point cloud data described with reference to the flowcharts ofFIGS.12and13is executed.

In step S104, the extraction unit642extracts the point cloud data corresponding to the object region for the recognized object on the basis of the set extraction condition.

In step S105, the correction unit643corrects the distance information from the first information processing unit620on the basis of the point cloud data extracted by the extraction unit642. The corrected distance information is output as a rangefinding result of the object that becomes the recognition target.

According to the above processing, it is possible to narrow down the point cloud data corresponding to the recognition target, and it becomes possible to perform comparison between the distance information correction accurately with a low load. Furthermore, since the extraction condition (clustering condition) of the point cloud data is set according to the state of the object that is the recognition target, it is possible to more reliably extract the point cloud data corresponding to the object that is the recognition target. As a result, it is possible to correct distance information more accurately, and eventually it becomes possible to obtain the distance to the object more accurately, and it becomes possible to suppress false recognition (false detection) of the object, and to prevent detection omission of the object to be detected.

In the above-described embodiment, the sensor used in the sensor fusion processing is not limited to the millimeter wave radar, and may be a LiDAR or an ultrasonic sensor. Furthermore, the sensor data obtained by the rangefinding sensor is not limited to point cloud data obtained by the LiDAR, and distance information indicating the distance to the object obtained by the millimeter wave radar may be used.

Although an example in which the vehicle is the recognition target has been mainly described above, a discretionary object other than a vehicle can be the recognition target.

Furthermore, the present technology can also be applied to a case of recognizing a plurality of types of objects.

Furthermore, in the above description, an example of recognizing an object in front of the vehicle1has been described, but the present technology can also be applied to a case of recognizing an object in another direction around the vehicle1.

Moreover, the present technology can also be applied to a case of recognizing an object around a moving body other than a vehicle. For example, moving bodies such as a motorcycle, a bicycle, a personal mobility, an airplane, a ship, a construction machine, and an agricultural machine (tractor) are assumed. Furthermore, the moving body to which the present technology can be applied includes, for example, a moving body that is remotely driven (operated) without being boarded by a user, such as a drone or a robot.

Furthermore, the present technology can also be applied to a case of performing recognition processing of a target at a fixed place such as a monitoring system, for example.

6. Configuration Example of Computer

The above-described series of processing can be executed by hardware and can be executed by software. In a case where the series of processing is executed by software, a program constituting the software is installed from a program recording medium to a computer incorporated in dedicated hardware, a general-purpose personal computer, and the like.

FIG.24is a block diagram illustrating the configuration example of hardware of a computer that executes the above-described series of processing by a program.

The evaluation apparatus340and the information processing apparatus600described above are achieved by a computer1000having the configuration illustrated inFIG.24.

A CPU1001, a RPM1002, and a RAM1003are connected to one another by a bus1004.

An input/output interface1005is further connected to the bus1004. An input unit1006including a keyboard and a mouse, and an output unit1007including a display and a speaker are connected to the input/output interface1005. Furthermore, a storage unit1008including a hard disk and a nonvolatile memory, a communication unit1009including a network interface, and a drive1010that drives a removable medium1011are connected to the input/output interface1005.

In the computer1000configured as described above, for example, the CPU1001loads, into the RAM1003via the input/output interface1005and the bus1004, and executes a program stored in the storage unit1008, whereby the above-described series of processing is performed.

The program executed by the CPU1001is provided, for example, by being recorded in the removable medium1011or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit1008.

Note that the program executed by the computer1000may be a program in which processing is performed in time series in the order described in the present description, or may be a program in which processing is performed in parallel or at necessary timing such as when a call is made.

In the present description, a system means a set of a plurality of constituent elements (apparatuses, modules (components), and the like), and it does not matter whether or not all the constituent elements are in the same housing. Therefore, a plurality of apparatuses housed in separate housings and connected via a network and one apparatus in which a plurality of modules is housed in one housing are both systems.

The embodiment of the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present technology.

Furthermore, the effects described in the present description are merely examples and not to be limited to this, and other effects may be present.

Moreover, the present technology can have the following configurations.

An information processing apparatus including:

an extraction unit that extracts, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image among the sensor data obtained by a rangefinding sensor.

The information processing apparatus according to (1), in which

the extraction unit sets an extraction condition of the sensor data on the basis of the object having been recognized.

The information processing apparatus according to (2), in which

the extraction unit excludes, from an extraction target, the sensor data corresponding to a region overlapping another object region for another object in the object region.

The information processing apparatus according to (2) or (3), in which

the extraction unit excludes, from an extraction target, the sensor data in which a difference between a speed of the object having been recognized and a speed calculated on the basis of a time-series change of the sensor data is larger than a predetermined speed threshold.

The information processing apparatus according to any of (2) to (4), in which

the extraction unit excludes, from an extraction target, the sensor data in which a distance to the object having been recognized is larger than a predetermined distance threshold.

The information processing apparatus according to (5), in which

the extraction unit sets the distance threshold in accordance with the object having been recognized.

The information processing apparatus according to (6), in which

the camera and the rangefinding sensor are mounted on a moving body, and

the extraction unit changes the distance threshold in accordance with a moving speed of the moving body.

The information processing apparatus according to any of (2) to (7), in which

in a case where the object region is larger than a predetermined area, the extraction unit sets only sensor data corresponding to a vicinity of a center of the object region as an extraction target.

The information processing apparatus according to (8), in which

in a case where the object region is smaller than a predetermined area, the extraction unit sets sensor data corresponding to an entirety of the object region as an extraction target.

The information processing apparatus according to any of to (9), in which

in a case where the sensor data corresponding to the object region is in a predetermined positional relationship, the extraction unit sets only the sensor data corresponding to a part of the object region as an extraction target.

The information processing apparatus according to any of to (10), in which

in a case where the object region exists higher than a predetermined height in the imaged image, the extraction unit sets sensor data of a plurality of frames corresponding to the object region as an extraction target.

The information processing apparatus according to any of (2) to (11), in which

in a case where a difference is speed calculated on the basis of time-series change in the sensor data between an upper part and a lower part of the object region is larger than a predetermined threshold, the extraction unit excludes, from an extraction target, the sensor data corresponding to an upper part of the object region.

The information processing apparatus according to any of to (12), in which

the extraction unit sets, as an extraction target, sensor data of a plurality of frames corresponding to the object region in accordance with weather.

The information processing apparatus according to any of (1) to (13) further including:

a comparison unit that compares the sensor data extracted by the extraction unit with distance information obtained by sensor fusion processing based on the imaged image and other sensor data.

The information processing apparatus according to any of (1) to (13) further including:

a sensor fusion unit that performs sensor fusion processing based on the imaged image and other sensor data; and

a correction unit that corrects distance information obtained by the sensor fusion processing on the basis of the sensor data extracted by the extraction unit.

The information processing apparatus according to any of (1) to (15), in which

the rangefinding sensor includes a LiDAR, and

the sensor data is point cloud data.

The information processing apparatus according to any of (1) to (15), in which

the rangefinding sensor includes a millimeter wave radar, and

the sensor data is distance information indicating a distance to the object.

An information processing method, in which

an information processing apparatus extracts, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image among the sensor data obtained by a rangefinding sensor.

A program for causing a computer to execute

processing of extracting, on the basis of an object recognized in an imaged image obtained by a camera, sensor data corresponding to an object region including the object in the imaged image among the sensor data obtained by a rangefinding sensor.

REFERENCE SIGNS LIST