Perceiving roadway conditions from fused sensor data

The present invention extends to methods, systems, and computer program products for perceiving roadway conditions from fused sensor data. Aspects of the invention use a combination of different types of cameras mounted to a vehicle to achieve visual perception for autonomous driving of the vehicle. Each camera in the combination of cameras generates sensor data by sensing at least part of the environment around the vehicle. The sensor data form each camera is fused together into a view of the environment around the vehicle. Sensor data from each camera (and, when appropriate, each other type of sensor) is fed into a central sensor perception chip. The central sensor perception chip uses a sensor fusion algorithm to fuse the sensor data into a view of the environment around the vehicle.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not applicable.

BACKGROUND

1. Field of the Invention

This invention relates generally to the field of autonomous vehicles, and, more particularly, to perceiving roadway conditions from fused sensor data.

2. Related Art

Autonomous driving solutions typically use LIDAR sensors to perceive the environment surrounding a vehicle. LIDAR sensors are mounted on a vehicle, often on the roof. The LIDAR sensors have moving parts enabling sensing of the environment 360-degrees around the vehicle out to a distance of around 100-150 meters. Sensor data from the LIDAR sensors is processed to perceive a “view” of the environment around the vehicle. The view is used to automatically control vehicle systems, such as, steering, acceleration, braking, etc. to navigate within the environment. The view is updated on an ongoing basis as the vehicle navigates (moves within) the environment.

DETAILED DESCRIPTION

The present invention extends to methods, systems, and computer program products for perceiving roadway conditions from fused sensor data.

Many autonomous driving vehicles use LIDAR sensors to view the environment around a vehicle. However, LIDAR sensors are relatively expensive and include mechanical rotating parts. Further, LIDAR sensors are frequently mounted on top of vehicles limiting aesthetic designs. Aspects of the invention can provide reliable autonomous driving with lower cost sensors and improved aesthetics.

For example, aspects of the invention use a combination of different types of cameras mounted to a vehicle to achieve visual perception for autonomous driving of the vehicle. Each camera in the combination of cameras generates sensor data by sensing at least part of the environment around the vehicle. The sensor data form each camera is fused together into a view of the environment around the vehicle.

The combination of cameras can include a multipurpose time-of-flight (TOF) camera with a processing chip, such as, for example, a Red-Green-Blue-Infrared (RGB-IR) complementary metal-oxide semiconductor (CMOS) chip. The TOF camera is configured to switch between camera mode and LIDAR mode. The TOF camera uses a narrow field-of-view (FOV) lens to facilitate longer distance object tracking capability. The TOF camera can be used for collision control and adaptive cruise control.

The combination of cameras can also include one or more RGB-IR cameras. The one or more RGB-IR cameras use a wider FOV to sense objects closer to the vehicle. Intensity information from the IR signals can be used during the night as well as in other low (or no) light environments.

In one aspect, one or more other sensors are also mounted to the vehicle. The one or more other sensors can be of the same or different sensor types, such as, for example, Radar, ultrasonic, global positioning system (GPS), inertial measurement unit (IMU), etc. The one or more other sensors also generate sensor data by sensing at least part of the environment around the vehicle.

Sensor data from each camera (and, when appropriate, each other type of sensor) is fed into a central sensor perception chip. The central sensor perception chip uses a sensor fusion algorithm to fuse the sensor data into a view of the environment around the vehicle.

Aspects of the invention can be implemented in a variety of different types of computing devices.FIG. 1illustrates an example block diagram of a computing device100. Computing device100can be used to perform various procedures, such as those discussed herein. Computing device100can function as a server, a client, or any other computing entity. Computing device100can perform various communication and data transfer functions as described herein and can execute one or more application programs, such as the application programs described herein. Computing device100can be any of a wide variety of computing devices, such as a mobile telephone or other mobile device, a desktop computer, a notebook computer, a server computer, a handheld computer, tablet computer and the like.

Computing device100includes one or more processor(s)102, one or more memory device(s)104, one or more interface(s)106, one or more mass storage device(s)108, one or more Input/Output (I/O) device(s)110, and a display device130all of which are coupled to a bus112. Processor(s)102include one or more processors or controllers that execute instructions stored in memory device(s)104and/or mass storage device(s)108. Processor(s)102may also include various types of computer storage media, such as cache memory.

Memory device(s)104include various computer storage media, such as volatile memory (e.g., random access memory (RAM)114) and/or nonvolatile memory (e.g., read-only memory (ROM)116). Memory device(s)104may also include rewritable ROM, such as Flash memory.

Mass storage device(s)108include various computer storage media, such as magnetic tapes, magnetic disks, optical disks, solid state memory (e.g., Flash memory), and so forth. As depicted inFIG. 1, a particular mass storage device is a hard disk drive124. Various drives may also be included in mass storage device(s)108to enable reading from and/or writing to the various computer readable media. Mass storage device(s)108include removable media126and/or non-removable media.

I/O device(s)110include various devices that allow data and/or other information to be input to or retrieved from computing device100. Example I/O device(s)110include cursor control devices, keyboards, keypads, barcode scanners, microphones, monitors or other display devices, speakers, printers, network interface cards, modems, cameras, lenses, radars, CCDs or other image capture devices, and the like.

Display device130includes any type of device capable of displaying information to one or more users of computing device100. Examples of display device130include a monitor, display terminal, video projection device, and the like.

Interface(s)106include various interfaces that allow computing device100to interact with other systems, devices, or computing environments as well as humans. Example interface(s)106can include any number of different network interfaces120, such as interfaces to personal area networks (PANs), local area networks (LANs), wide area networks (WANs), wireless networks (e.g., near field communication (NFC), Bluetooth, Wi-Fi, etc., networks), and the Internet. Other interfaces include user interface118and peripheral device interface122.

Bus112allows processor(s)102, memory device(s)104, interface(s)106, mass storage device(s)108, and I/O device(s)110to communicate with one another, as well as other devices or components coupled to bus112. Bus112represents one or more of several types of bus structures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

FIG. 2illustrates an example roadway environment200that facilitates perceiving roadway conditions from fused sensor data. Roadway environment200includes vehicle201, such as, for example, a car, a truck, or a bus. Vehicle201may or may not contain any occupants, such as, for example, one or more passengers. Roadway environment200can include roadway markings (e.g., lane boundaries), pedestrians, bicycles, other vehicles, signs, or any other types of objects. Vehicle201can be moving within roadway environment200, such as, for example, driving on a road.

Vehicle201includes camera211A, camera211B, camera221, perception chip203, perception algorithm202, vehicle control systems204, and vehicle operating components206. Cameras211A,211B, and221can be mounted to vehicle201to face in the direction vehicle201is moving (e.g., forward or backwards). In one aspect, vehicle201also includes a corresponding set of cameras facing the other direction. For example, vehicle201can include a set of from facing cameras and a set of rear facing cameras.

Camera211A includes sensor components212A (e.g., a lens, an aperture, a shutter, a sensor plate, an IR emitter, an IR detector, etc.) and application-specific integrated circuit (ASIC)213A. ASIC213A can include digital signal processing (DSP) functionality to perform various operations on image sensor data captured by sensor components212A. Similarly, camera211B includes sensor components212B (e.g., a lens, aperture, shutter, sensor plater, an IR emitter, an IR detector, etc.) and application-specific integrated circuit (ASIC)213B. ASIC213B can include digital signal processing (DSP) functionality to perform various operations on image sensor data captured by sensor components212B.

Cameras211A and211B can be similar types, or even the same type, of camera. Cameras211A and211B have fields-of-view216A and216B respectively. Fields-of-view216A and216B can be similar and possibly even essentially the same. Within fields-of-view216A and216B, cameras211A and211B respectively can sense roadway environment200from vehicle201out to approximately distance threshold231.

In one aspect, cameras211A and211B are Red-Green-Blue/Infrared (RGB/IR) cameras. Thus, cameras211A and211B can generate images where each image section includes a Red pixel, a Green pixel, a Blue pixel, and an IR pixel. The RGB pixel intensities are used when there is sufficient light (e.g., during daytime). The intensity information from the IR pixels can be used during the night as well as in other low (or no) light environments to sense roadway environment200. Low (or no) light environments can include travel through tunnels or other environments where natural light is obstructed.

Camera221includes sensor components222(e.g., a lens, an aperture, a shutter, a sensor plate, a laser, a sensor for detecting laser reflections, etc.) and application-specific integrated circuit (ASIC)223. ASIC223can include digital signal processing (DSP) functionality to perform various operations on image sensor data captured by sensor components222. Camera221has field-of-view226. Within field-of-view226, camera221can sense roadway environment200from vehicle201to beyond distance threshold232. Camera221can be used for collision control and adaptive cruise control.

In one aspect, camera221is a multipurpose time-of-flight (TOF) camera with a processing chip, such as, for example, a Red-Green-Blue-Infrared (RGB-IR) complementary metal-oxide semiconductor (CMOS) chip. Camera221can be configured to operate in different modes when objects are sensed at different distances from vehicle201. Camera221can operate primarily in LIDAR mode when objects within field-of-view226are detected beyond distance threshold232. On the other hand, camera221can switch between camera mode and LIDAR mode when objects within field-of-view226are detected between distance threshold231and distance threshold232. For example, camera221can operate approximately 50% of the time in LIDAR mode and 50% of the time in camera mode when objects within field-of-view226are detected between distance threshold231and distance threshold232.

Camera221can generate images where each image section includes a Red pixel, a Green pixel, a Blue pixel, and an IR pixel. The RGB pixel intensities are used when there is sufficient light (e.g., during daytime). In LIDAR mode, the laser emits a pulse of IR wavelength. A processing chip (e.g., within ASIC223) reads the time of flight information to process depth of objects. The processing chip can set appropriate IR pixel intensity information based on object depths. LIDAR mode and IR pixel intensity can be used during the night, in other low (or no) light environments, or when otherwise appropriate, to sense roadway environment200.

In one aspect, distance threshold231is approximately 20 meters from vehicle201and distance threshold232is approximately 100 meters from vehicle201. However, other distances for distance thresholds231and232are also possible. For example, distance threshold231can range from 0-20 meters and distance threshold232can range from 20-200 meters.

Perception chip203can be a general or special purpose processing unit, such as for example, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), etc. Alternately and/or in combination, perception chip203can also include logic circuits, such as, for example, in an ASIC or Field-Programmable Gate Array (FPGA). Perception algorithm202runs on perception chip203.

In general, perception algorithm202is configured to fuse sensor data from different cameras (and possibly other sensors) into a view of roadway environment200. Perception algorithm202can process camera sensor data to identify objects of interest within roadway environment200. Perception algorithm202can also classify objects of interest within roadway environment200. Object classifications can include: lane boundaries, cross-walks, signs, control signals, cars, trucks, pedestrians, etc. Some object classifications can have sub-classifications. For example, a sign can be classified by sign type, such as, a stop sign, a yield sign, a school zone sign, a speed limit sign, etc.

Perception algorithm202can also determine the location of an object within roadway environment200, such as, for example, between vehicle201and distance threshold231, between distance threshold231and distance threshold232, or beyond distance threshold232. If an object is moving, perception algorithm202can also determine a likely path of the object.

In one aspect, perception algorithm202includes a neural network architected in accordance with a multi-layer (or “deep”) model. A multi-layer neural network model can include an input layer, a plurality of hidden layers, and an output layer. A multi-layer neural network model may also include a loss layer. For classification of fused camera sensor data (e.g., an image), values in the sensor data (e.g., pixel-values) are assigned to input nodes and then fed through the plurality of hidden layers of the neural network. The plurality of hidden layers can perform a number of non-linear transformations. At the end of the transformations, an output node yields a perceived view of roadway environment200.

Perception algorithm202can send roadway views to vehicle control systems204. Vehicle control systems204(e.g., cruise control, lane changing, collision avoidance, braking, steering, etc.) are configured to control vehicle operating components206(e.g., accelerator, brakes, steering wheel, transmission, etc.) to autonomously operate vehicle201in roadway environment200. Vehicle control systems204can change the configuration of vehicle operating components206based on roadway views received from perception algorithm202. Changes to vehicle operating components206can facilitate turning vehicle201, speeding up vehicle201, slowing down vehicle201, reversing the direction of vehicle201, etc.

Vehicle201can also include one or more other sensors. The one or more other sensors can be of the same or different sensor types, such as, for example, Radar, ultrasonic, inertial measurement unit (IMU), global positioning system (GPS), etc. The one or more other sensors can also generate sensor data by sensing at least part of roadway environment200. Perception algorithm202can fuse sensor data from these other sensors with sensor data from cameras211A,211B, and221(as well as any other cameras) into a view of roadway environment200.

FIG. 3illustrates a flow chart of an example method300for perceiving roadway conditions from fused sensor data. Method300will be described with respect to the components and data of environment200.

Method300includes for each of one or more cameras mounted to a vehicle, receiving sensor data from the camera, the sensor data sensed from an area between the vehicle and a specified distance away from the vehicle within a field-of-view of the camera, each of the one or more cameras having the field-of-view (301). For example, perception algorithm202can receive sensor data214A from camera211A. Camera211A can sense sensor data214A from an area between vehicle201and distance threshold231within field-of-view216A. Camera211A can send sensor data214A to perception algorithm202. Similarly, perception algorithm202can receive sensor data214B from camera211B. Camera211B can sense sensor data214B from an area between vehicle201and distance threshold231within field-of-view216B. Camera211B can send sensor data214B to perception algorithm202. Objects within fields-of-view216A and216B can be clustered and tracked through image information. Sensor data214A and214B can indicate objects being tracked within fields-of-view216A and216B.

Method300includes for another camera mounted to the vehicle, receiving other sensor data from the other camera. the other sensor data sensed from an area beyond the specified distance within another field-of-view of the other camera, the other field-of-view being narrower than the field-of-view (302). For example, perception algorithm202can receive sensor data224from camera221. In general, camera221can sense sensor data224anywhere within field-of-view226. Camera221can send sensor data224to perception algorithm202. Using different mechanisms, objects within field-of-view226can tracked. Sensor data224can indicate objects being tracked within field-of-view226.

In one aspect, camera221operates in LIDAR mode to sense sensor data224from an area beyond distance threshold232within field-of-view226. Beyond distance threshold232, objects can be tracked based on three-dimensional (3D) point clouds. In another aspect, camera221switches between camera mode and LIDAR mode to sense sensor data224from an area between distance threshold231and distance threshold232within field-of-view226. Between distance threshold231and distance threshold232, objects can be clustered and tracked through both 3D point clouds and image information.

In a further aspect, camera211senses sensor data224from an area around (and potentially somewhat inside of) distance threshold231, for example, when objects are close to the area covered by cameras211A and211B. Camera221estimates the position of objects through a motion tracking, such as, for example, Kalman filter, optical flow, etc.

Method300includes fusing the sensor data from each of the one or more cameras and the other sensor data into a roadway view of the roadway environment (303). For example, perception algorithm202can fuse sensor data214A, sensor data214B, and sensor data224into roadway view226of road way environment200. Roadway view236can indicate objects that are being tracked with in fields-of-view216A,216B, and226.

Method300includes sending the roadway view to vehicle control components so that the vehicle control components can adjust the configuration of vehicle operating components based on the roadway view (304). For example, perception algorithm can send roadway view236to vehicle control systems204. Vehicle control systems204can receive roadway view236from perception algorithm202. Vehicle control systems204can use roadway view236to adjust vehicle operating components206. Adjusting vehicle operating components206can change the configuration of vehicle201to adapt to roadway environment200. For example, vehicle control systems204can adjust vehicle operating components206to increase the speed of vehicle201(e.g., accelerate), to decrease the speed of vehicle201(e.g., decelerate or brake), to reverse directions of vehicle201(e.g., change gears from drive to reverse or vice versa), change the trajectory of vehicle201for a turn or lane merge (e.g., adjust the angle of front tires), etc.

Cameras211A,211B, and221(as well as any other cameras and/or other sensors) can collect sensor data on an ongoing basis as vehicle201is in motion. The sensor data can be sent to perception algorithm202. Perception algorithm202can fuse the sensor data into additional roadway views. Vehicle control systems204can use the additional roadway views to adjust vehicle operating components206on an ongoing basis to autonomously operate vehicle201within roadway environment200in a safe manner.

FIG. 4illustrates an example vehicle401that can perceive roadway conditions from fused sensor data. As depicted, vehicle401has front mounted cameras411A,411B, and421. Front mounted cameras411A,411B, and421have corresponding fields-of-view416A,416B, and426respectively. Cameras411A and411B can be RGB/IR cameras. Cameras411A and411B can sense objects between vehicle401and distance threshold431that are within fields-of-view416A and416B. Cameras411A and411B can use image information to track objects within fields-of-view416A and416B.

Camera421can be a Time-Of-Flight (TOF) based camera capable of switching between a camera mode and a LIDAR mode. Camera421can sense objects within field-of-view426. Camera421can use different mechanisms to sense objects depending on how far away the objects are from vehicle401. Camera421can primarily use LIDAR mode and 3D point clouds to track objects outside of distance threshold432. Camera421can switch between camera mode and LIDAR mode and use 3D point clouds and image information to track objects between distance thresholds431and432. Camera421can primarily use camera mode and image information to track objects approaching or that recently moved inside of distance threshold431.

A fusion algorithm can fuse together sensor data received from front mounted cameras411A,411B, and421into a roadway view of the road in front of vehicle401. Vehicle control systems can use the roadway view to adjust vehicle operating controls to autonomously operate vehicle401on the roadway in a safe manner.

FIG. 5illustrates an example data flow500for collecting sensor data for fusion into a roadway view. Data flow500starts at501. In502, a laser is triggered for a smaller field-of-view for a Time-of-Flight (TOF) camera. In503, objects (e.g., in a roadway) are tracked based 3D point clouds read from the TOF camera. The 3D point cloud includes RGB pixels as well as IR pixels. The TOF camera detects504if any objects are within a further distance threshold (e.g.,232,432, etc.). If not, data flow500loop back to502. If so, an object may be between the further distance threshold and a closer distance threshold (e.g.,231,431, etc.) and data flow proceeds to505and506.

At505and506, the TOF camera is switched between laser mode and camera mode. At507, objects (e.g., within the roadway) are tracked based 3D point clouds with depth read from the TOF camera. At509, objects (e.g., within the roadway) are tracked using image information. At508, a fusion algorithm fuses sensor data for object tracking with depth. At510, the TOF camera detects if any objects are within the closer distance threshold. If not, data flow500loops back to505and506.

If so, data flow500proceeds to511. At511, the TOF camera uses a Kalman filter or optical flow to estimate object positions. At512, the TOF camera sends objects position to a perception algorithm. At513, large field-of-view RGB/IR cameras generate image data. Objects are tracked from within the image data. At514, tracked objects (e.g., within the roadway) are sent to the perception algorithm. At515, other sensors generate sensor data. Objects are tracked within the sensor data. At516, tracked objects (e.g., within the roadway) are sent to the perception algorithm. At517, the fusion algorithm fuses tracked objects into a roadway view (e.g., of the roadway).

FIG. 6illustrates an example of a chip600with RGB-IR pixels. In one aspect, chip600is a CMOS/Charge-Coupled Device (CCD) chip. As depicted, chip600has four different kinds of pixels: Red pixels, Green pixels, Blue pixels, and IR pixels. Pixels are grouped into sections. Each section includes one Red pixel, one Green pixel, one Blue pixel, and one IR pixel. For example, section601includes Red pixel611, Green pixel612, Blue pixel613, and IR pixel614. The sections are arranged into rows and columns, including row602and column603.

Chip600can be used to store local sensor information in a camera (e.g., pixel intensities). An ASIC within the camera can process the local sensor information stored in the chip. The ASIC can forward the processed sensor information to a central sensor perception chip, such as, for example, perception chip203. At the central sensor perception chip, a fusion algorithm can fuse the processed sensor information along with other processed sensor data from other cameras (and/or other sensors) into a roadway view. The roadway view can be used to facilitate autonomous operation of a vehicle.

In one aspect, one or more processors are configured to execute instructions (e.g., computer-readable instructions, computer-executable instructions, etc.) to perform any of a plurality of described operations. The one or more processors can access information from system memory and/or store information in system memory. The one or more processors can transform information between different formats, such as, for example, sensor data, image data, pixel values, pixel intensities, roadway views, etc.

System memory can be coupled to the one or more processors and can store instructions (e.g., computer-readable instructions, computer-executable instructions, etc.) executed by the one or more processors. The system memory can also be configured to store any of a plurality of other types of data generated by the described components, such as, for example, sensor data, image data, pixel values, pixel intensities, roadway views, etc.