Fusion of motion and appearance features for object detection and trajectory prediction

Techniques and examples pertaining to objection detection and trajectory prediction for autonomous vehicles are described. A processor receives an input stream of image frames and fuses a spatiotemporal input stream of the image frames and an appearance-based stream of the image frames using a deep neural network (DNN) to generate an augmented stream of the image frames. The processor performs object detection and trajectory prediction of one or more objects in the image frames based on the augmented stream.

TECHNICAL FIELD

The present disclosure generally relates to autonomous vehicles and, more particularly, to a system of objection detection and trajectory prediction for autonomous vehicles.

BACKGROUND

Autonomous vehicles are generally capable of sensing the environment and navigating without human input. As a requirement, autonomous vehicles need to be able to attend to and classify potentially moving objects in dynamic surroundings. However, the capability of tracking multiple objects within video sequences and predicting where the multiple objects are going to be located in the future remains a challenge. While existing efforts attained results in predicting trajectories of an object based on previous locations of the object, the models used tend to lack the capability to extract spatiotemporal feature dynamics from videos to enhance detections and improve trajectory predictions for object tracking.

DETAILED DESCRIPTION

State of the art object detection within images and videos has focused on first extracting feature maps from static images and then feeding the extracted feature maps into classification and regression models for object detection, classification and localization, respectively. Under proposed schemes and concepts in accordance with the present disclosure, a spatiotemporal motion input stream (herein interchangeably referred as “spatiotemporal input stream”) may be utilized to augment an RGB appearance-based stream for enhanced object detection and trajectory prediction within videos/image frames. A focus of the proposed schemes and concepts in accordance with the present disclosure is on the use of deep convolutional neural networks for fusing a spatiotemporal input stream and the appearance-based stream in order to predict future object locations.

The spatiotemporal input stream may be derived, for example and without limitation, from optical flow calculations, spatiotemporal filters and/or network stream specialized in extracting motion information. The input streams may be derived from RGB image sequences, and may encode motion information derived from how pixels are changing from frame to frame. For instance, a dense optical flow may track the angle and magnitude of how each pixel moves between sequential frames.

FIG. 1illustrates example architectures110,120and130with which two input streams can be combined in accordance with an embodiment of the present disclosure.FIG. 1shows conceptually how two input streams of image frames may be combined or otherwise fused together within a shallow convolutional neural network (CNN) composed of alternating convolutional and pooling layers. The two input streams may include a spatiotemporal input stream of image frames and an appearance-based stream of image frames, and may be concatenated for use by utilizing architectures110, architectures120or architectures130.

Referring to part (A) ofFIG. 1, architecture110is herein referred as an “early-fusion” architecture. In architecture110, the two input streams (e.g., a spatiotemporal input stream of image frames and an appearance-based stream of image frames), labeled as “motion input” and “appearance input” inFIG. 1, may be stacked as inputs for object detection and trajectory prediction without any individual processing by the CNN.

Referring to part (B) ofFIG. 1, architecture120is herein referred as a “late-fusion” architecture. In architecture120, the two input streams (e.g., a spatiotemporal input stream of image frames and an appearance-based stream of image frames), labeled as “motion input” and “appearance input” inFIG. 1, may be processed by the CNN in two separate streams/stacks to create two separate sets of feature maps that are combined before object detection and trajectory prediction.

Referring to part (C) ofFIG. 1, architecture130is herein referred as a “slow-fusion” architecture. In architecture130, the two input streams (e.g., a spatiotemporal input stream of image frames and an appearance-based stream of image frames), labeled as “motion input” and “appearance input” inFIG. 1, may be processed individually for a few convolutional and pooling layers and then processed together in a single stack before object detection and trajectory prediction.

The combining or fusion of the appearance-based stream and the spatiotemporal input stream may be enhanced with recurrent connections. Connections of recurrent neural networks (RNNs) may enable the networks to use outputs from previous image frames as inputs to current image frames, and thus allow neutral networks to maintain state information. For example, in an event that a network detects a vehicle at a location within a current image frame, the current state of the network may be impacted and it may be more likely for the network to detect the vehicle at that location in a subsequent image frame. Recurrent connections may be utilized within the convolutions of RNNs employed for feature extraction. By maintaining state information, the network(s) may aggregate motion information over time to aid in future predictions.

FIG. 2illustrates an example scenario200in which embodiments in accordance with the present disclosure may be utilized. Scenario200is an illustrative and non-limiting example of how recurrent connections may be implemented for prediction of object bounding boxes one time step in the future. Recurrent connections may also be used during the final stage of object classification and trajectory prediction.

Referring toFIG. 2, at time t=0, two input streams (e.g., a spatiotemporal input stream of image frames and an appearance-based stream of image frames), labeled as “motion” and “image” inFIG. 2, may be fused by a CNN to create a first set of spatial feature maps, which may be provided as inputs to a convolutional RNN to create a first set of spatiotemporal feature maps as inputs to a detector at time t=0 for measurement. At time t=1, the first set of spatiotemporal feature maps may be processed by convolutions to provide a second set of spatiotemporal feature maps as inputs to a detector at time t=1 for prediction. Also, at time t=1, two input streams (e.g., a spatiotemporal input stream of image frames and an appearance-based stream of image frames), labeled as “motion” and “image” inFIG. 2, may be fused by a CNN to create a second set of spatial feature maps, which may be provided along with the second set of spatiotemporal feature maps as inputs to a convolutional RNN to create a third set of spatiotemporal feature maps as inputs to a detector at time t=1 for measurement. At time t=2, the third set of spatiotemporal feature maps may be processed by convolutions to provide a fourth set of spatiotemporal feature maps as inputs to a detector at time t=2 for prediction. Also, at time t=2, two input streams (e.g., a spatiotemporal input stream of image frames and an appearance-based stream of image frames), labeled as “motion” and “image” inFIG. 2, may be fused by a CNN to create a third set of spatial feature maps, which may be provided along with the fourth set of spatiotemporal feature maps as inputs to a convolutional RNN to create a fifth set of spatiotemporal feature maps as inputs to a detector at time t=2 for measurement. At time t=3, the fifth set of spatiotemporal feature maps may be processed by convolutions to provide a sixth set of spatiotemporal feature maps as inputs to a detector at t=3 for prediction. The above-described operations may continue as recurrence.

FIG. 3illustrates an example system300in accordance with an embodiment of the present disclosure. System300may include a vehicle350which may be an autonomous vehicle or a manually-driven vehicle. Vehicle350may include an apparatus305, which may be an electronic control unit (ECU) of vehicle350. Apparatus305may include a processor310. In some embodiments, apparatus305may also include one or more image sensors340(1)-340(N), with N being a positive integer equal to or greater than 1. Each of the one or more image sensors340(1)-340(N) may be, for example and without limitation, a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) image sensor, and may be capable of capturing one or more video images. The captured video images may be toward the front of vehicle350, the rear of vehicle350and/or any suitable direction(s) around vehicle350.

Processor310may be communicatively coupled to each of the one or more image sensors340(1)-340(N) via wireless and/or wired medium(s) to receive the video images from the one or more image sensors340(1)-340(N) as an input stream of image frames. Processor310may generate a spatiotemporal input stream of the image frames from the received stream of the image frames, and generate an appearance-based stream (e.g., RGB appearance-based stream) of the image frames from the received stream of the image frames. Processor310may then combine or otherwise fuse the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using one or more deep neural network (DNNs) to generate an augmented stream of the image frames. For instance, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, processor310may augment the appearance-based stream of the image frames with the spatiotemporal input stream of the image frames to generate the augmented stream of the image frames. In generating the spatiotemporal input stream of the image frames, processor310may perform either of the following: (1) generating the spatiotemporal input stream of the image frames using optical flow calculations and spatiotemporal filters; or (2) generating the spatiotemporal input stream of the image frames using a network stream which is adapted to extract motion information from the input stream of the image frames. With the augmented stream of the image frames, processor310may perform object detection and trajectory prediction of one or more objects in the image frames based on the augmented stream.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, processor310may fuse the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using a convolutional neural network (CNN) that concatenates the spatiotemporal input stream and the appearance-based stream.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, processor310may concatenate the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with an early-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are separately stacked, without individual processing, as inputs for object detection and trajectory prediction. Alternatively, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, processor310may concatenate the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a late-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed in two separate streams to create two separate sets of feature maps that are combined to form an input for object detection and trajectory prediction. Alternatively, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, processor310may concatenate the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a slow-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed separately for one or more layers before being combined to form an input for object detection and trajectory prediction.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, processor310may fuse the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using recurrent neural networks (RNNs) with recurrent connections to utilize outputs from a previous video frame of the image frames as inputs to a current video frame of the image frames.

In some embodiments, apparatus305may also include a storage device330(e.g., a memory device) that stores one or more sets of processor-executable instructions or codes335therein. Processor310may execute the instructions or codes335to render the DNN, CNN and/or RNNs to perform the above-described operations.

In some embodiments, the DNN, CNN and/or RNNs may be implemented in or otherwise executed by one or more remote servers380. In such cases, apparatus305may include a transceiver320capable of wirelessly communicating with the one or more remote servers380, which is connected to a network370, via base station360. In some embodiments, transceiver320may wirelessly communicate directly with base station360. Alternatively or additionally, transceiver320may wirelessly communicate indirectly with base station360via one or more other vehicles (not shown) and/or one or more other base stations (not shown). That is, transceiver320may be capable of wireless communications using one or more radio access technologies and in compliance with one or more wireless communications protocols, standards and specifications.

FIG. 4illustrates an example process400in accordance with the present disclosure. Process400may include one or more operations, actions, or functions shown as blocks such as410,420and430as well as sub-blocks422,424,426and428. Although illustrated as discrete blocks, various blocks of process400may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the desired implementation. The blocks of process400may be implemented in the order shown inFIG. 4and, alternatively, in a different order. Process400may be implemented by or in apparatus300. Process400may begin with block410.

At420, process400may involve processor310fusing a spatiotemporal input stream of the image frames and an appearance-based stream of the image frames (e.g., RGB appearance-based stream) using a deep neural network (DNN) to generate an augmented stream of the image frames. In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310performing a number of operations as represented by sub-blocks422,424,426and428to be described below. Process400may proceed from420to430.

At430, process400may involve processor310performing object detection and trajectory prediction of one or more objects in the image frames based on the augmented stream. For example, based on the augmented stream, processor310may detect one or more objects in the image frames and then determine or otherwise predict a path of movement, a trajectory or a location at a future time for each of the one or more objects.

At422, process400may involve processor310generating the spatiotemporal input stream of the image frames from the received stream of the image frames. In some embodiments, in generating the spatiotemporal input stream of the image frames from the received stream of the image frames, process400may involve processor310performing either an operation represented by sub-block426or another operation represented by sub-block428. Process400may proceed from422to424.

At424, process400may involve processor310generating the appearance-based stream of the image frames from the received stream of the image frames.

At426, process400may involve processor310generating the spatiotemporal input stream of the image frames using optical flow calculations and spatiotemporal filters.

At428, process400may involve processor310generating the spatiotemporal input stream of the image frames using a network stream which is adapted to extract motion information from the input stream of the image frames.

In some embodiments, in receiving the input stream of the image frames, process400may involve processor310receiving the input stream of the image frames from one or more image sensors on a vehicle (e.g., from one or more image sensors340(1)-340(N)).

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310augmenting, using the DNN, the appearance-based stream of the image frames with the spatiotemporal input stream of the image frames to generate the augmented stream of the image frames.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using a CNN that concatenates the spatiotemporal input stream and the appearance-based stream.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310concatenating the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with an early-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are separately stacked, without individual processing, as inputs for object detection and trajectory prediction.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310concatenating the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a late-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed in two separate streams to create two separate sets of feature maps that are combined to form an input for object detection and trajectory prediction.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310concatenating the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN with a slow-fusion architecture in which the spatiotemporal input stream and the appearance-based stream are processed separately for one or more layers before being combined to form an input for object detection and trajectory prediction.

In some embodiments, in fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using the DNN, process400may involve processor310fusing the spatiotemporal input stream of the image frames and the appearance-based stream of the image frames using RNNs with recurrent connections to utilize outputs from a previous video frame of the image frames as inputs to a current video frame of the image frames.

At least some embodiments of the present disclosure have been directed to computer program products comprising such logic (e.g., in the form of software) stored on any computer useable medium. Such software, when executed in one or more data processing devices, causes a device to operate as described herein.