Systems and methods for providing future object localization

In one embodiment, a system includes one or more vehicle sensors for capturing host data and a processor having modules. The data receiving module identifies one or more proximate vehicles within the environment based on one or more of the host data and proximate data received from the one or more proximate vehicles. The motion prediction module generates a first joint uncertainty distribution based on an initial joint uncertainty model and a host model distribution. The motion prediction module also samples host kinematic predictions based on the first joint uncertainty distribution and the host data. The object localization module generates a second joint uncertainty distribution based on the initial joint uncertainty model and an object prediction model distribution. The object localization module also samples proximate kinematic predictions based on the second joint uncertainty distribution and the proximate data.

BACKGROUND

Predicting the future motion of agents in dynamic environments is one of the important tasks in vehicle control and vehicle navigation. Typically, a distribution of all possible paths is modeled to tackle the multi-modality of future forecast. However, such predict distribution is either naively learned in a data driven manner with no consideration of the uncertainty or simply generated to same different types of motion using deep generative models. Additionally, single-modal future forecasting has been performed, however the uncertainty of such models is restricted to be epistemic and overlooks noise inherent in the dataset, which is infeasible to recover from a small number of the observations. Also, such models are not practical to deploy in autonomous driving (AD) and advanced driving assistance systems (ADAS).

BRIEF DESCRIPTION

According to one aspect, a system for future object localization is described. The system includes one or more vehicle sensors for capturing host data and a processor. The processor includes a data receiving module, a motion prediction module, and an object localization module. The data receiving module identifies one or more proximate vehicles within the environment based on one or more of the host data and proximate data received from the one or more proximate vehicles. The motion prediction module generates a first joint uncertainty distribution based on an initial joint uncertainty model and a host model distribution. The motion prediction module also samples host kinematic predictions based on the first joint uncertainty distribution and the host data. The object localization module generates a second joint uncertainty distribution based on the initial joint uncertainty model and an object prediction model distribution. The object localization module also samples proximate kinematic predictions based on the second joint uncertainty distribution and the proximate data.

According to another aspect, a computer implemented method for future object localization is provided. The method includes receiving host data from a host vehicle and proximate data from one or more proximate vehicles. The host data includes a first series of image frames of an environment from the host vehicle. The method also includes generating a first joint uncertainty distribution based on an initial joint uncertainty model and a host model distribution. The method further includes sampling host kinematic predictions based on the first joint uncertainty distribution and the host data. The method yet further includes generating a second joint uncertainty distribution based on the initial joint uncertainty model and an object prediction model distribution. The method includes sampling proximate kinematic predictions based on the second joint uncertainty distribution and the proximate data. The method further includes displaying predicted trajectories of the one or more proximate vehicles based on the host kinematic predictions and the proximate kinematic predictions.

According to still another aspect, a non-transitory computer readable storage medium storing instructions that when executed by a computer, which includes a processor perform a method associated with future object localization. The method includes receiving host data from a host vehicle and proximate data from one or more proximate vehicles. The host data includes a first series of image frames of an environment from the host vehicle. The method also includes generating a first joint uncertainty distribution based on an initial joint uncertainty model and a host model distribution. The method further includes sampling host kinematic predictions based on the first joint uncertainty distribution and the host data. The method yet further includes generating a second joint uncertainty distribution based on the initial joint uncertainty model and an object prediction model distribution. The method includes sampling proximate kinematic predictions based on the second joint uncertainty distribution and the proximate data. The method further includes displaying predicted trajectories of the one or more proximate vehicles based on the host kinematic predictions and the proximate kinematic predictions.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting.

A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus may transfer data between the computer components. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus can also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others.

“Computer communication”, as used herein, refers to a communication between two or more computing devices (e.g., computer, personal digital assistant, cellular telephone, network device) and can be, for example, a network transfer, a file transfer, an applet transfer, an email, a hypertext transfer protocol (HTTP) transfer, and so on. A computer communication can occur across, for example, a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a circuit switching system, a packet switching system, among others. Computer communication may also include an ad hoc network, a mobile ad hoc network, a vehicular ad hoc network (VANET), a vehicle-to-vehicle (V2V) network, a vehicle-to-everything (V2X) network, a vehicle-to-infrastructure (V2I) network, among others. Computer communication can utilize any type of wired, wireless, or network communication protocol including, but not limited to, Ethernet (e.g., IEEE 802.3), WiFi (e.g., IEEE 802.11), communications access for land mobiles (CALM), WiMax, Bluetooth, Zigbee, ultra-wideband (UWAB), multiple-input and multiple-output (MIMO), telecommunications and/or cellular network communication (e.g., SMS, MMS, 3G, 4G, LTE, 5G, GSM, CDMA, WAVE), satellite, dedicated short range communication (DSRC), among others.

A “disk”, as used herein can be, for example, a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk can be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD ROM). The disk can store an operating system that controls or allocates resources of a computing device.

A “memory”, as used herein can include volatile memory and/or non-volatile memory. Non-volatile memory can include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory can include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM). The memory can store an operating system that controls or allocates resources of a computing device.

A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system. A module may also include logic, a software controlled microprocessor, a discrete logic circuit, an analog circuit, a digital circuit, a programmed logic device, a memory device containing executing instructions, logic gates, a combination of gates, and/or other circuit components. Multiple modules may be combined into one module and single modules may be distributed among multiple modules.

“Object”, as used herein, refers to any items in the roadway and may include proximate vehicles, pedestrians crossing the roadway, other vehicles, obstacles, animals, debris, potholes, etc. Further, an ‘object may include most any traffic conditions, road conditions, weather conditions, etc. Examples of objects may include, but are not necessarily limited to other vehicles (e.g., proximate vehicle), buildings, landmarks, obstructions in the roadway, road segments, intersections, etc. Thus, objects may be found, detected, or associated with a path, one or more road segments, etc. along a route on which a host vehicle is travelling or is projected to travel along.

An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a wireless interface, a physical interface, a data interface and/or an electrical interface.

A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that may be received, transmitted and/or detected. Generally, the processor may be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor may include various modules to execute various functions.

A “value” and “level”, as used herein may include, but is not limited to, a numerical or other kind of value or level such as a percentage, a non-numerical value, a discrete state, a discrete value, a continuous value, among others. The term “value of X” or “level of X” as used throughout this detailed description and in the claims refers to any numerical or other kind of value for distinguishing between two or more states of X. For example, in some cases, the value or level of X may be given as a percentage between 0% and 100%. In other cases, the value or level of X could be a value in the range between 1 and 10. In still other cases, the value or level of X may not be a numerical value, but could be associated with a given discrete state, such as “not X”, “slightly x”, “x”, “very x” and “extremely x”.

A “vehicle”, as used herein, refers to any moving vehicle that is capable of carrying one or more human occupants and is powered by any form of energy. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, go-karts, amusement ride cars, rail transport, personal watercraft, drones, and aircraft. In some cases, a motor vehicle includes one or more engines. Further, the term “vehicle” may refer to an electric vehicle (EV) that is capable of carrying one or more human occupants and is powered entirely or partially by one or more electric motors powered by an electric battery. The EV may include battery electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). The term “vehicle” may also refer to an autonomous vehicle and/or self-driving vehicle powered by any form of energy. The autonomous vehicle may or may not carry one or more human occupants. Further, the term “vehicle” may include vehicles that are automated or non-automated with pre-determined paths or free-moving vehicles.

A “vehicle system,” as used herein can include, but is not limited to, any automatic or manual systems that can be used to enhance the vehicle, driving, and/or safety. Exemplary vehicle systems include, but are not limited to: an electronic stability control system, an anti-lock brake system, a brake assist system, an automatic brake prefill system, a low speed follow system, a cruise control system, a collision warning system, a collision mitigation braking system, an auto cruise control system, a lane departure warning system, a blind spot indicator system, a lane keep assist system, a navigation system, an electronic power steering system, visual devices (e.g., camera systems, proximity sensor systems), a climate control system, an electronic pretensioning system, a monitoring system, a passenger detection system, a vehicle suspension system, a vehicle seat configuration system, a vehicle cabin lighting system, an audio system, a sensory system, among others.

I. SYSTEM OVERVIEW

Referring now to the drawings, wherein the showings are for purposes of illustrating one or more exemplary embodiments and not for purposes of limiting same,FIG.1is a schematic view of the operating environment100for implementing systems and methods for future object localization according to an exemplary embodiment of the present disclosure. The components of the operating environment100, as well as the components of other systems, hardware architectures, and software architectures discussed herein, may be combined, omitted, or organized into different architectures for various embodiments.

Generally, the operating environment100includes a host vehicle102with an electronic control unit (ECU)104that executes one or more applications, operating systems, vehicle system and subsystem user interfaces, among others. The ECU104may also execute a future localization application106that may be configured to complete future localization of one or more objects, such as proximate vehicles and/or pedestrians. For example, turning toFIG.2, the vehicle environment200includes proximate vehicles202,204,206,208, and210. While the embodiments described are vehicle examples for clarity, shown as vehicles, the systems and methods for future object localization may also be applied to other embodiments, such as robotics, path planning, and swarm management, among others.

The future localization may include a prediction of future locations, positions, scales, depths, motions, and trajectories of one or more of the proximate vehicles202-210located within the surrounding environment200of the host vehicle102. The surrounding environment may include an area including a vicinity of the host vehicle102. For example, the surrounding environment of the host vehicle102may include any number of traffic configurations, for example, an intersection at which the host vehicle102is located (e.g., stopped) and/or a roadway on which the host vehicle102is driven (e.g., lane of a highway).

The future localization application106may communicate with and utilize a neural network108to encode temporal information to provide location and scale information pertaining to the host vehicle102and the one or more proximate vehicles located within the surrounding environment200of the host vehicle102. For example, the host positions, host motions, location, and/or scale information pertaining to the host vehicle102is host data and the proximate positions, proximate motions, location and/or scale information pertaining to one or more of the proximate vehicles202-210is proximate data. Accordingly, the future localization application106may utilize the neural network108to encode the host data and the proximate data. Suppose that the host data includes a dense optical flow to provide pixel-level information about proximate vehicle motion, scale change, and appearance.

The future localization application106may further utilize the neural network108to decode the encoded host data and proximate data to generate a localization dataset110, including a joint uncertainty model. The future localization application106may also combine the encoded host data and proximate data to thereby output a change in a predicted future location of a predicted future bounding box associated with one or more of the proximate vehicles202-210within the surrounding environment200of the host vehicle102.

In some embodiments, the future localization application106may also input data provided by a vehicle controller112of the host vehicle102that pertains to motion of the host vehicle102that may also be provided to the neural network108to decode and output information pertaining to predicted bounding boxes. The future localization application106may also process information from various sources provided as inputs and may utilize the neural network108to provide various functions, that may include, but is not limited to object classification, feature recognition, multilayer perceptions, and autonomous driving commands.

The neural network108may be configured as a multi-stream Recurrent Neural Network (RNN) and may include an encoder-decoder structure that includes a plurality of fully connected layers. Alternatively, the neural network108may be configured as a convolutional neural network (CNN). The neural network108may utilize machine learning/deep learning to provide artificial intelligence capabilities. For example, the neural network108may utilize machine learning/deep learning to encode the temporal information and optical flow information from past observations and decode predicted future bounding boxes based on images of the surrounding environment200of the host vehicle102.

The neural network108may be configured to process the future vehicle localization information from the decoded predicted future bounding boxes and may build and maintain the localization dataset110that may be collected for one or more roadway environmental scenarios (e.g., intersection scenarios). In some embodiments, the future localization application106may access and analyze the localization dataset110to provide motion planning capabilities while executing autonomous driving commands that may be provided to autonomously control the host vehicle102to preemptively adapt to predicted future locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment200and/or the host vehicle102. In additional embodiments, the future localization application106may access and analyze the localization dataset110to provide warnings to a driver of the host vehicle102that may be provided to alert the driver for preemptive collision avoidance purposes based on the predicted future locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment200and/or the host vehicle102.

With continued reference toFIG.1, in addition to the ECU104and the vehicle controller112, the host vehicle102may include a plurality of components, for example, a memory114, a vehicle camera system116that is operably connected to one or more cameras118, and a plurality of vehicle systems120. In an exemplary embodiment, the ECU104may be configured to operably control the plurality of components of the host vehicle102.

In one or more embodiments, the ECU104may include a microprocessor, one or more application-specific integrated circuit(s) (ASIC), or other similar devices. The ECU104may also include internal processing memory, an interface circuit, and bus lines for transferring data, sending commands, and communicating with the plurality of components of the host vehicle102. The ECU104may also include a communication device (not shown) for sending data internally in the host vehicle102and communicating with externally hosted computing systems (e.g., external to the host vehicle102). Generally, the ECU104communicates with the memory114to execute the one or more applications, operating systems, vehicle system and subsystem user interfaces, and the like that are stored within the memory114.

In one embodiment, the ECU104may operably control the vehicle controller112to process and execute an autonomous driving plan based on one or more of an intended destination of the host vehicle102, one or more proximate vehicles located within the surrounding environment of the host vehicle102, one or more future predicted locations of one or more of the proximate vehicles as determined by the future localization application106, and/or one or more external factors that may include, but may not be limited to, a lane in which the host vehicle102is traveling, status of traffic signals, traffic patterns, traffic regulations, etc. As discussed below, in some embodiments, the future localization application106may predict a future motion of the host vehicle102based on the autonomous driving plan processed by the vehicle controller112.

In one embodiment, the vehicle controller112may additionally provide one or more commands to one or more of the vehicle systems120and/or one or more control units (not shown) of the host vehicle102, including, but not limited to an engine control unit, a braking control unit, a transmission control unit, a steering control unit, and the like to control the host vehicle102to be autonomously driven based on the autonomous driving plan and/or data communicated by the future localization application106to autonomously or semi-autonomously control the host vehicle102. In other words, the host vehicle102may be autonomously driven based on one or more factors that may influence the autonomous driving plan (e.g., lane in which the host vehicle102is traveling, status of traffic signals, traffic patterns, traffic regulations, etc.) and/or to preemptively adapt to predicted locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment200based on encoding and decoding final hidden states output by the neural network108. In another embodiment, suppose that the host vehicle is a robot, the robot may traverse an environment based on the one or more factors that influence the path plan to adapt to one or more objects in the surrounding environment200of the robot.

The memory114may be configured to store one or more executable files associated with one or more operating systems, applications, associated operating system data, application data, vehicle system and subsystem user interface data, and the like that are executed by the ECU104. In one or more embodiments, the memory114of the host vehicle102may be accessed by the future localization application106to store data, for example, a first series of image frames of the surrounding environment200from the one or more cameras118. In some embodiments, the memory114may include one or more traffic participant models, as will be discussed below, associated with one or more types of proximate vehicles that represent values that include a range of sizes and features (based on image data) that are associated to different types of proximate vehicles.

In an exemplary embodiment, the memory114may include components of the neural network108. As discussed above, the neural network108may be configured as a RNN that is configured to process computer/machine based/deep learning that may be centered on one or more forms of data that are provided to the neural network108. In addition to being hosted on the memory114, in some embodiments, the neural network108, subsets of the neural network108, and/or subsets of data may be used by the neural network108may be hosted on an externally hosted server infrastructure (not shown) that may be configured to communicate with the ECU104of the host vehicle102through the communication device of the ECU104.

In one or more embodiments, the neural network108may include a neural network processing unit122that may provide processing capabilities to be configured to utilize machine learning/deep learning to provide artificial intelligence capabilities that may be utilized to output data to the future localization application106and to build and maintain the localization dataset110. The neural network processing unit122may process information that is provided as inputs and may utilize the localization dataset110to access stored future localization data to provide various functions, that may include, but may not be limited to, object classification, feature recognition, computer vision, speed recognition, machine translation, autonomous driving commands, and the like.

In an exemplary embodiment, the neural network108may be configured as a RNN encoder-decoder structure that is operably controlled by the neural network processing unit122and includes a host data encoder124, a proximate data encoder126, and a future localization decoder128. The host data encoder124and proximate data encoder126may be configured as gated recurrent unit encoders. In an exemplary embodiment, the host data encoder124may be configured to encode the host data. The host data may include past observations captured within one or more images with respect to the past bounding box trajectory of the one or more proximate vehicles202-210located within the surrounding environment200. The host data encoder124may thereby provide location and scale information pertaining to both the host vehicle102and one or more proximate vehicles202-210located within the surrounding environment of the host vehicle102.

The proximate data encoder126may be configured to encode dense optical flow information of motion, scale, and/or appearance change of one or more proximate vehicles and background captured within one or more images. As discussed below, the future localization application106may be configured to fuse the data output by the host data encoder124and the proximate data encoder126to provide the encoded past location, scale, and corresponding optical flow fields of each of the one or more proximate vehicles located within the surrounding environment of the host vehicle102. The fused data may be communicated to the future localization decoder128to extrapolate future bounding box trajectories of each of the one or more proximate vehicles located within the surrounding environment200.

More specifically, the future localization decoder128may be configured to generate a joint uncertainty model as will be described in greater detail with respect to method300ofFIG.3and method400ofFIG.4. Likewise, the future localization decoder128may be configured to decode future bounding boxes based on the one or more images of the surrounding environment200. The proximate data encoder126to decode future bounding boxes to thereby allow the future localization application106to output predicted locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment of the host vehicle102to be updated to the localization dataset110.

FIG.5is an exemplary diagram of an architecture500associated with training the system for future object localization ofFIG.1. The Future ego-motion prediction stream models the uncertainty of future ego-behavior. The Future object localization stream encodes past bounding box and flow information to predict future motion of a proximate vehicle conditioned on the sampled future ego-motion of the host vehicle102. The resulting distribution is multi-modal and uncertainty aware.

As will be discussed more detail below, based on one or more commands provided by the future localization application106to the neural network processing unit122of the neural network108, one or more past bounding box trajectories of one or more proximate vehicles based on one or more images may be encoded by the host data encoder124to provide location and scale information. Therefore, the future localization application106utilize the neural network processing unit122and/or the future localization application106may include a processor.

The host data include a dense optical flow of a first set of images associated with the one or more proximate vehicles202-210located within the surrounding environment200of the host vehicle102to provide pixel level information of the motion, scale, and/or appearance change of each of the one or more proximate vehicles. In some embodiments, the future localization application106may additionally provide one or more commands associated with the ego-motion planning of the host vehicle102to the future localization decoder128. The future localization decoder128may be configured to decode and predict future bounding boxes216associated with the one or more proximate vehicles located within the surrounding environment of the host vehicle102. As discussed below, the future localization application106may thereby predict future locations, positions, scales, depths, and trajectories associated to each of the one or more proximate vehicles located within the surrounding environment of the host vehicle102.

In some embodiments, the future predicted locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment of the host vehicle102may be provided as vehicle localization data that is added to the localization dataset110. In some configurations, the future localization application106may utilize the localization dataset110to provide motion planning capabilities while executing autonomous driving commands that may be provided to autonomously control the host vehicle102to preemptively adapt to the predicted locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment of the host vehicle102.

Referring again toFIG.1, the vehicle camera system116may include one or more cameras118that are positioned at one or more areas of the surrounding environment of the host vehicle102. In an exemplary embodiment, the surrounding environment of the host vehicle102may be defined as a predetermined area located in around (front/sides/behind) the host vehicle102(e.g., road environment in front, sides, and/or behind of the host vehicle102) that may be included within the vehicle's travel path. The one or more cameras118of the vehicle camera system116may be disposed at external front portions of the host vehicle102, including, but not limited to different portions of the vehicle dashboard, vehicle bumper, vehicle front lighting units, vehicle fenders, and the windshield. In one embodiment, the one or more cameras may be configured as RGB cameras that may capture RGB bands that are configured to capture rich information about object appearance, as well as relationships and interactions between the host vehicle102and objects within the surrounding environment of the host vehicle102which may include one or more proximate vehicles. The one or more cameras118may be configured to capture the images (e.g., images, videos) of the surrounding environment of the host vehicle102in one or more resolutions. For example, the one or more cameras118may be configured to capture video clips of a front facing surrounding environment of the host vehicle102with 1920×1200 resolutions and 10 fps.

As discussed, image data may be provided by the vehicle camera system116to one or more components of the future localization application106to be further evaluated and processed based on utilization of the neural network108. In some embodiments, the future localization application106may be configured to execute the image logic to perform feature extraction extract on the image(s). The image extraction may include the extraction of one or more spatial-temporal features and may pertain to object and scene recognition. In one embodiment, the image logic may also be utilized to determine one or more sets of image coordinates associated with one or more objects that may include, but may not be limited to, proximate vehicles (e.g., pedestrians, bikers, other vehicles), roadway attributes (e.g., lane markings, off-ramps, curbs), and road side objects (e.g., traffic light, stop sign).

In one or more embodiments, the vehicle systems120may include one or more systems that may be utilized to autonomously control the host vehicle102and/or one or more functions and features of the host vehicle102. For example, the vehicle systems120may include, but are not limited to, any automatic or manual systems that may be used to enhance the vehicle driving. It is understood that the vehicle systems120may include various vehicle sensors (not shown) that sense and measure different stimuli (e.g., a signal, a property, a measurement, a quantity) associated with the host vehicle102and/or a particular vehicle system120.

For example, some vehicle sensors may include radar and laser sensors mounted to the exterior of the host vehicle102. The sensors may be any type of sensor, for example, acoustic, electric, environmental, optical, imaging, light, pressure, force, thermal, temperature, proximity, among others. In some embodiments, one or more of the vehicle systems120may include vehicle sensors for detecting objects surrounding the host vehicle102. For example, proximity sensors, radar sensors, laser sensors, LIDAR sensors, and other optical sensors, may be used to detect objects, such as proximate vehicles, within the surrounding environment of the host vehicle102.

The vehicle systems120may include Advanced Driver Assistance Systems (ADAS), for example, an adaptive cruise control system, a blind spot monitoring system, a collision mitigation system, a lane departure warning system, among others that may be utilized to provide warnings/alerts to the driver of the host vehicle102(e.g., if the host vehicle102is being driven by a driver and not autonomously) for preemptive collision avoidance purposes based on the predicted locations, positions, scales, depths, and trajectories of one or more proximate vehicles within the surrounding environment of the host vehicle102and/or the host vehicle102.

II. FUTURE OBJECT LOCALIZATION APPLICATION AND RELATED METHODS

Referring now toFIG.3, a method300for generating a joint uncertainty model associated with future object localization according to an exemplary embodiment.FIG.3will also be described with reference toFIGS.1,2, and5. As shown inFIG.3, the method for generating a joint uncertainty model can be described by three stages, namely, aleatoric modeling, epistemic modeling, and joint modeling. For simplicity, the method300will be described by these stages, but it is understood that the elements of the method300can be organized into different architectures, blocks, stages, and/or processes.

At block302, the host data is received. The host data may be received from any number of vehicle sensors and/or vehicle systems as described above. For example, the host data may be received from the one or more cameras118. The host data, as well as the proximate data, may include model as well as past observations and historical data associated with the host vehicle102and the proximate vehicles202-210.

Aleatoric uncertainty comes from inherent noise in the observations due to the probabilistic variability. To model this type of uncertainty during training, the neural network108incorporates noise parameters (μt, Σyt) at time t, where denotes the p denotes the mean and Σytdenotes the co-variance matrix from the ground-truth label yt. Accordingly, at block304, the method300includes determining a co-variance matrix based on the host data to create a prediction distribution. The co-variance matrix Σytis learned using a negative log-likelihood loss function as follows:

Thus, (μt, Σyt) can be predicted at T observed time-steps from time Tobs+1 to Tpred. Accordingly, Eq. 1 can be used to compute how likely the observations come from the posterior distribution(μt, Σyt). To avoid zeros in the denominator Eq. 1 may be rewritten as:

Epistemic uncertainty is caused by the model's weight parameters that are inadequately measured from the observations. To reduce this type of uncertainty during training, the neural network108may use additional measurements, employ dropout deep learning to avoid overfitting. Therefore, at block306, the method300includes calculating a model distribution q(ω) based on the host data. In one embodiment, the dataset X, Y the posterior over weights P(ω|X, Y) is approximated using distribution q(ω).

At block308, the method300includes generating multiple sample outputs using the distribution q(ω). For example, during inference, the neural network processing unit122of the neural network108may generate N samples from the distribution q(ω) of the network's learned weight parameters w using dropout.

At block310, the method300includes calculating the variance based on the sampled outputs. For example, N number of noisy outputs are used to compute the variance Σy between the predicted outputs fi(x) and ground-truth labels ytat each time-step t as shown with respect to Eq. 2

Rather than merely considering the future possibilities of the host vehicle102, the proximate vehicles202-210are additionally considered. Accordingly, multiple possible futures of the proximate vehicles202-210is determined and the uncertainty of each of the possible futures.

At block312, the method300includes combining the variances from aleatoric and epistemic models as a joint uncertainty model and update the prediction distribution. The neural network108updates the noise parameters (μy, Σy) by adding the aleatoric uncertainty given in Eq. 2 to epistemic uncertainty in Eq. 3. The total variance and mean can then be computed according to:

As a result, the neural network108outputs the noise parameters for the data posterior distribution(μt, Σyt) with the learned distribution of the distribution q(ω) during interference. Different node connectionsiare sampled for N times using dropout, and corresponding aleatoric and epistemic uncertainty according to Eq. 4.

Referring now toFIG.4, a method400for a host vehicle stage and a proximate vehicle stage associated with future object localization according to an exemplary embodiment.FIG.4will also be described with reference toFIGS.1-3and5. Generally, the host vehicle stage corresponds to the host architecture502ofFIG.5and the proximate vehicle stage corresponds to the proximate architecture504. For simplicity, the method400will be described by these stages, but it is understood that the elements of the method400can be organized into different architectures, blocks, stages, and/or processes.

At block402, the method400includes receiving the host data. The host data is associated with the movement of the host vehicle102, referred to as ego motion. The host data is received by the data receiving module130. As described above, the host data may be received from any number of vehicle sensors and/or vehicle systems as described above. For example, the host data may be received from the one or more cameras118. The host data, as well as the proximate data, may include model as well as past observations and historical data associated with the host vehicle102and the proximate vehicles202-210.

At block404, the method400includes encoding the host data. For example, the data receiving module130may receive the host data and encode the host data according to the past ego motion encoder506shown inFIG.5. In one embodiment, the past ego motion encoder506may be configured as gated recurrent unit (GRU) encoders. In one embodiment, multi-layer perception (MLP) may be used to convert the host data to the embedding of the GRU. The prediction output of the FRU may be a 5-dimensional vector at a future step in time from the observed time Tobs+1 to Tpred. In another exemplary embodiment, the host data encoder124may be configured to encode temporal information of past observations based on a first series of image frames of the environment200. Accordingly, the past motion of the host vehicle102, past ego motion, associated with the host data, is encoded.

At block406, the method400includes decoding the encoded host data to generate a first joint uncertainty distribution based on an host model distribution. To generate future motion through a future ego motion decoder508, shown inFIG.5, drawn from the weightsEthe weight distribution q(E). The motion prediction module132then predicts future host vehicle positions and motions. For example, the motion prediction module132, by virtue of the future ego motion encoder510generates multiple modes of prediction over the uncertainty distribution P(E|xE,E) over velocity, v, and a yaw rate θ, where E={v, θ} is the future ego-motion and xEis past ego motion. The motion prediction module132generates the first joint uncertainty distribution based on an initial joint uncertainty model and the host data. Furthermore, the future ego motion encoder510

At block408, the method400includes sampling host kinematic predictions based on the first joint uncertainty distribution and the host data. The host kinematic predictions predict future motions of host vehicle102.

At block410, the method400includes receiving sensor data from around the host vehicle. At block412, proximate data is received based on sensor data. The proximate data may include past observations and historical data associated with the the proximate vehicles202-210. The proximate data is received by the data receiving module130. In some embodiments, the proximate data may be received from the proximate vehicles. The proximate data may also be calculated based on the information received from the one or more cameras118. For example, the proximate data may be calculated based on the first series of images. The motion, position, speed, velocity, and trajectory, among others of one or more of the proximate vehicles202-210may be determined based on the first series of image frames of an environment200from the host vehicle102.

Additionally or alternatively, the proximate data may be received via computer communication including V2V communication. Accordingly, the vehicle sensors of the host vehicle102, including, for example, a transceiver (not shown) may include information about objects in the environment200at block410. In response to receiving that information, the data receiving module130may determine that the information is proximate data.

At block414, the method400includes encoding the proximate data. For example, the data receiving module130may be received and encoded according to the past ego motion encoder506shown inFIG.5. In one embodiment, the past flow encoder512may be configured as gated recurrent unit (GRU) encoders. In a similar manner as described above, multi-layer perception (MLP) may be used to convert the host data to the embedding of the GRU. Additionally or alternatively, the proximate data may be encoded using a bounding box encoder.

At block416the method400includes decoding the encoded data to a uncertainty distribution based on the proximate data and the future object prediction. Accordingly, a second joint uncertainty distribution is generated based on the initial joint uncertainty model. In particular, to model the future behavior of the proximate vehicles202-210respective to the noisy ego-motion, the object localization module decoder134uses the output of future ego-motion prediction from the motion prediction module132as a prior. In this way, a bounding box decoder514reacts to each modality of the host vehicle102while predicting future motion of the proximate vehicles202-210. Similar to joint uncertainty modeling of the future ego motion decoder508, weights wBfor the bounding box decoder514are drawn from the weight distribution q(wB). The object localization module decoder134estimates the noise parameters for the center (cx, cy) and the dimension (w, h) of the bounding box using the weights wB.

At block418the method400includes sampling proximate kinematic predictions based on the second joint uncertainty distribution and the host data. The proximate kinematic predictions predict future motions of proximate vehicles202-210. Accordingly, the path of the host vehicle102may be predicted as host kinematic predictions based on the first joint uncertainty distribution and the path of the proximate vehicles202-210may be predicted as proximate kinematic predictions based on the second joint uncertainty distribution. For example, a bounding box prediction may be given by B={cx, cy, w, h} by sampling from the uncertainty distribution P (B|G, wB, Ê,), and G=Φ(xF) and concatenation of past flow xFand bounding box xBencoding.

Additionally or alternatively, the sampling may include displaying predicted trajectories of the one or more proximate vehicles based on the host kinematic predictions and the proximate kinematic predictions. Suppose that the host data includes the first series of image frames of the environment200. The object localization module decoder134is further configured to generate a second series of image frames of the environment200including predicted trajectories of the one or more proximate vehicles based on the host kinematic predictions and the proximate kinematic predictions. For example, the bounding boxes and trajectories may be displayed for a user and/or vehicle occupant. Returning toFIG.2, for the proximate vehicle202, the bounding boxes212and/or the predicted trajectories214and216may be shown.

The systems and method described herein may use the pre-trained ego-motion prediction module with epistemic uncertainty and is trained jointly with future object localization. The use of these uncertainty models to condition the motion forecast of proximate vehicles202-210significantly improves the overall performance. This comparison validates the use of the uncertainty to model more robust interactions of proximate vehicles202-210with the host vehicle102. The data receiving module130, the motion prediction module132, and the object localization module decoder134are trained with aleatoric uncertainty and epistemic uncertainty as described with respect toFIG.3. These baseline models further decrease the error rate compared aleatoric uncertainty and epistemic uncertainty alone so that it better predicts future motion of the proximate vehicles202-210as well as the bounding box locations and scales associated with the sampled host kinematic predictions and the sampled proximate kinematic predictions.

Still another aspect involves a computer-readable medium including processor-executable instructions configured to implement one aspect of the techniques presented herein. An aspect of a computer-readable medium or a computer-readable device devised in these ways is illustrated inFIG.6, wherein an implementation600includes a computer-readable medium608, such as a CD-R, DVD-R, flash drive, a platter of a hard disk drive, etc., on which is encoded computer-readable data606. This encoded computer-readable data606, such as binary data including a plurality of zero's and one's as shown in606, in turn includes a set of processor-executable computer instructions604configured to operate according to one or more of the principles set forth herein. In this implementation600, the processor-executable computer instructions604may be configured to perform a method602, such as the method300ofFIG.3. In another aspect, the processor-executable computer instructions604may be configured to implement a system, such as the operating environment100ofFIG.1. Many such computer-readable media may be devised by those of ordinary skill in the art that are configured to operate in accordance with the techniques presented herein.

Generally, aspects are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media as will be discussed below. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform one or more tasks or implement one or more abstract data types. Typically, the functionality of the computer readable instructions are combined or distributed as desired in various environments.

FIG.7illustrates a system700including a computing device712configured to implement one aspect provided herein. In one configuration, the computing device712includes at least one processing unit716and memory718. Depending on the exact configuration and type of computing device, memory718may be volatile, such as RAM, non-volatile, such as ROM, flash memory, etc., or a combination of the two. This configuration is illustrated inFIG.7by dashed line714.

In other aspects, the computing device712includes additional features or functionality. For example, the computing device712may include additional storage such as removable storage or non-removable storage, including, but not limited to, magnetic storage, optical storage, etc. Such additional storage is illustrated inFIG.7by storage720. In one aspect, computer readable instructions to implement one aspect provided herein are in storage720. Storage720may store other computer readable instructions to implement an operating system, an application program, etc. Computer readable instructions may be loaded in memory718for execution by the processing unit716, for example.

The computing device712includes input device(s)724such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, or any other input device. Output device(s)722such as one or more displays, speakers, printers, or any other output device may be included with the computing device712. Input device(s)724and output device(s)722may be connected to the computing device712via a wired connection, wireless connection, or any combination thereof. In one aspect, an input device or an output device from another computing device may be used as input device(s)724or output device(s)722for the computing device712. The computing device712may include communication connection(s)726to facilitate communications with one or more other devices730, such as through network728, for example.

Various operations of aspects are provided herein. The order in which one or more or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated based on this description. Further, not all operations may necessarily be present in each aspect provided herein.