Vehicle taillight recognition based on a velocity estimation

A method for controlling an ego vehicle in an environment includes associating, by a velocity model, one or more objects within the environment with a respective velocity instance label. The method also includes selectively, by a recurrent network of the taillight recognition system, focusing on a selected region of the sequence of images according to a spatial attention model for a vehicle taillight recognition task. The method further includes concatenating the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The method still further planning a trajectory of the ego vehicle based on inferring, at a classifier of the taillight recognition system, an intent of each object of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label.

BACKGROUND

Field

Certain aspects of the present disclosure generally relate to vehicle taillight recognition and, more particularly, to a system and method for using a velocity estimation for recognizing a vehicle taillight.

Background

Autonomous agents (e.g., vehicles, robots, etc.) rely on machine vision for sensing a surrounding environment by analyzing areas of interest in images of the surrounding environment. Although scientists have spent decades studying the human visual system, a solution for realizing equivalent machine vision remains elusive. Realizing equivalent machine vision is a goal for enabling truly autonomous agents. Machine vision is distinct from the field of digital image processing because of the desire to recover a three-dimensional (3D) structure of the world from images and use the 3D structure for fully understanding a scene. That is, machine vision strives to provide a high-level understanding of a surrounding environment, as performed by the human visual system.

In operation, autonomous agents may rely on a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding scene of the autonomous agent. For example, a CNN may be trained to identify and track objects captured by sensors, such as light detection and ranging (LIDAR) sensors, sonar sensors, red-green-blue (RGB) cameras, RGB-depth (RGB-D) cameras, and the like. The sensors may be coupled to, or in communication with, a device, such as an autonomous agent. Object detection applications for autonomous agents may analyze sensor image data for detecting objects in the surrounding scene from the autonomous agent.

Autonomous agents, such as driverless cars and robots, may interact with other non-autonomous vehicles. Therefore, when planning a trajectory for the autonomous agent, an autonomous driving system may predict an intent of other vehicles, such as other non-autonomous and/or other autonomous vehicles. In such examples, vehicle taillight recognition may be used to determine an intent of another vehicle. In some examples, after recognizing a taillight, the autonomous driving system may determine if a brake light or a turn signal is activated. Accordingly, recognizing a vehicle taillight may be used to predict an intent of an autonomous dynamic object (ADO) vehicle and to plan a trajectory of an ego vehicle based on the predicted intent. Various systems are used to recognize vehicle taillights. It may be desirable to further improve an accuracy of such systems.

SUMMARY

In one aspect of the present disclosure, a method for controlling an ego vehicle in an environment includes associating, by a velocity model of a taillight recognition system, one or more objects within the environment with a respective velocity instance label. The method further includes selectively, by a recurrent network of the taillight recognition system, focusing on a selected region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task. The method still further includes concatenating the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The method also includes inferring, at a classifier of the taillight recognition system, an intent of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label. The method further includes planning a trajectory of the ego vehicle based on inferring the intent of the one or more objects.

Another aspect of the present disclosure is directed to an apparatus including means for associating, by a velocity model of a taillight recognition system, one or more objects within the environment with a respective velocity instance label. The apparatus further includes means for selectively, by a recurrent network of the taillight recognition system, focusing on a selected region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task. The apparatus still further includes means for concatenating the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The apparatus also includes means for inferring, at a classifier of the taillight recognition system, an intent of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label. The apparatus further includes means for planning a trajectory of the ego vehicle based on inferring the intent of the one or more objects.

In another aspect of the present disclosure, a non-transitory computer-readable medium with non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code to associate, by a velocity model of a taillight recognition system, one or more objects within the environment with a respective velocity instance label. The program code further includes program code to selectively focus, by a recurrent network of the taillight recognition system, on a selected region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task. The program code still further includes program code to concatenate the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The program code also includes program code to infer, at a classifier of the taillight recognition system, an intent of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label. The program code further includes program code to plan a trajectory of the ego vehicle based on inferring the intent of the one or more objects.

An apparatus comprising a processor, and a memory coupled with the processor and storing instructions operable, when executed by the processor, to cause the apparatus to associate, by a velocity model of a taillight recognition system, one or more objects within the environment with a respective velocity instance label. Execution of the instructions also cause the apparatus to selectively focus, by a recurrent network of the taillight recognition system, on a selected region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task. Execution of the instructions also cause the apparatus to concatenate the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. Execution of the instructions further cause the apparatus to infer, at a classifier of the taillight recognition system, an intent of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label. Execution of the instructions still further cause the apparatus to plan a trajectory of the ego vehicle based on inferring the intent of the one or more objects.

DETAILED DESCRIPTION

Based on the teachings, one skilled in the art should appreciate that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently of or combined with any other aspect of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth. In addition, the scope of the present disclosure is intended to cover such an apparatus or method practiced using other structure, functionality, or structure and functionality in addition to, or other than the various aspects of the present disclosure set forth. It should be understood that any aspect of the present disclosure disclosed may be embodied by one or more elements of a claim.

As discussed, it may be desirable for a vehicle, such as an autonomous vehicle, to recognize vehicle taillights. For example, taillight recognition may allow the vehicle to determine if a brake light or a turn signal is active. In some examples, a collision may be avoided if an ego vehicle determines that a brake light or a turn signal of another vehicle (e.g., an autonomous dynamic object (ADO) vehicle) is active. That is, vehicle taillight recognition may be used for predicting an intent of an ADO vehicle and planning a trajectory of an ego vehicle. Various systems are used to recognize vehicle taillights. It may be desirable to further improve an accuracy of such systems.

Aspects of the present disclosure are directed to an end-to-end framework for vehicle taillight recognition. In some aspects, vehicle velocity is used as a prior for improving taillight recognition. Specifically, the velocity of an ADO vehicle may be used in conjunction with an initial prediction of the ADO vehicle's intent to recognize a taillight of the ADO vehicle. In some examples, the taillight recognition system may use the velocity to focus on moving vehicles, thereby improving the taillight recognition. That is, based on the determined velocity, the taillight recognition system may reduce weights of stationary vehicles and focus more attention on moving vehicles. The velocity may be determined using various methodologies. In some examples, a flow network is used to determine the velocity. The flow network generates a two-dimensional (2D) flow vector for each grid in a cell, where the cell represents a snapshot of an environment.

In some examples, an initial intent of an ADO vehicle may be predicted via a convolutional neural network (CNN)-long short-term memory (CNN-LSTM) framework for vehicle taillight recognition. In such examples, a spatial attention model is integrated with a CNN of the CNN-LSTM framework for training the CNN-LSTM framework to selectively focus on certain regions of the images in a sequence of images. In addition, a temporal attention model may be integrated with an LSTM network of the CNN-LSTM framework to train the LSTM network for frame selection within a selected region of the sequence of images. For example, the spatial attention model may be configured along a temporal dimension (e.g., direction) to focus on portions of the sequence of images important to vehicle taillight recognition. In some examples, the velocity may be used during training to improve the focus on portions of the sequence of images important to vehicle taillight recognition.

Operation of autonomous vehicles and semi-autonomous vehicles may be controlled or adjusted based on predicted actions (e.g., behaviors) of surrounding agents, such as vehicles and pedestrians. For example, a route may be planned for an autonomous vehicle (e.g., an ego vehicle) based on the predicted actions of surrounding agents (e.g., an ado vehicle). In addition, a route may be adjusted to avoid a collision based on the predicted actions of surrounding agents. In the present disclosure, unless otherwise noted, a vehicle refers to an autonomous agent or a semi-autonomous agent.

Conventional vehicles are controlled based on predicted trajectories of surrounding agents. The trajectories of surrounding agents may be predicted using Markov chains, or other probabilistic approaches, to provide a low-level prediction of movements. The trajectories predict where an agent may travel from a current time to a future time. These predicted trajectories may adjust a route of the autonomous agent (e.g., an ego vehicle) to avoid a collision based on the predicted actions of surrounding agents (e.g., an ado vehicle).

Development of perception technologies enable autonomous vehicles to drive well on the road without human involvement. Nevertheless, higher level concepts such as intention prediction, human-machine interaction, and vehicle-to-vehicle communication remain open questions. In particular, intention prediction of ado vehicles is an important feature for autonomous driving safety. Achieving attention prediction of the ado vehicle by an ego vehicle involves understanding gestures from the ado vehicle (e.g., turn and brake signals).

According to aspects of the present disclosure, the velocity of surrounding objects may be used in conjunction with a CNN-LSTM framework that obtains spatial and temporal dimensions via one or more attention models. The use of velocity with spatial attention models and temporal attention models may improve an emphasis on focal regions of the images as well as more important time steps of the image sequence. As a result, the proposed framework outperforms the baseline results by achieving better inference prediction of an ado vehicle's intention(s) with improved performance accuracy.

Various aspects of the present disclosure may be implemented in an agent, such as a vehicle. The vehicle may operate in either an autonomous mode, a semi-autonomous mode, or a manual mode. In some examples, the vehicle may switch between operating modes.FIG.1Ais a diagram illustrating an example of a vehicle100in an environment150, in accordance with various aspects of the present disclosure. In the example ofFIG.1A, the vehicle100may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle. As shown inFIG.1A, the vehicle100may be traveling on a road110. A first vehicle104may be ahead of the vehicle100and a second vehicle116may be adjacent to the ego vehicle100. In this example, the vehicle100may include a 2D camera108, such as a 2D red-green-blue (RGB) camera, and a LiDAR sensor106. Other sensors, such as radar and/or ultrasound, are also contemplated. Additionally, or alternatively, although not shown inFIG.1A, the vehicle100may include one or more additional sensors, such as a camera, a radar sensor, and/or a LiDAR sensor, integrated with the vehicle in one or more locations, such as within one or more storage locations (e.g., a trunk). Additionally, or alternatively, although not shown inFIG.1A, the vehicle100may include one or more force measuring sensors.

In one configuration, the 2D camera108captures a 2D image that includes objects in the 2D camera's108field of view114. The LiDAR sensor106may generate one or more output streams. The first output stream may include a three-dimensional (3D) cloud point of objects in a first field of view, such as a 360° field of view112(e.g., bird's eye view). The second output stream124may include a 3D cloud point of objects in a second field of view, such as a forward facing field of view.

The 2D image captured by the 2D camera includes a 2D image of the first vehicle104, as the first vehicle104is in the 2D camera's108field of view114. As is known to those of skill in the art, a LiDAR sensor106uses laser light to sense the shape, size, and position of objects in an environment. The LiDAR sensor106may vertically and horizontally scan the environment. In the current example, the artificial neural network (e.g., autonomous driving system) of the vehicle100may extract height and/or depth features from the first output stream. In some examples, an autonomous driving system of the vehicle100may also extract height and/or depth features from the second output stream.

The information obtained from the sensors106,108may be used to evaluate a driving environment. In some examples, the information obtained from the sensors106,108may identify whether the vehicle100is at an interaction or a crosswalk. Additionally, or alternatively, the information obtained from the sensors106,108may identify whether one or more dynamic objects, such as pedestrians, are near the vehicle100.

FIG.1Bis a diagram illustrating an example a vehicle100, in accordance with various aspects of the present disclosure. It should be understood that various aspects of the present disclosure may be directed to an autonomous vehicle.

The autonomous vehicle may include be an internal combustion engine (ICE) vehicle, fully electric vehicle (EVs), or another type of vehicle. The vehicle100may include drive force unit165and wheels170. The drive force unit165may include an engine180, motor generators (MGs)182and184, a battery195, an inverter197, a brake pedal186, a brake pedal sensor188, a transmission152, a memory154, an electronic control unit (ECU)156, a shifter158, a speed sensor160, and an accelerometer162.

The engine180primarily drives the wheels170. The engine180can be an ICE that combusts fuel, such as gasoline, ethanol, diesel, biofuel, or other types of fuels which are suitable for combustion. The torque output by the engine180is received by the transmission152. MGs182and184can also output torque to the transmission152. The engine180and MGs182and184may be coupled through a planetary gear (not shown inFIG.1B). The transmission152delivers an applied torque to one or more of the wheels170. The torque output by engine180does not directly translate into the applied torque to the one or more wheels170.

MGs182and184can serve as motors which output torque in a drive mode, and can serve as generators to recharge the battery195in a regeneration mode. The electric power delivered from or to MGs182and184passes through the inverter197to the battery195. The brake pedal sensor188can detect pressure applied to brake pedal186, which may further affect the applied torque to wheels170. The speed sensor160is connected to an output shaft of transmission152to detect a speed input which is converted into a vehicle speed by ECU156. The accelerometer162is connected to the body of vehicle100to detect the actual deceleration of vehicle100, which corresponds to a deceleration torque.

The transmission152may be a transmission suitable for any vehicle. For example, transmission152can be an electronically controlled continuously variable transmission (ECVT), which is coupled to engine180as well as to MGs91and92. Transmission20can deliver torque output from a combination of engine180and MGs91and92. The ECU156controls the transmission152, utilizing data stored in memory154to determine the applied torque delivered to the wheels170. For example, ECU156may determine that at a certain vehicle speed, engine180should provide a fraction of the applied torque to the wheels170while one or both of the MGs182and184provide most of the applied torque. The ECU156and transmission152can control an engine speed (NE) of engine180independently of the vehicle speed (V).

The ECU156may include circuitry to control the above aspects of vehicle operation. Additionally, the ECU156may include, for example, a microcomputer that includes a one or more processing units (e.g., microprocessors), memory storage (e.g., RAM, ROM, etc.), and I/O devices. The ECU156may execute instructions stored in memory to control one or more electrical systems or subsystems in the vehicle. Furthermore, the ECU156can include one or more electronic control units such as, for example, an electronic engine control module, a powertrain control module, a transmission control module, a suspension control module, a body control module, and so on. As a further example, electronic control units may control one or more systems and functions such as doors and door locking, lighting, human-machine interfaces, cruise control, telematics, braking systems (e.g., anti-lock braking system (ABS) or electronic stability control (ESC)), or battery management systems, for example. These various control units can be implemented using two or more separate electronic control units, or a single electronic control unit.

The MGs182and184each may be a permanent magnet type synchronous motor including for example, a rotor with a permanent magnet embedded therein. The MGs182and184may each be driven by an inverter controlled by a control signal from ECU156so as to convert direct current (DC) power from the battery195to alternating current (AC) power, and supply the AC power to the MGs182and184. In some examples, a first MG182may be driven by electric power generated by a second MG184. It should be understood that in embodiments where MGs182and184are DC motors, no inverter is required. The inverter, in conjunction with a converter assembly may also accept power from one or more of the MGs182and184(e.g., during engine charging), convert this power from AC back to DC, and use this power to charge the battery195(hence the name, motor generator). The ECU156may control the inverter, adjust driving current supplied to the first MG182, and adjust the current received from the second MG184during regenerative coasting and braking.

The battery195may be implemented as one or more batteries or other power storage devices including, for example, lead-acid batteries, lithium ion, and nickel batteries, capacitive storage devices, and so on. The battery195may also be charged by one or more of the MGs182and184, such as, for example, by regenerative braking or by coasting during which one or more of the MGs182and184operates as generator. Alternatively (or additionally, the battery195can be charged by the first MG182, for example, when vehicle100is in idle (not moving/not in drive). Further still, the battery195may be charged by a battery charger (not shown) that receives energy from engine180. The battery charger may be switched or otherwise controlled to engage/disengage it with battery195. For example, an alternator or generator may be coupled directly or indirectly to a drive shaft of engine180to generate an electrical current as a result of the operation of engine180. Still other embodiments contemplate the use of one or more additional motor generators to power the rear wheels of the vehicle100(e.g., in vehicles equipped with 4-Wheel Drive), or using two rear motor generators, each powering a rear wheel.

The battery195may also power other electrical or electronic systems in the vehicle100. In some examples, the battery195can include, for example, one or more batteries, capacitive storage units, or other storage reservoirs suitable for storing electrical energy that can be used to power one or both of the MGs182and184. When the battery195is implemented using one or more batteries, the batteries can include, for example, nickel metal hydride batteries, lithium ion batteries, lead acid batteries, nickel cadmium batteries, lithium ion polymer batteries, or other types of batteries.

The vehicle100may operate in one of an autonomous mode, a manual mode, or a semi-autonomous mode. In the manual mode, a human driver manually operates (e.g., controls) the vehicle100. In the autonomous mode, an autonomous control system (e.g., autonomous driving system) operates the vehicle100without human intervention. In the semi-autonomous mode, the human may operate the vehicle100, and the autonomous control system may override or assist the human. For example, the autonomous control system may override the human to prevent a collision or to obey one or more traffic rules.

FIG.1Cillustrates an example implementation of the aforementioned system and method for a convolutional neural network (CNN)-long short-term memory (CNN-LSTM) framework for vehicle taillight recognition using a system-on-a-chip (SOC)130of a vehicle vision system for an autonomous vehicle100. The SOC130may include a single processor or multi-core processors (e.g., a central processing unit (CPU)132), in accordance with certain aspects of the present disclosure. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), delays, frequency bin information, and task information may be stored in a memory block. The memory block may be associated with a neural processing unit (NPU)138, a CPU132, a graphics processing unit (GPU)134, a digital signal processor (DSP)136, a dedicated memory block148, or may be distributed across multiple blocks. Instructions executed at a processor (e.g., CPU132) may be loaded from a program memory associated with the CPU132or may be loaded from the dedicated memory block148.

The SOC130may also include additional processing blocks configured to perform specific functions, such as the GPU134, the DSP136, and a connectivity block143, which may include fourth generation long term evolution (4G LTE) connectivity, unlicensed Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like. In addition, a multimedia processor142in combination with a display128may, for example, classify and categorize poses of objects in an area of interest, according to the display128illustrating a view of the vehicle100. In some aspects, the NPU138may be implemented in the CPU132, DSP136, and/or GPU134. The SOC130may further include a sensor processor144, image signal processors (ISPs)146, and/or navigation120, which may, for instance, include a global positioning system.

The SOC130may be based on an Advanced Risk Machine (ARM) instruction set or the like. In another aspect of the present disclosure, the SOC130may be a server computer in communication with the autonomous vehicle100. In this arrangement, the autonomous vehicle100may include a processor and other features of the SOC130. In this aspect of the present disclosure, instructions loaded into a processor (e.g., CPU132) or the NPU138of the autonomous vehicle100may include code for detecting/recognizing vehicle taillights of an ado vehicle in a region of interest in an image captured by the sensor processor144. The instructions loaded into a processor (e.g., CPU132) may also include code for planning and control (e.g., intention prediction of the ado vehicle) in response to the vehicle taillights of the ado vehicle detected/recognized in the region of interest in the image captured by the sensor processor144.

FIG.2is a block diagram illustrating a software architecture200that may modularize artificial intelligence (AI) functions for planning and control of an autonomous agent for inferring ado vehicle intention in response to ado vehicle taillight recognition, according to aspects of the present disclosure. Using the architecture, a controller application202may be designed such that it may cause various processing blocks of an SOC220(for example a CPU222, a DSP224, a GPU226and/or an NPU228) to perform supporting computations during run-time operation of the controller application202.

The controller application202may be configured to call functions defined in a user space204that may, for example, provide for taillight recognition of ado vehicles. The controller application202may make a request to compile program code associated with a library defined in a taillight prediction application programming interface (API)206to perform taillight recognition of an ado vehicle. This request may ultimately rely on the output of a convolutional neural network configured to focus on portions of the sequence of images critical to vehicle taillight recognition.

A run-time engine208, which may be compiled code of a runtime framework, may be further accessible to the controller application202. The controller application202may cause the run-time engine208, for example, to take actions for controlling the autonomous agent. When an ado vehicle is detected within a predetermined distance of the autonomous agent, the run-time engine208may in turn send a signal to an operating system210, such as a Linux Kernel212, running on the SOC220. The operating system210, in turn, may cause a computation to be performed on the CPU222, the DSP224, the GPU226, the NPU228, or some combination thereof. The CPU222may be accessed directly by the operating system210, and other processing blocks may be accessed through a driver, such as drivers214-218for the DSP224, for the GPU226, or for the NPU228. In the illustrated example, the deep neural network may be configured to run on a combination of processing blocks, such as the CPU222and the GPU226, or may be run on the NPU228, if present.

FIG.3is a diagram illustrating an example of a hardware implementation for a vehicle control system300, according to aspects of the present disclosure. The vehicle control system300may be a component of a vehicle, a robotic device, or other device. For example, as shown inFIG.3, the vehicle control system300is a component of a vehicle100. Aspects of the present disclosure are not limited to the vehicle control system300being a component of the vehicle100, as other devices, such as a bus, boat, drone, or robot, are also contemplated for using the vehicle control system300. In the example ofFIG.3, the vehicle system may include a taillight recognition system390. In some examples, taillight recognition system390is configured to perform operations, including operations of the process700described with reference toFIG.7.

The vehicle control system300may be implemented with a bus architecture, represented generally by a bus330. The bus330may include any number of interconnecting buses and bridges depending on the specific application of the vehicle control system300and the overall design constraints. The bus330links together various circuits including one or more processors and/or hardware modules, represented by a processor320, a communication module322, a location module318, a sensor module302, a locomotion module323, a planning module324, and a computer-readable medium313. The bus330may also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further.

The vehicle control system300includes a transceiver314coupled to the processor320, the sensor module302, a comfort module308, the communication module322, the location module318, the locomotion module323, the planning module324, and the computer-readable medium313. The transceiver314is coupled to an antenna333. The transceiver314communicates with various other devices over a transmission medium. For example, the transceiver314may receive commands via transmissions from a user or a remote device. As another example, the transceiver314may transmit driving statistics and information from the comfort module308to a server (not shown).

In one or more arrangements, one or more of the modules302,313,314,318,320,322,323,324,390, can include artificial or computational intelligence elements, such as, neural network, fuzzy logic or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules302,313,314,318,320,322,323,324,390can be distributed among multiple modules302,313,314,318,320,322,323,324,390described herein. In one or more arrangements, two or more of the modules302,313,314,318,320,322,323,324,390of the vehicle control system300can be combined into a single module.

The vehicle control system300includes the processor320coupled to the computer-readable medium313. The processor320performs processing, including the execution of software stored on the computer-readable medium313providing functionality according to the disclosure. The software, when executed by the processor320, causes the vehicle control system300to perform the various functions described for a particular device, such as the vehicle328, or any of the modules302,313,314,318,320,322,323,324,390. The computer-readable medium313may also be used for storing data that is manipulated by the processor320when executing the software.

The sensor module302may be used to obtain measurements via different sensors, such as a first sensor303A and a second sensor303B. The first sensor303A and/or the second sensor303B may be a vision sensor, such as a stereoscopic camera or a red-green-blue (RGB) camera, for capturing 2D images. In some examples, one or both of the first sensor303A or the second sensor303B may be used to identify an intersection, a crosswalk, or another stopping location. Additionally, or alternatively, one or both of the first sensor303A or the second sensor303B may identify objects within a range of the vehicle100. In some examples, one or both of the first sensor303A or the second sensor303B may identify a pedestrian or another object in a crosswalk. The first sensor303A and the second sensor303B are not limited to vision sensors as other types of sensors, such as, for example, light detection and ranging (LiDAR), a radio detection and ranging (radar), sonar, and/or lasers are also contemplated for either of the sensors303A,303B. The measurements of the first sensor303A and the second sensor303B may be processed by one or more of the processor320, the sensor module302, the comfort module308, the communication module322, the location module318, the locomotion module323, the planning module324, in conjunction with the computer-readable medium313to implement the functionality described herein. In one configuration, the data captured by the first sensor303A and the second sensor303B may be transmitted to an external device via the transceiver314. The first sensor303A and the second sensor303B may be coupled to the vehicle328or may be in communication with the vehicle328.

Additionally, the sensor module302may configure the processor320to obtain or receive information from the one or more sensors303A and303B. The information may be in the form of one or more two-dimensional (2D) image(s) and may be stored in the computer-readable medium313as sensor data. In the case of 2D, the 2D image is, for example, an image from the one or more sensors303A and303B that encompasses a field-of-view about the vehicle100of at least a portion of the surrounding environment, sometimes referred to as a scene. That is, the image is, in one approach, generally limited to a subregion of the surrounding environment. As such, the image may be of a forward-facing (e.g., the direction of travel) 30, 90, 120-degree field-of-view (FOV), a rear/side facing FOV, or some other subregion as defined by the characteristics of the one or more sensors303A and303B. In further aspects, the one or more sensors303A and303B may be an array of two or more cameras that capture multiple images of the surrounding environment and stitch the images together to form a comprehensive 330-degree view of the surrounding environment. In other examples, the one or more images may be paired stereoscopic images captured from the one or more sensors303A and303B having stereoscopic capabilities.

The location module318may be used to determine a location of the vehicle328. For example, the location module318may use a global positioning system (GPS) to determine the location of the vehicle328. The communication module322may be used to facilitate communications via the transceiver314. For example, the communication module322may be configured to provide communication capabilities via different wireless protocols, such as Wi-Fi, long term evolution (LTE), 3G, etc. The communication module322may also be used to communicate with other components of the vehicle328that are not modules of the vehicle control system300. Additionally, or alternatively, the communication module322may be used to communicate with an occupant of the vehicle100. Such communications may be facilitated via audio feedback from an audio system of the vehicle100, visual feedback via a visual feedback system of the vehicle, and/or haptic feedback via a haptic feedback system of the vehicle.

The locomotion module323may be used to facilitate locomotion of the vehicle328. As an example, the locomotion module323may control movement of the wheels. As another example, the locomotion module323may be in communication with a power source of the vehicle328, such as an engine or batteries. Of course, aspects of the present disclosure are not limited to providing locomotion via wheels and are contemplated for other types of components for providing locomotion, such as propellers, treads, fins, and/or jet engines.

The vehicle control system300also includes the planning module324for planning a route or controlling the locomotion of the vehicle328, via the locomotion module323. A route may be planned to a passenger based on compartment data provided via the comfort module308. In one configuration, the planning module324overrides the user input when the user input is expected (e.g., predicted) to cause a collision. The modules may be software modules running in the processor320, resident/stored in the computer-readable medium313, one or more hardware modules coupled to the processor320, or some combination thereof.

The taillight recognition system390may be in communication with the sensor module302, the transceiver314, the processor320, the communication module322, the location module318, the locomotion module323, the planning module324, and the computer-readable medium313. In some examples, the behavior planning system may be implemented as a machine learning model, such as a vehicle control system300as described with reference toFIG.3. Working in conjunction with one or more of the sensors303A,303B, the sensor module302, and/or one or more other modules313,314,318,320,322,323,324, the taillight recognition system390may associate, by a velocity model (e.g., velocity head560) of a taillight recognition system, one or more objects within the environment with a respective velocity instance label. The taillight recognition system390may also selectively focus, by a recurrent network (e.g., LSTM430) of the taillight recognition system, on a selected region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task. Additionally, the taillight recognition system390may concatenate the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. Furthermore, the taillight recognition system390may infer, at a classifier (e.g., classifier570) of the taillight recognition system, an intent of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label. Also, the taillight recognition system390may plan a trajectory of the ego vehicle based on inferring the intent of the one or more objects.

As discussed, various aspects of the present disclosure use a CNN-LSTM framework for recognizing a vehicle taillight.FIG.4is a block diagram illustrating a CNN-LSTM framework400for vehicle taillight recognition, according to aspects of the present disclosure. The CNN-LSTM framework400integrates a spatial attention model420and a temporal attention model440for vehicle taillight recognition. An overview of the CNN-LSTM framework400is shown inFIG.4.

Representatively, an input image sequence402of the CNN-LSTM framework400is a chunk of a video sequence, typically sampled by window-sliding along the temporal direction. In this configuration, an instance detection/segmentation technique is used to extract an ego vehicle's bounding boxes from video frames of the input images sequence. For example, assume X={x}t=1T+1is a chunk of a video with T+1 frames, where xtis the tthframe in the chunk. First, a frame difference x′tis calculated to align the vehicles in successive frames:
x′t|Ψt→t+1(Xt)−Xt+1|, fort=1, 2, . . . ,T.(1)

In EQUATION 1, Ψt→t+1(·) is a warping function, from the tthframe to the next frame. In the example ofFIG.4, each image is forwarded to certain layers of a CNN410(e.g., CNNfront) to obtain deep features in the lthplayers, denoted by Zl={ztl}t=1T. The input of the spatial attention model420, ztl, is forwarded to 2D convolutional layers (shown as 1×1 conv). The convolutional layers may be concatenated with an internal state ct−1, or a hidden variable ht−1(not shown inFIG.4) from an LSTM network430. Then, the concatenated tensors are forwarded to the fully connected layers (FC), the hyperbolic tangent (tanh) layer, and the softmax layer (not shown inFIG.4) of the spatial attention model420, to obtain the attention weights αt. An element wise product of the attention weights αtand ztlmay be computed at the CNN410to obtain selective deep features, which are then forwarded to the remaining portions of layers of the CNN410(e.g., CNNend) for latent features Zf={ztf}i=1T. The latent features ztfare then forwarded to the LSTM network430for encoding temporal dependencies.

The input and the output of the LSTM network430are recurrent over time steps. At time step t, the input is the latent feature ztffrom the CNN410, the hidden variable ht−1, and the internal states ct−1, while the output is the hidden unit htand a memory cell ctfor the next time step t+1. Both htand ctare updated and then passed to the LSTM network430at each time step.

The LSTM network430outputs a set of hidden variables H={ht}i=1Tand a set of memory cells C={ct}i=1T, which may be used in the temporal attention model440. The temporal attention model440calculates the attention by dot-production between decoder context and encoder representations. Instead of multiple decoder layers, however, a single layer is used to receive the output of the LSTM network430. In some implementations (not shown inFIG.4), the temporal attention model440receives inputs htand ct, that are fused to be a set of state summaries D={dt}i=1T. Then, a matrix product of the hidden variables H and state summaries D may be performed, where the results are forwarded to a softmax layer for attention selection. The attention weights are then applied to the hidden variables H. The adjusted hidden variables H′ are followed by fully connected layers and the tanh layer to obtain class probability distribution P={pt}i=1T.

In some other implementations, as shown inFIG.4, the temporal attention model440receives inputs ht−1and ct−1. A matrix product of the inputs ht−1and ct−1may be performed, where the results are forwarded to a softmax layer for attention selection. The softmax layer may generate a temporal attention βtcorresponding to the tthstate summary. A matrix product of the temporal attention βt(e.g., attention weights) and the hidden variable ht−1may be forwarded to the fully connected (FC) layers and the tanh layer to obtain class probability distribution P={pt}i=1T.

Visual attention has been shown to be an effective mechanism in image applications, by selectively focusing on certain regions in images. In this aspect of the present disclosure, the spatial attention model420is relied on for region selection. First, the inputs Zl, D convolutional layers ϕ1 and ϕ2 with kernel size 1 are provided. The convolutional layer ϕ1 has both input and output channel d, while the convolutional layer ϕ2 has input channel d and output channel 1. Then, the 2D attention weights αt(i, j) for each time step t at coordinate (i, j) are defined by:

αt=Wa⁢tan⁢h⁡(Waz⁢ϕ2(ϕ1(ztl))+Wah⁢ht+bazh)+ba,(2)αt(i,j)=exp⁡(at(i,j))i∑⁢j∑⁢exp⁡(at(i,j)),at⁢ϵ⁢ℝ2,(3)
where Wa, Waz, Wahare the learnable parameter matrices, and bazh, baare the bias vectors. The attention weight αtis a 2D matrix where each cell spatially corresponds to the vector in ztl. A softmax selection is adopted to emphasize the corresponding regions in the latent features. By performing element-wise production with αt, the weighted ztlare forwarded to the CNNendof the CNN410.

In a sequence, the input at each time t contains temporal information with different importance for the final classification. For example, the moment when taillights are flashing is more valuable than others while the network tries to recognize the state of the vehicle's taillights. Hence, an attention model is equipped along a temporal direction to emphasize critical moments for vehicle taillight recognition.

Based on the outputs provided by the LSTM network430, the temporal attention model440, and the spatial attention model420are integrated into the CNN-LSTM framework400. For example, a temporal attention fit βt,ucorresponding to the tthstate summary and the uthhidden variable is computed as a dot-product between dtand hu, and then followed by a soft-selection:

In EQUATION 4, the state summary dtat time step t is defined by:
dt=Wdhht+Wdctanh(ct)+bdhc,  (5)
where Wdh, Wdcthe learnable parameter matrices, and bdis a bias vector. This implies how the uthinput contributes to the tthoutput. Therefore, the output hidden variable is adjusted according to βt,u−,
h′t=Σu=1Tβt,uhu,  (6)

The adjusted hidden variable h′tis then forwarded to the fully connected layers and tanh layer to obtain a final prediction ptfor the tthtime step:
pt=Wptanh (Wphh′icj+bphc)+bp, (7)
where Wp, Wph, Wpcare the learnable parameter matrices, and bphc, baare the bias vectors.

During training, the objective loss is cross entropy loss between the predictions and labels. To take the temporal dependence of an input sequence into account, we focus on the prediction of the last frame (e.g., pT) containing sufficient information from all the previous frames. In other words, the loss of the last frame is computed and back propagated to all frames in the sequence with the same loss.

Due to the mutual influence of the three models, the CNN-LSTM framework400is optimized effectively. First, the CNN410along with LSTM network430are trained from scratch. This allows the main stream of the CNN-LSTM framework400to achieve certain convergence. Then, the CNN-LSTM framework400, along with the temporal attention model440, are fine-tuned based on the pre-training of the spatial attention model420in the first step. Finally, the spatial attention model420, the temporal attention model440, and the CNN-LSTM framework400are fine-tuned from the pre-training of the temporal attention model440in the second step. Such progressive training can enable effective network convergence and better performance results.

For example, after learning, the CNN-LSTM framework400is equipped with attention models (e.g.,420and440) for spatial and temporal dimensions. The spatial attention model420and the temporal attention model440enable the CNN-LSTM framework400to emphasize focal regions of the images of the input image sequence402, as well as more important time steps of the input image sequence402. In one aspect of the present disclosure, integration of the spatial attention model420with the CNN410of the CNN-LSTM framework400enables training of the CNN-LSTM framework400to selectively focus on certain images in the input image sequence402for region selection. In addition, integration of the temporal attention model440with the LSTM network430of the CNN-LSTM framework400helps train the LSTM network430to perform frame selection within the sequence of images from the region selection. For example, the spatial attention model420may be configured along a temporal dimension (e.g., direction) to focus on portions of the sequence of images from the region selected as important to vehicle taillight recognition.

Furthermore, as discussed, in some examples, a velocity of objects in a scene may be determined by a machine learning model. In some such examples, a flow network is used to determine a velocity of objects in a scene.FIG.5Ais a block diagram illustrating an example of a flow network500, in accordance with various aspects of the present disclosure. The flow network500may be a component of the taillight recognition system390described with reference toFIG.3. Additionally, or alternatively, one or more elements of the flow network500may be executed via the SOC130and/or processor320as described with reference toFIGS.1and3, respectively. The flow network500may estimate a 2D birds-eye view (BeV) flow by combining a pillar feature network506and a flow estimation network520.

As shown in the example ofFIG.5A, a pillar feature network506may receive a first LiDAR sweep502associated with a current time (t) and a second LiDAR sweep504associated with a previous time (t−1). The two sweeps502,504may be consecutive in time and may be aligned into a same coordinate frame. That is, the original coordinates of the second LiDAR sweep504may be transformed to a coordinate frame of the first LiDAR sweep502using the odometry information of an ego agent, such as the vehicle100described with reference toFIGS.1A,1B,1C, and3. The LiDAR sweeps502,504may be referred to as point clouds or 3D point clouds. The two point clouds502,504may be encoded by the pillar feature network506to generate two BeV pseudo-images514A,514B, where each cell in the BeV pseudo-images514A,514B has a learned embedding based on points that fall inside it.

As discussed, the pillar feature network506may be used to extract two BeV pseudo-images514A,514B from 3D point clouds502,504. In some examples, the same pillar feature network506may be used to process each 3D point cloud502,504. In other examples, different pillar feature networks506may be used to process each 3D point cloud502,504. In such examples, the different pillar feature networks506may share weights for a point network510. The process for extracting each BeV pseudo-image514A,514B may include voxelizing a respective point cloud514A,514B via a voxelizer508. The voxelizer508may discretize an x-y plane, thus creating a set of pillars (e.g., grid cells) in birds-eye-view. The vowelizing point cloud may be structured as a (D, P, N)-shaped tensor where the variable D represents a number of point descriptors, the variable P represents a number of pillars, and the variable N represents a number of points per pillar. In some examples, the number of point descriptors D may be set to nine, where the first four values denote coordinates x, y, z, and reflectance r. The next five values are the distances to an arithmetic mean xc, yc, zc, of all points in a pillar and an offset xp, ypfrom a pillar center.

The tensor generated by the voxelizer508may be processed by a multi-layered point network (e.g., pointnet)510. The pointnet510may generate a feature map having a shape (C, P, N). The feature map may be compressed by a max operation over a last dimension, resulting in a (C, P) encoded feature map with a C-dimensional feature embedding for each pillar. Finally, the encoded features may be scattered back to original pillar locations, via a scatter module512, to create a BeV pseudo-image tensor514A or514B having a shape (C, H, W), where H indicates a height and W indicates a width of the BeV pseudo-image514A or514B.

In some examples, a 2D BeV flow estimation is performed to accurately associate the embeddings (e.g., pillar features) between the 2D BeV pseudo-images514A,514B. In some examples, architecture parameters such as receptive field and correlation layer parameters may be adjusted to account for a maximum relative motion that would be expected to be encountered between consecutive LiDAR sweeps502,504(given the time delta between frames, grid resolution, and typical vehicle speeds). As shown inFIG.5A, the pillar features may be further encoded via a feature pyramid network516. The feature pyramid network516may include multiple layers518. The encoded features of the second point cloud504may be warped based on a warping function522. The warping function522may align the tail lights between two images. A cost volume layer524is then used to estimate the flow between the first point cloud502and the second point cloud504by matching a cost. That is, the cost volume layer524may determine a correlation between the two feature maps associated with the point clouds502,504. The flow estimator526may fuse extracted features in a current scale level with the estimated flows from lower scales. Finally, a context network528is applied to exploit contextual information for additional refinement. The context network528is a feed-forward CNN based on dilated convolutions, along with batch normalization and rectified linear activation unit (ReLU). As shown inFIG.5A, the features received at the warping function522, and flow estimator526may be upsampled.

The 2D BeV flow530generated by the flow network500may identify the motion of objects surrounding an ego agent associated with the flow network500. In some examples, the 2D BeV flow530may include a 2D flow vector for each grid cell. In such examples, the 2D flow vector may be a single mean velocity and co-variance per object cluster. A grid cell may be a cell in an occupancy grid map (OGM). Occupancy grid maps may be used to represent scene obstacle occupancy for robotics applications. Estimation of a per cell motion state within an occupancy grid may be referred to as dynamic occupancy grid map (DOGMa) estimation.

FIG.5Bis a block diagram illustrating an example of an end-to-end framework for a taillight recognition system550that uses velocity as a prior for predicting an intent of a vehicle, in accordance with various aspects of the present disclosure. In some examples, the taillight recognition system550may be a component of a vehicle, such as an ego vehicle100. Additionally, the taillight recognition system550may determine an intent of one or more other vehicles, such as ADO vehicles.

As shown in the example ofFIG.5B, a CNN410receives an input image sequence402as described with reference toFIG.4. The CNN410may output latent features of the input image sequence402to the LSTM network430. The latent features may include spatial and temporal features generated by one or more images. Additionally, as shown in the example ofFIG.5B, a flow network500may generate a 2D flow vector for each grid cell in a scene. As discussed, the 2D flow vector represents a single mean velocity and co-variance per object cluster. The 2D flow vector may be processed by a velocity head560to determine a velocity of each object. In some examples, as shown inFIG.5B, the 2D flow vector (e.g., flow from time t−1 to time t, shown as flow (142)) and the embedding E1 and E2 from the respective BeV pseudo-images514A,514B may be processed by a velocity head560to determine a velocity of each object in a scene (e.g., environment).

The velocity may be concatenated with an output of the LSTM network430, such as, ht−1and ct−1as shown inFIG.4. The output of the LSTM network430may be a selected region in a frame. The selected region may be a region corresponding to a taillight of one or more objects in a frame. As discussed, the selected region may be concatenated with a respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The velocity instance label may be generated by the velocity head560. In some other examples, a temporal attention model440may be optional and the concatenated output of the LSTM network430may be received directly at the classifier. In such examples, the LSTM network430, selects frames within the selected region of the sequence of images according to a temporal attention model for the vehicle taillight recognition task. Additionally, in such examples, the velocity instance label may be concatenated with the selected frames within the selected region.

The concatenated output may be processed by a classifier570to predict an intent of a vehicle (e.g., an ADO vehicle) in the input image sequence402. The classifier570may infer an intent of each object of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label.

As shown inFIG.5B, the velocity predicted by the velocity head560may be associated with a velocity lossv. Furthermore, the prediction by the classifier570may be associated with a classification lossclass(c, g). During training, the taillight recognition system550may be trained to reduce a total losstotal, which may be a sum of velocity lossv, the classification lossclass(c, g), and an optical flow lossf, where the variable c represents a predicted class and the variable g represents a ground truth, such that the loss is determined based on the predicted class and the groundtruth class. During training, annotated object tracks may be used to generate one or more 2D BeV flow ground truths. For each object, the training estimates the instantaneous velocity from the difference in object positions divided by the elapsed time between consecutive frames. Additionally, during training, it is assumed that only labeled dynamic objects can have a valid velocity, and non-labeled obstacles and background should have zero velocity. Note this assumption may be violated in practice and does not provide direct supervision for potential out-of-ontology moving objects, still, static objects have zero flow. The optical flow loss optical flow loss Lf captures the loss between the labeled dynamic objects associated with a velocity, non-labeled obstacles and background associated with a zero velocity, and respective groundtruths.

As discussed, in some implementations, the taillight recognition system390is trained end-to-end. The end-to-end training may improve an overall parked car classification. That is, because the taillight recognition system390is trained end-to-end, the total losstotal, which may be a sum of the velocity lossv, the classification lossclass(c, g), and the optical flow lossf, may be minimized for the system as a whole. In contrast, conventional systems are not trained end-to-end. Rather, conventional parked car classification systems may separately train each component. Because each component is separately trained, the loss may not be minimized for the system as a whole, therefore, the parked car classification may less accurate in comparison to a system that is trained end-to-end.

FIG.5Cillustrates a block diagram for training a taillight recognition system390, in accordance with various aspects of the present disclosure. In one configuration, data (x) may be received from different sensors of the vehicle100and/or other information sources. As an example, the data (x) may include the LiDAR sweeps502,504, the input sequence402and/or other features as described inFIGS.4,5A, and5B. The data source may also store ground truth vectors (y*) corresponding to the sensor data (x). The ground truth vectors (y*) may store ground truth information for the flow determined by the flow network500, the velocity determined by the velocity network560, and the parked car classification determined by the parked vehicle classifier570.

The parked vehicle recognition system390may be initialized with a set of parameters w. The parameters may be used by layers of the parked vehicle recognition system390to set weights and biases of the associated networks, such as the vehicle feature model500, the flow network520, the velocity network570, and/or the parked vehicle classifier550. During training, the taillight recognition system390receives sensor data (x) to transform the sensor data (x) to a vector (y). In the example ofFIG.5D, the taillight recognition system390is represented as a function FO. The output (y) includes the flow estimation, the velocity estimation, and the taillight classification.

The output (y) of the parked vehicle recognition system390is received at a loss function580. The loss function580compares the output (y) to the ground truth vector (y*). The error is the difference (e.g., loss) between the output (y) and the ground truth vector (y*). The error is output from the loss function608to the model600. The error is backpropagated through the parked vehicle recognition system390to update the parameters. The training may be performed during an offline phase of the taillight recognition system390. As discussed, the taillight recognition system390may be trained to reduce a total losstotal, which may be a sum of the velocity lossv, the classification lossclass(c, g), and the optical flow lossf.

FIG.6illustrates examples600of different vehicle taillight states, in accordance with various aspects of the present disclosure. The taillight examples ofFIG.6may be identified by a taillight recognition system, such as the taillight recognition system390described with reference toFIGS.3,5B, and5C.

In the example ofFIG.6, eight different taillight states a-h are based on all combinations of brake and turn lights. As shown in examples (a)-(h), each state is denoted by three letters of “B” (brake), “L” (left), and “R” (right), in which either the corresponding letter is displayed when on, or a letter “O” (off) is displayed the corresponding signal is off. For example, in example (a), all taillights are off (OOO). In example (b), only the brake light is on (BOO); in example (c) only the left turn light is on (OLO); in example (d), both the brake light and the left turn light are on (BLO); in example (e), only the right turn light is on (OOR); in example (f), both the brake light and the right turn light are on (BOR); in example (g), both the left turn light and the right turn light are on (OLR); and in example (h), all the taillights are on (BLR).

As discussed, example (e) ofFIG.6is an example of an “OOR” taillight state, in which the right-turn signal is flashing. From frame 1, the spatial attentions are uniformly on the rear of the vehicle. With the right-turn signal off from frame 2 to frame 3, the spatial attention emphasizes the region of the right-turn signal until the last frame. Then, the temporal attention pays increased attention to frame 4, while the right-turn signal is on at this time step. Meanwhile, the temporal attention weight goes up to 0.8, which implies that the network pays more attention to significant changes of the signal.

Additionally, example (h) ofFIG.6is an example of “BLR” taillight state, in which the brake signal is on and both turn signals are flashing. In some examples, the spatial attention focuses on both sides of the vehicle. Both left-turn and right-turn signals start to turn on at frame 4. This triggers the network to pay temporal attention to frame 4. The signals stay off until frame 7, while the temporal attention weight goes up to 0.6, which is higher than that in frame 4. This occurs because the CNN-LSTM framework400tends to learn a cycle of signal flashing for this chunk of sequences from the input image sequence402.

FIG.7is a flowchart illustrating a process700for vehicle taillight recognition, according to aspects of the present disclosure. The process700may be performed by the taillight recognition system390and/or associated components, as described with reference toFIGS.3,4,5A,5B, and5C. As shown inFIG.7, the process700begins by associating, by a velocity model (e.g., velocity head560) of a taillight recognition system, one or more objects within the environment with a respective velocity instance label. In some examples, the process700generating, via a flow model (e.g., flow network500) of the taillight recognition system, a two-dimensional (2D) flow vector for each cell grid of a group of cell grids based on a first representation and a second representation of the environment. The process700may then determine the respective velocity instance label of each object of the one or more objects based on the 2D flow vector for each cell grid. In such examples, the process700may obtain the first representation via a first light detection and ranging (LiDAR) sweep performed at a first time period, and also obtain the second representation via a second LiDAR sweep performed at a second time period.

At block704, the process700selectively focuses, by a recurrent network (e.g., LSTM430) of the taillight recognition system, on a selected region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task. In some examples, the process700identifies regions of interest in the sequence of images. In such examples, the selected region is the region of interest.

At block706, the process700concatenates the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. At block708, the process700infers, at a classifier (e.g., classifier570) of the taillight recognition system, an intent of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label. At block710, the process700plans a trajectory of the ego vehicle based on inferring the intent of the one or more objects. In some examples, planning the trajectory includes adjusting the trajectory of the ego vehicle to avoid a collision with one object of the one or more objects.

In some examples, the flow model is associated with a flow loss, the velocity model is associated with a velocity loss, and the classifier is associated with a classification loss. Furthermore, the process700may train the taillight recognition system in an end-to-end manner to minimize a sum of the flow loss, the velocity loss, and the classification loss.

The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a processor configured according to the present disclosure, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine specially configured as described herein. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The processor may be responsible for managing the bus and processing, including the execution of software stored on the machine-readable media. Examples of processors that may be specially configured according to the present disclosure include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Software shall be construed broadly to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Machine-readable media may include, by way of example, random access memory (RAM), flash memory, read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable media may be embodied in a computer-program product. The computer-program product may comprise packaging materials.