Methods and apparatuses for operating a self-driving vehicle

Aspects of the present disclosure may include methods, apparatuses, and computer readable media for receiving one or more images having a plurality of objects, receiving a notification from an occupant of the self-driving vehicle, generating an attention map highlighting the plurality of objects based on at least one of the one or more images and the notification, and providing at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

TECHNICAL FIELD

The present disclosure generally relates to controlling self-driving vehicles.

BACKGROUND

Self-driving vehicles may provide more comfort to occupants because the vehicles may require minimum human input in when navigating on the roads. The occupants may read, converse with each other, use phones, or even sleep as the vehicles autonomously drive from the origin to the destination. Further, self-driving vehicles may be safer than human driven vehicles by eliminating distractions, fatigue, and emotions that may cause drivers to incorrectly or dangerously operate the vehicles.

Algorithms for implementing autonomous driving may be important to the development of self-driving vehicles. These algorithms may include training a deep neural network to map a dashcam image to steering controls, implementing a statement model using a dilated deep neural network and recurrent neural network to predict a vehicle's motion, and other computer visions/machine learning techniques. When implementing autonomous driving, the algorithms may be unable to address the different needs of the occupants, such as comfort (e.g., operating the vehicles with minimum “jerking” motions to prevent motion sickness) speed (e.g., arriving at the destination at early as possible without violating traffic laws), and/or fuel conservation (e.g., reducing rapid acceleration or deceleration). Therefore, improvements in algorithms for operating self-driving vehicles may be desirable.

SUMMARY

Aspects of the present disclosure may include receiving one or more images having a plurality of objects, receiving a notification from an occupant of the self-driving vehicle, generating an attention map highlighting the plurality of objects based on at least one of the one or more images and the notification, and providing at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

Other aspects of the present disclosure may include a self-driving vehicle having a memory and one or more processors configured to perform the steps of receiving one or more images having a plurality of objects, receiving a notification from an occupant of the self-driving vehicle, generating an attention map highlighting the plurality of objects based on at least one of the one or more images and the notification, and providing at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

Some aspects of the present disclosure may include computer readable media having instructions stored therein, the instructions, when executed by one or more processors of a self-driving vehicle, cause the one or more processors to receive one or more images having a plurality of objects, receive a notification from an occupant of the self-driving vehicle, generate an attention map highlighting the plurality of objects based on at least one of the one or more images and the notification, and provide at least one of a steering control or a velocity control to operate the self-driving vehicle based on the attention map and the notification.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting.

A “processor,” as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other computing that may be received, transmitted and/or detected.

A “bus,” as used herein, refers to an interconnected architecture that is operably connected to transfer data between computer components within a singular or multiple systems. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus may also be a vehicle bus that interconnects components inside a vehicle using protocols, such as Controller Area network (CAN), Local Interconnect Network (LIN), among others.

A “memory,” as used herein may include volatile memory and/or non-volatile memory. Non-volatile memory may include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM) and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and/or direct RAM bus RAM (DRRAM).

An “operable connection,” as used herein may include a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a physical interface, a data interface and/or an electrical interface.

A “vehicle,” as used herein, refers to any moving vehicle that is powered by any form of energy. A vehicle may carry human occupants or cargo. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, a passenger bus, motorcycles, scooters, ATVs, generators, lawnmowers boats, personal watercraft, and aircraft. In some cases, a motor vehicle includes one or more engines.

Turning toFIG. 1, an example of an environment100for providing control commands to a self-driving vehicle110may include an occupant112. The self-driving vehicle110may further include one or more cameras160and/or input device162communicatively coupled to a prediction system170having a visual encoder172, an notification encoder174, a visual attention generator176, and a vehicle controller178. The one or more cameras160may be capable of capturing still or moving images in a vicinity of the self-driving vehicle110, such as in front of, next to, or behind the vehicle110. The input device162may include a microphone, a physical keyboard, a key pad, a virtual keyboard or other devices capable of receiving notification from the occupant112. The visual encoder172may be configured to process images and extract one or more visual feature vectors from the processed images. The visual encoder172may provide descriptions associated with the one or more visual feature vectors. The notification encoder174may accept input notification (e.g., “pedestrians are in crosswalk” or “drive slow in a school zone”), such as voice input notification and/or textual input notification. The visual attention generator176may rely on the visual features vectors to generate one or more attention heatmaps including attention weight “particles” (spatial points on the processed image indicating the salient portions). The vehicle controller178may utilize the attention heatmaps and/or the input notification to provide controlling commands (e.g., stop, decelerate, or turn) to the self-driving vehicle110.

In some implementations, the example of the environment100may include one or more images114captured by the one or more cameras160and/or one or more notification116received by the input device162. The one or more images114may illustrate the surroundings near the self-driving vehicle110, such as the front of, next to, or behind the self-driving vehicle110. The one or more images114may include images captured at different times, different angles with respect to the self-driving vehicles110, different camera resolutions, different color schemes (i.e., full color, black/white), etc. In a non-limiting example, the one or more images114may include a first object120a, a second object120b, a third object120c, a pedestrian122, and a moving car124. The objects120may be approximately stationary, and the pedestrian122and the moving car124may be moving or approximately stationary. The one or more notification116may be provided by the occupant112relating to driving practices, safety, comfort, or other scenarios.

In some implementations, during normal operations, the prediction system170may analyze the one or more images114captured by the one or more cameras160. After capturing the one or more images114, the prediction system170may utilize the visual encoder172to preprocess the one or more images114and extract a set of visually descriptive latent vectors. The notification encoder174may process and/or analyze the one or more notification116, such as performing a speech-to-text conversion of the one or more notification116and extracting the content of the one or more notification116. The visual attention generator176may utilize the descriptive latent vectors relating to the one or more images114and/or the content of the one or more notification116to generate at least one attention heatmap highlighting at least some portions of the one or more images114. The vehicle controller178may rely on the at least one attention heatmap to output control signals to control the acceleration and/or the steering of the self-driving vehicle110.

WhileFIG. 1shows an example of providing commands to the self-driving vehicle110by a person, the algorithms, methods, apparatuses, and/or computer media described in the present disclosure are not so limited. For example, the algorithms, methods, apparatuses, and/or computer media described in the present disclosure may be implemented in a self-driving vehicle, including levels 1-5 autonomous vehicles. Further, in other examples, the algorithms, methods, apparatuses, and/or computer media described in the present disclosure may be implemented in robotic appliances, pattern prediction applications, modeling and/or simulations applications, video games, computer games, and/or other applications.

Aspects of the present disclosure may be implemented using hardware, software, or a combination thereof and may be implemented in one or more computer systems or other processing systems. In an aspect of the present disclosure, features are directed toward one or more computer systems capable of carrying out the functionality described herein. For example, features of the prediction system170may be implemented as one or more computer systems described inFIG. 2. An example of such the computer system200is shown inFIG. 2.

The computer system200includes one or more processors, such as the processor204. The processor204is connected to a communication infrastructure206(e.g., a communications bus, cross-over bar, or network). Various software aspects are described in terms of this example computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement aspects of the disclosure using other computer systems and/or architectures.

The processor204may include the visual encoder172having a convolutional neural network (CNN)172afor obtaining a set of visually-descriptive latent vectors. The processor204may include the notification encoder174having a textual encoder174aand a notification long short-term memory (LSTM)174b. The textual encoder174amay convert the one or more notification116spoken by the occupant112into texts. The notification LSTM174bmay encode the one or more notification116into a fixed size latent vector representing the content of the one or more notification116. The processor204may include the vehicle controller178having a control LSTM178athat tracks the current state of the self-driving vehicle110and outputs control signals to control the steering and the acceleration of the self-driving vehicle110.

The computer system200may include a display interface202that forwards graphics, text, and other data from the communication infrastructure206(or from a frame buffer not shown) for display on a display unit230. Computer system200also includes a main memory208, preferably random access memory (RAM), and may also include a secondary memory210. The secondary memory210may include, for example, a hard disk drive212, and/or a removable storage drive214, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, a universal serial bus (USB) flash drive, etc. The removable storage drive214reads from and/or writes to a removable storage unit218in a well-known manner. Removable storage unit218represents a floppy disk, magnetic tape, optical disk, USB flash drive etc., which is read by and written to removable storage drive214. As will be appreciated, the removable storage unit218includes a computer usable storage medium having stored therein computer software and/or data.

Alternative aspects of the present disclosure may include secondary memory210and may include other similar devices for allowing computer programs or other instructions to be loaded into computer system200. Such devices may include, for example, a removable storage unit222and an interface220. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units222and interfaces220, which allow software and data to be transferred from the removable storage unit222to computer system200.

Computer system200may also include a communications interface224. Communications interface224allows software and data to be transferred between computer system200and external devices. Examples of communications interface224may include a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. Software and data transferred via communications interface224are in the form of signals228, which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface224. These signals228are provided to communications interface224via a communications path (e.g., channel)226. This path226carries signals228and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, an RF link and/or other communications channels. In this document, the terms “computer program medium” and “computer usable medium” are used to refer generally to media such as a removable storage drive218, a hard disk installed in hard disk drive212, and signals228. These computer program products provide software to the computer system200. Aspects of the present disclosure are directed to such computer program products.

Computer system200may include a camera interface240for receiving image data from the one or more cameras160. The camera interface240may communicate with the one or more cameras160via wired or wireless communications media. The image data may be transmitted in Joint Photographic Experts Group (JPEG) format, Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), Windows Bitmap (BMP) format, Portable Network Graphics (PNG) format, or other suitable formats.

The computer system200may include an input interface242for receiving input notification, such as voice input, gesture input, and/or text input, from the input device162. The input device162may include a microphone, a physical keyboard, a key pad, a virtual keyboard or other devices capable of receiving notification from the occupant112.

Computer programs (also referred to as computer control logic) are stored in main memory208and/or secondary memory210. Computer programs may also be received via communications interface224. Such computer programs, when executed, enable the computer system200to perform the features in accordance with aspects of the present disclosure, as discussed herein. In particular, the computer programs, when executed, enable the processor204to perform the features in accordance with aspects of the present disclosure. Accordingly, such computer programs represent controllers of the computer system200.

In an aspect of the present disclosure where the method is implemented using software, the software may be stored in a computer program product and loaded into computer system200using removable storage drive214, hard drive212, or communications interface220. The control logic (software), when executed by the processor204, causes the processor204to perform the functions described herein. In another aspect of the present disclosure, the system is implemented primarily in hardware using, for example, hardware components, such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).

FIG. 3illustrates a block diagram of various example system components, in accordance with an aspect of the present disclosure.FIG. 3shows a communication system300usable in accordance with aspects of the present disclosure. The communication system300includes one or more accessors360,362(also referred to interchangeably herein as one or more “users”) and one or more terminals342,366. In one aspect, data for use in accordance with aspects of the present disclosure is, for example, input and/or accessed by accessors360,362via terminals342,366, such as personal computers (PCs), minicomputers, mainframe computers, microcomputers, telephonic devices, or wireless devices, such as personal digital assistants (“PDAs”) or a hand-held wireless devices coupled to a server343, such as a PC, minicomputer, mainframe computer, microcomputer, or other device having a processor and a repository for data and/or connection to a repository for data, via, for example, a network344, such as the Internet or an intranet, and couplings345,346,364. The couplings345,346,364include, for example, wired, wireless, or fiberoptic links. In another example variation, the method and system in accordance with aspects of the present disclosure operate in a stand-alone environment, such as on a single terminal.

Referring toFIG. 4, and referencingFIG. 2, in some implementations, an example of an algorithm400for operating the self-driving vehicle110may utilize a visual (spatial) attention mechanism that highlights image regions to generate an output, such as a steering and/or an acceleration command. The visual encoder172of the prediction system170may optionally preprocess the one or more images114. For example, the visual encoder172may down-sample the one or more images114at, for example, 10 kilohertz (kHz). The visual encoder172of the prediction system170may resize the one or more images114to a dimensionality, such as 90×160×3. Each image of the one or more images114may then be normalized by subtracting the mean from the raw pixels and dividing by the standard deviation. The visual encoder172may change the saturation, hue, and brightness of the one or more images114for achieving robustness during a training phase. Other preprocessing mechanisms may also be used to prepare the one or more images114.

In some examples, the CNN172aof the visual encoder172may obtain a set of visually-descriptive latent vectors (xt,i) at time t, where each vector may contain a high-level visual description in certain input region of the one or more images114. The set of visually-descriptive latent vectors xt,imay collectively form a convolutional feature cube Xt. By feeding an image of the one or more images114through the algorithm at each time t, the CNN172amay construct the cube Xtof size w×h×d. The cube Xtmay have l (=w×h) (spatially) different visually-descriptive latent vectors xt,ieach of which may be a d-dimensional feature slice corresponding to a certain input region of the one or more images114. Mathematically, the cube Xtmay be defined as follows: Xt={t,1,t,2, . . . , xt,l}, where xt,i, ∈dfor i ∈ {1, 2, . . . , l}. Choosing a subset of these vectors may allow the prediction system170to focus selectively on different parts of the one or more images114(i.e., attention).

In some implementations, the notification encoder174may accept two types of notification, i.e., the goal-oriented and the stimulus-driven notification, without any input-level separation (i.e., the occupant112may input both types of notification similarly). The notification LSTM174bmay encode the one or more notification116and to generate the fixed-size latent vector. The notification encoder174may receive a variable-length notification (e.g., verbal or textual), such as the one or more notification116, and output a latent vector u representing the one or more notification116. For verbal notification, the textual encoder174aof the notification encoder174may perform a speech-to-text conversion and generate textual representations of the one or more notification116. The notification LSTM174bmay receive the textual representations of the one or more notification116(directly input by the occupant112or converted by the textual encoder174a) and produce a latent vector u. The latent vector u may have the same dimension as the visually descriptive latent vectors xt,i. The notification encoder174, the textual encoder174a, and/or the notification LSTM174bmay understand the one or more notification116and ground it into the vehicle controller. The one or more notification116may be given offline, or at the beginning of a trip, e.g., “look out for pedestrians” or “drive gently (occupant gets carsick).” Thus, notification encoding may be prepared ahead of the vehicle controller178generating control commands. Formally, the notification LSTM174bmay generate a d-dimensional latent vector u ∈d.

In certain examples, the notification encoder174may rely on a synthetic token <none> to indicate a lack of input from the occupant112.

Still referring toFIG. 4and referencingFIG. 2, in some implementations, the visual attention generator176may first use an element-wise multiplication to combine the latent vector u from the notification encoder174and each of the descriptive-visual latent vectors xt,ifrom the visual encoder172to obtain feature vectors zt,i=xt,i⊙ u. While the vehicle controller178may accept a new image at every time t (thus, updating xt,i), the latent vector u may remain the same or change, depending on the presence of additional inputs from the occupant112.

In some implementations, the visual attention generator176may generate one or more attention heatmaps402. Visual attention provides introspective explanations by filtering out non-salient image regions, while image areas inside the attended region have potential causal effect on the output. Next, the visual attention generator176may attempt to find a context Yt={yt,1, yt,2, . . . , yt,l} by minimizing a loss function, where yt,i=π(αt,i, xt,i)=αt,i, xt,ifor i={1, 2, . . . , l}. The scalar attention weight value at,iin [0, 1] may be associated with a certain location of the one or more images114is such that Σiαt,i=1. The visual attention generator176may use a multi-layer perceptron fattnto generate αt,i, i.e., αt,i=fattn(xt,i, ht−1) conditioned on the previous hidden state ht−1, and the current feature vector xt,i. Softmax regression function may be used to obtain the final attention weight. Based on the values of αt,iand the previous hidden state ht−1, the visual attention generator176may generate one or more attention heatmaps402having highlights404that bring visual focus to portions of the one or more attention heatmaps402.

In certain examples, to internalize stimulus-driven notification to certain images of the one or more images114, the example algorithm400includes a loss term, i.e., the Kullback-Leibler divergence (DKL), between two attention heatmaps (i.e., generated with and without notification) to make the driving model refer to the same salient objects:

where αwand αwoare the attention maps generated by the vehicle controller with and without notification given, respectively. The term hyperparameter λamay control the strength of the regularization term.

In some implementations, the vehicle controller178and/or the control LSTM178amay utilize a loss function, which includes three terms: (1)p, which may be proportional to the error (i.e., |ev(t)|+|es(t)|, where ev(t)=v(t)−{circumflex over (v)}(t) and es(t)=s(t)−ŝ(t), (2)d, which may be proportional to the derivative of the error

(i.e.,ddt⁢es(t)⁢and⁢ddt⁢es(t)),
and (3)i, which may be proportional to the integral of the error. The vehicle controller178and/or the control LSTM178amay use the difference in the future course θ(t)−a cardinal direction in which the self-driving vehicle110is to be steered. The vehicle controller178and/or the control LSTM178amay approximate a steering wheel angle st≈L/r, where L is the length of wheelbase and r is the radius of the vehicle's path. Then, the vehicle controller178and/or the control LSTM178amay approximate the vehicle's course

θ⁡(t)≈v⁡(t)⁢τr≈s⁡(t)⁢v⁡(t)
after the unit time τ=1, using the following loss function:

where T is the number of timesteps. The vehicle controller178and/or the control LSTM178amay use hyperparameters λdand λito control the strength of the terms.

Still referring toFIG. 4, in some aspects, the vehicle controller178and/or the control LSTM178amay utilize the latent vector u based on the one or more notification116and/or the one or more attention heatmaps402based on the one or more images114to generate a steering angle control ŝ(t) and a velocity control {circumflex over (v)}(t) for the self-driving vehicle110. In one non-limiting example, the vehicle controller178may transmit the steering angle control ŝ(t) signal to a mechanical, electrical, pneumatic, or hydraulic steering system known to one skilled in the art to control the steering wheel of the self-driving vehicle110, and the velocity control {circumflex over (v)}(t) signal to a mechanical, electrical, pneumatic, or hydraulic acceleration system known to one skilled in the art to control the velocity and/or acceleration of the self-driving vehicle110.

Turning toFIG. 5, a diagram500indicates a distribution of goal-oriented notification and a diagram550indicates a distribution of stimulus-driven notification. There exists a variety of notification for both types of notification. The diagram500with goal-oriented notification includes notification that mainly start with “drive/go straight,” “stop at,” and “turn right/left,” while the stimulus-driven notification start with “there is/are” and “the car/light/traffic.” The distributions of the goal-oriented notification and the stimulus-driven notification may be stored in the main memory208, the secondary memory210, the removable storage units218,222, or other suitable storage medium of the prediction system170.

In one non-limiting example, the goal-oriented notification and the stimulus-driven notification of the diagrams500,550may be derived from annotations of one or more human annotators. The one or more human annotators may be shown a quantity of video clips, for example 5,675 video clips (over 32 hours), each of which may be on average 20 seconds in length. Each video may contain around 1-2 driving activities, e.g., passing through an intersection, lane change, stopping, etc. The videos may be randomly collected from a large-scale driving video dataset. The dataset may contain camera videos—which are captured by a single front-view camera mounted in a fixed position on the roof top of the vehicle. The videos may contain the typical driver's activities (i.e., turning, merging, lane following, etc.) on various road types (i.e., highway, residential roads with and without lane markings, etc.). Alongside the video data, the dataset may provide a set of time-stamped controller area network (CAN) bus records, which contain human driver control inputs (i.e., steering wheel angle, accelerator/brake pedal). The annotators may enter the action description and attention description separately, for example, “The driver crossed lanes from right to left lane” and “There was construction happening on the road”, respectively. Each video clip may include 4-5 action descriptions (25,549 in total) and 3-4 attention descriptions (20,080 in total).

Referring toFIG. 6, in some implementations, an example of method600for operating a self-driving vehicle based on an attention map and a notification from an occupant.

At block602, the method600may receive one or more images having a plurality of objects. For example, the visual encoder172may receive the one or more images114having the pedestrian122and the moving car124.

At block604, the method600may receive a notification from an occupant of the vehicle. For example, the notification encoder174, the textual encoder174a, and/or the notification LSTM174bmay receive the one or more notification116from the occupant112of the self-driving vehicle110.

At block606, the method600may generate an attention map highlighting the plurality of objects based on at least one of the one or more images or the notification. For example, the visual attention generator176may generate one or more attention maps402highlighting the pedestrian122and the moving car124based on the one or more images114and the one or more notification116.

At block608, the method600may provide at least one of a steering control or a velocity control to control an operation of the vehicle based on the attention map and the notification. For example, the vehicle controller178may provide at least the steering angle control ŝ(t) and the velocity control {circumflex over (v)}(t) to operate the self-driving vehicle110based on the one or more attention maps402and the one or more notification116.