Autonomous Vehicle Communication Gateway Architecture

A gateway processor coordinates communication among an autonomous vehicle components boundary domain, a vehicle boundary domain, and an oversight server. The gateway processor receives a message from the oversight server. The gateway processor determines a priority level, a domain tag data, and destination data associated with the message. The priority level indicates a scheduling requirement associated with the message. The domain tag data indicates that the message is associated with a particular domain from among the autonomous vehicle components boundary domain, the vehicle components boundary domain, or the security domain. The destination data indicates that the message is designated to a particular component within the particular domain. In response, the gateway processor schedules the message to be transmitted to the particular domain. The gateway processor routes the message to the particular component.

TECHNICAL FIELD

The present disclosure relates generally to autonomous vehicles. More particularly, the present disclosure is related to an autonomous vehicle communication gateway architecture.

BACKGROUND

One aim of autonomous vehicle technology is to provide vehicles that can safely navigate with limited or no driver assistance. Various internal components of an autonomous vehicle communicate messages and instructions to one another to facilitate the autonomous and mechanical operations of the autonomous vehicle.

SUMMARY

This disclosure recognizes various problems and previously unmet needs related to autonomous vehicle communication, and more specifically to the lack of efficiency in data communication and data routing for internal and external communication for autonomous vehicles. Certain embodiments of the present disclosure provide unique technical solutions to technical problems of current autonomous vehicle technologies, including those problems described above to improve autonomous vehicle communication technology.

The disclosed system provides improvements to the autonomous vehicle communication technology, for example, by improving the data routing or data communication among the components of an on-board control device associated with the autonomous vehicle. In one example, the disclosed system improves the data routing among the components of the control device by establishing particular boundary domains for various components of the control device. For example, the disclosed system may establish an autonomous vehicle components boundary domain that includes a first set of components configured to facilitate autonomous operations of the autonomous vehicle. In another example, the disclosed system may establish a vehicle components boundary domain that includes a second set of components configured to facilitate non-autonomous operations of the autonomous vehicle. In another example, the disclosed system may establish a security boundary domain that includes a third set of components configured to facilitate authentication of components in the control device, authentication of messages received from an external device (e.g., an oversight server, etc.), messages received from an internal component with respect to the control device (e.g., any component in a boundary domain to another).

The disclosed system is configured to establish trusted communication paths among any combination of the boundary domains. For example, the disclosed system is configured to provide initial security keys to each component in each boundary domain, and query the security key from the component for authenticating the component (e.g., in response to receiving a request from the component to initiate a communication with another component). If the received security key matches or corresponds to an initially provided security key, it is determined that the component is authenticated, and communication from the component is safe and trusted.

The disclosed system is further configured to provide improvement to data routing technology, in general, and to data routing between components of the autonomous vehicle, in specific. For example, upon receiving a message, the control device may evaluate the received message and determine its priority level. If the priority level of the message is determined to be high (e.g., as indicated by priority flag bits or priority data field included in the message), the control device may move the message to be on top of a scheduling queue, or route the message to a particular scheduling queue that is dedicated for messages with high priority levels. Similarly, if the priority level of the message is determined to be medium (e.g., as indicated by priority flag bits or priority data field included in the message), the control device may route the message to a particular scheduling queue that is dedicated to messages with medium priority levels; and if the priority level of the message is determined to be low (e.g., as indicated by priority flag bits or priority data field included in the message), the control device may route the message to a particular scheduling queue that is dedicated for messages with low priority levels. In this manner, if a particular message includes particular instructions that need to be executed more urgently than other messages, the particular message may be associated with the high priority level, and its execution may be prioritized or escalated. Thus, the disclosed system improves the underlying operation of the autonomous vehicle, and network communication among components of the autonomous vehicle. This, in turn, leads to improving the autonomous vehicle navigation technology, for example, by escalating the execution of messages with high priority levels, instructions that may include urgent navigation instructions or any other suitable instruction/information may be accessed and acted upon quicker compared to the current autonomous vehicle navigation technology. This leads to providing safer driving conditions and experiences for autonomous vehicles, surrounding vehicles, and pedestrians.

In one embodiment, a system comprises a memory and a gateway processor. The memory is configured to store a first message. The gateway processor is operably coupled to the memory. The gateway processor is configured to coordinate communications among an autonomous vehicle components boundary domain, a vehicle components boundary domain, and an oversight server. The gateway processor receives the first message from the oversight server. The first message is associated with one of the autonomous vehicle components boundary domain, the vehicle components boundary domain, or a security domain. The security domain comprises a set of components configured to facilitate authentication of received messages. The gateway processor determines a priority level associated with the first message, wherein the priority level associated with the first message indicates a scheduling requirement associated with the first message. The gateway processor identifies a domain tag data associated with the first message, wherein the domain tag data indicates that the first message is associated with a particular domain from among the autonomous vehicle components boundary domain, the vehicle components boundary domain, or the security domain. The gateway processor identifies a destination data associated with the first message, wherein the destination data indicates that the first message is designated to a particular component within the particular domain. The gateway processor schedules the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data. The gateway processor routes the first message to the particular component based at least in part upon the destination data.

DETAILED DESCRIPTION

As described above, previous technologies fail to provide efficient, reliable, and safe solutions to facilitate secured and trusted communication among various components of an autonomous vehicle and route and schedule an incoming message. The present disclosure provides various systems, methods, and devices to facilitate secured and trusted communication among various components of an autonomous vehicle and route and schedule an incoming message. Embodiments of the present disclosure and its advantages may be understood by referring toFIGS.1through6.FIGS.1through6are used to describe a system and method to facilitate secured and trusted communication among various components of an autonomous vehicle and route and schedule an incoming message.

System Overview

FIG.1illustrates an embodiment of a system100configured to implement a communication gateway architecture for autonomous vehicles402to facilitate secured and trusted communication among various components of the autonomous vehicle402. In certain embodiments, the system100comprises the autonomous vehicle402communicatively coupled with an oversight server170via a network110. Network110enables communication among the components of the system100. Network110allows the autonomous vehicle402to communicate with other autonomous vehicles402, systems, oversight server170, databases, devices, etc. The autonomous vehicle402comprises a control device450. The control device450comprises a gateway processor120is signal communication with a memory126. Memory126stores software instructions128that when executed by the gateway processor120cause the gateway processor120to perform one or more operations described below. The oversight server170comprises a processor172in signal communication with a memory178. Memory178stores software instructions180that when executed by the processor172cause the oversight server170to perform one or more operations described herein. In other embodiments, system100may not have all of the components listed and/or may have other elements instead of, or in addition to, those listed above. System100may be configured as shown or in any other configuration.

In general, the system100provides improvements to the autonomous vehicle technology, for example, by improving the data routing or data communication among the components of the control device450. In one example, the system100may improve the data routing among the components of the control device450by establishing particular boundary domains for various components of the control device450. For example, the system100may establish the autonomous vehicle components boundary domain102that includes a first set of components configured to facilitate autonomous operations of the autonomous vehicle402. In another example, the system100may establish the vehicle components boundary domain104that includes a second set of components configured to facilitate non-autonomous operations of the autonomous vehicle402. In another example, the system100may establish the security boundary domain106that includes a third set of components configured to facilitate authentication of components in the control device450, authentication of messages140received from an external device (e.g., oversight server170, etc.), messages140received from an internal component with respect to the control device450(e.g., any component in a boundary domain to another).

The system100is configured to establish trusted communication paths among any combination of the boundary domains102,104,106using the security boundary domain106. For example, the system100is configured to provide initial security keys156to each component in each boundary domain102,104, and106, and query the security key156from the component for authenticating the component (e.g., in response to receiving a request from the component to initiate a communication with another component). If the received security key156matches or corresponds to an initially provided security key156, it is determined that the component is authenticated, and communication from the component is safe and trusted.

The system100is further configured to provide improvement to data routing technology, in general, and to data routing between components of the autonomous vehicle, in specific. For example, upon receiving a message140(from an external device, such as the oversight server170), the control device450may evaluate the received message140and determine its priority level210. If the priority level210of the message140is determined to be high (e.g., as indicated by priority flag bits or priority data field included in the message140), the control device450may move the message140to be on top of a scheduling queue, or route the message140to a particular scheduling queue that is dedicated for messages140with high priority levels. Similarly, if the priority level210of the message140is determined to be medium (e.g., as indicated by priority flag bits or priority data field included in the message140), the control device450may route the message140to a particular scheduling queue that is dedicated for messages140with medium priority levels; and if the priority level210of the message140is determined to be low (e.g., as indicated by priority flag bits or priority data field included in the message140), the control device450may route the message140to a particular scheduling queue that is dedicated for messages140with low priority levels. In this manner, if a particular message140includes particular instructions that need to be executed more urgently than other messages, the particular message140may be associated with the high priority level, and its execution may be prioritized. Thus, system100improves the underlying operation of the autonomous vehicle, and network communication among components of the autonomous vehicle402. This, in turn, leads to improving the autonomous vehicle navigation technology, for example, by escalating the execution of messages with high priority levels, instructions that may include navigation instructions, updated map data134, or any other suitable instruction/information may be accessed and acted upon quicker compared to the current autonomous vehicle navigation technology. This leads to providing safer driving conditions and experience for the autonomous vehicle402, surrounding vehicles, and pedestrians.

System Components

Network110may include any interconnecting system capable of transmitting audio, video, signals, data, messages, or any combination of the preceding. Network110may include all or a portion of a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a personal area network (PAN), a wireless PAN (WPAN), an overlay network, a software-defined network (SDN), a virtual private network (VPN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., cellular networks, such as 4G or 5G), a plain old telephone (POT) network, a wireless data network (e.g., WiFi, WiGig, WiMAX, etc.), a long-term evolution (LTE) network, a universal mobile telecommunications system (UMTS) network, a peer-to-peer (P2P) network, a Bluetooth network, a near field communication (NFC) network, a Zigbee network, a Z-wave network, a WiFi network, and/or any other suitable network.

Example Autonomous Vehicle

In one embodiment, the autonomous vehicle402may include a semi-truck tractor unit attached to a trailer to transport cargo or freight from one location to another location (seeFIG.4). The autonomous vehicle402is generally configured to travel along a road in an autonomous mode. The autonomous vehicle402may navigate using a plurality of components described in detail inFIGS.4-6. The operation of the autonomous vehicle402is described in greater detail inFIGS.4-6. The corresponding description below includes brief descriptions of certain components of the autonomous vehicle402.

Control device450may be generally configured to control the operation of the autonomous vehicle402and its components and to facilitate autonomous driving of the autonomous vehicle402. The control device450may be further configured to determine a pathway in front of the autonomous vehicle402that is safe to travel and free of objects or obstacles, and navigate the autonomous vehicle402to travel in that pathway. This process is described in more detail inFIGS.4-6. The control device450may generally include one or more computing devices in signal communication with other components of the autonomous vehicle402(seeFIG.4). In this disclosure, the control device450may interchangeably be referred to as an in-vehicle control computer450.

The control device450may be configured to detect objects on and around a road traveled by the autonomous vehicle402by analyzing the sensor data130and/or map data134. For example, the control device450may detect objects on and around the road by implementing object detection machine learning modules132. The object detection machine learning modules132may be implemented using neural networks and/or machine learning algorithms for detecting objects from images, videos, infrared images, point clouds, audio feed, Radar data, etc. The object detection machine learning modules132are described in more detail further below. The control device450may receive sensor data130from the sensors446positioned on the autonomous vehicle402to determine a safe pathway to travel. The sensor data130may include data captured by the sensors446.

Sensors446may be configured to capture any object within their detection zones or fields of view, such as landmarks, lane markers, lane boundaries, road boundaries, vehicles, pedestrians, road/traffic signs, among others. In some embodiments, the sensors446may be configured to detect rain, fog, snow, and/or any other weather condition. The sensors446may include a detection and ranging (LiDAR) sensor, a Radar sensor, a video camera, an infrared camera, an ultrasonic sensor system, a wind gust detection system, a microphone array, a thermocouple, a humidity sensor, a barometer, an inertial measurement unit, a positioning system, an infrared sensor, a motion sensor, a rain sensor, and the like. In some embodiments, the sensors446may be positioned around the autonomous vehicle402to capture the environment surrounding the autonomous vehicle402. See the corresponding description ofFIG.4for further description of the sensors446.

Control Device

The control device450is described in greater detail inFIG.4. In brief, the control device450may facilitate the autonomous driving of the autonomous vehicle402. In the illustrated embodiment, the control device450includes the autonomous vehicle components boundary domain102, the vehicle components boundary domain104, and security boundary domain106. The control device450may establish these boundary domains based on the operations of various components of autonomous vehicle402.

Autonomous Vehicle Components Boundary Domain

The autonomous vehicle components boundary domain102may include a first set of components configured to facilitate the autonomous operations of the autonomous vehicle402. For example, the components in the autonomous vehicle components boundary domain102may be configured to engage the autonomous driving of the autonomous vehicle402, e.g., from a non-autonomous state to an autonomous state, execute various software instructions128for perception, actuation, control570(seeFIG.5), planning562(seeFIG.5), object detection (e.g., LiDAR-based object detection module512ofFIG.5, image-based object detection module518ofFIG.5, machine learning object detection module132), among others. The autonomous vehicle components boundary domain102may include the gateway processor120, one or more autonomous drive compute (ADS) units122a-c, a pulse per second (PPS) synchronization unit123, a network interface124, a controller area network (CAN) controller125, and a memory126. The components of the autonomous vehicle components boundary domain102are operably coupled to each other through wires and/or wireless communication.

The gateway processor120in signal communication with the ADC units122a-c, PPS unit123, network interface124, CAN controller125, memory126, and other components in other domains104,106. The gateway processor120may include one or more processing units that perform various functions as described herein. The memory126may store any data and/or instructions used by the gateway processor120to perform its functions. For example, the memory126may store software instructions128that when executed by the gateway processor120causes the control device450to perform one or more functions described herein.

The gateway processor120may be one of the data processors470described inFIG.4. The gateway processor120comprises one or more processors. The gateway processor120may be any electronic circuitry, including state machines, one or more central processing unit (CPU) chips, logic units, cores (e.g., a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or digital signal processors (DSPs). The gateway processor120may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The gateway processor120may be communicatively coupled to and in signal communication with the other components of the control device450. The one or more processors may be configured to process data and may be implemented in hardware or software. For example, the gateway processor120may be 8-bit, 16-bit, 32-bit, 64-bit, or of any other suitable architecture. The gateway processor120may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The one or more processors may be configured to implement various instructions. For example, the one or more processors may be configured to execute software instructions128to implement the functions disclosed herein, such as some or all of those described with respect toFIGS.1-6. In some embodiments, the function described herein is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry.

Each ADC unit122a-cmay include a hardware processing circuitry or a hardware processor that is configured to execute software algorithms that when executed facilitate one or more autonomous operations of the autonomous vehicle402. For example, each ADC unit122a-cmay be configured to facilitate engaging from non-autonomous state to autonomous state, autonomous driving of the autonomous vehicle, among others.

The ADC unit122a-cmay be one of the data processors470described inFIG.4. The ADC unit122a-ccomprises one or more processors operably coupled to the other components of the control device450, such as the gateway processor120. The ADC unit122a-cmay be any electronic circuitry, including state machines, one or more CPU chips, logic units, cores (e.g., a multi-core processor), FPGAs, ASICs, or DSPs. The ADC unit122a-cmay be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The ADC unit122a-cmay be communicatively coupled to and in signal communication with the network interface124, memory126, and other components of the control device450. The one or more processors may be configured to process data and may be implemented in hardware or software. For example, the ADC unit122a-cmay be 8-bit, 16-bit, 32-bit, 64-bit, or of any other suitable architecture. The ADC unit122a-cmay include an ALU for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The one or more processors may be configured to implement various instructions. For example, the one or more processors may be configured to execute software instructions (e.g., autonomous instructions) to implement the functions disclosed herein, such as some or all of those described with respect toFIGS.1-6. In some embodiments, the function described herein is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry. AlthoughFIG.1illustrates that the control device450includes three ADC units122a-c, the control device450may include any suitable number of ADC units122a-c.

The PPS synchronization unit123may be implemented in software and/or hardware and executed by the gateway processor120executing the software instructions128, and generally be configured to synchronize the operation timing between the components of the control device450. For example, the PPS synchronization unit123may distribute (or cause the gateway processor120to distribute) the timing for operations from the gateway processor120to other components of the control device450. For example, the PPS synchronization unit123may distribute among the components that the timing for the executing instructions is one instruction per millisecond, two instructions per millisecond, and the like. The PPS synchronization unit123may be interchangeably referred to herein as a timing synchronization component.

Network interface124may be a component of the network communication subsystem492described inFIG.4. The network interface124may be configured to enable wired and/or wireless communications. The network interface124may be configured to communicate data between the autonomous vehicle402and other devices, systems, or domains. For example, the network interface124may comprise an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-wave interface, a radio-frequency identification (RFID) interface, a WIFI interface, a local area network (LAN) interface, a wide area network (WAN) interface, a metropolitan area network (MAN) interface, a personal area network (PAN) interface, a wireless PAN (WPAN) interface, a modem, a switch, and/or a router. The gateway processor120may be configured to send and receive data using the network interface124. The network interface124may be configured to use any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.

CAN controller125may be a component of the vehicle subsystem interface460described inFIG.4. The CAN controller125may be configured to allow communication among the components of the control device450without a host computer device. The CAN controller125may be a message-based protocol or any other suitable type of communication protocols. The CAN controller125may allow serial and/or parallel data transmission. For example, for a high-priority message140, the communication of the message140may be prioritized over other message140with lower priority levels. For example, for a high-priority message140, the communication of the message140may be implemented with parallel data transmission, while other data are transmitted serially or queued in a scheduling queue according to their priority levels.

The memory126may be one of the data storages490described inFIG.4. The memory126may be volatile or non-volatile and may comprise read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), dynamic random-access memory (DRAM), and static random-access memory (SRAM). The memory126may include one or more of a local database, cloud database, network-attached storage (NAS), etc. The memory126may store any of the information described inFIGS.1-6along with any other data, instructions, logic, rules, or code operable to implement the function(s) described herein when executed by the gateway processor120and/or any of the ADC units122a-c. For example, the memory126may store software instructions128, sensor data130, object detection machine learning module132, map data134, routing plan136, driving instructions138, messages140, priority levels210, domain tag data212, destination data214, and/or any other data/instructions. The software instructions128include code that when executed by the gateway processor120and/or ADC unit122a-ccauses the control device450to perform the functions described herein, such as some or all of those described inFIGS.1-6. The memory126comprises one or more disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution.

Object detection machine learning modules132may be implemented by the gateway processor120and/or ADC unit122a-cexecuting software instructions128, and may be generally configured to detect objects and obstacles from the sensor data130. The object detection machine learning modules132may be implemented using neural networks and/or machine learning algorithms for detecting objects from any data type, such as images, videos, infrared images, point clouds, audio feed, Radar data, etc.

In some embodiments, the object detection machine learning modules132may be implemented using machine learning algorithms, such as Support Vector Machine (SVM), Naive Bayes, Logistic Regression, k-Nearest Neighbors, Decision Trees, or the like. In some embodiments, the object detection machine learning modules132may utilize a plurality of neural network layers, convolutional neural network layers, Long-Short-Term-Memory (LSTM) layers, Bi-directional LSTM layers, recurrent neural network layers, and/or the like, in which weights and biases of these layers are optimized in the training process of the object detection machine learning modules132. The object detection machine learning modules132may be trained by a training dataset that may include samples of data types labeled with one or more objects in each sample. For example, the training dataset may include sample images of objects (e.g., vehicles, lane markings, pedestrians, road signs, obstacles, etc.) labeled with object(s) in each sample image. Similarly, the training dataset may include samples of other data types, such as videos, infrared images, point clouds, audio feed, Radar data, etc. labeled with object(s) in each sample data. The object detection machine learning modules132may be trained, tested, and refined by the training dataset and the sensor data130. The object detection machine learning modules132use the sensor data130(which are not labeled with objects) to increase their accuracy of predictions in detecting objects. For example, supervised and/or unsupervised machine learning algorithms may be used to validate the predictions of the object detection machine learning modules132in detecting objects in the sensor data130.

Map data134may include a virtual map of a city or an area that includes the road traveled by an autonomous vehicle402. In some examples, the map data134may include the map558and map database1136(seeFIG.5for descriptions of the map558and map database1136). The map data134may include drivable areas, such as roads, paths, highways, and undrivable areas, such as terrain (determined by the occupancy grid module1160, seeFIG.5for descriptions of the occupancy grid module1160). The map data134may specify location coordinates of road signs, lanes, lane markings, lane boundaries, road boundaries, traffic lights, obstacles, etc.

Routing plan136may be a plan for traveling from a start location (e.g., a first autonomous vehicle launchpad/landing pad) to a destination (e.g., a second autonomous vehicle launchpad/landing pad). For example, the routing plan136may specify a combination of one or more streets, roads, and highways in a specific order from the start location to the destination. The routing plan136may specify stages, including the first stage (e.g., moving out from a start location/launch pad), a plurality of intermediate stages (e.g., traveling along particular lanes of one or more particular street/road/highway), and the last stage (e.g., entering the destination/landing pad). The routing plan136may include other information about the route from the start position to the destination, such as road/traffic signs in that routing plan136, etc.

Driving instructions138may be implemented by the planning module562(See descriptions of the planning module562inFIG.5). The driving instructions138may include instructions and rules to adapt the autonomous driving of the autonomous vehicle402according to the driving rules of each stage of the routing plan136. For example, the driving instructions138may include instructions to stay within the speed range of a road traveled by the autonomous vehicle402, adapt the speed of the autonomous vehicle402with respect to observed changes by the sensors446, such as speeds of surrounding vehicles, objects within the detection zones of the sensors446, etc.

Vehicle Components Boundary Domain

The vehicle components boundary domain104may include a second set of components configured to facilitate non-autonomous operations of the autonomous vehicle402. For example, the vehicle components boundary domain104may include components that perform mechanical operations of the autonomous vehicle402, such as vehicle drive subsystems442(seeFIG.4), vehicle control subsystem448(seeFIG.4), among others. For example, the vehicle components boundary domain104may include a communication module142, vehicle component controller144, vehicle components146, and authentication components148.

The communication module142may be or include a hardware processor, a modem, a router, or a network interface configured to provide software and/or hardware resources to other components of the control device450. The communication module142may be one of the components of the data processors470(seeFIG.4). The communication module142may be interchangeably referred to as a communication processor. The communication module142comprises one or more processors operably coupled to the other components of the control device450, such as the gateway processor120. The communication module142may be any electronic circuitry, including state machines, one or more CPU chips, logic units, cores (e.g., a multi-core processor), FPGAs, ASICs, or DSPs. The communication module142may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The communication module142may be communicatively coupled to and in signal communication with the vehicle component controller144, vehicle components146, authentication components148, and other components of the control device450. The one or more processors may be configured to process data and may be implemented in hardware or software. For example, the communication module142may be 8-bit, 16-bit, 32-bit, 64-bit, or of any other suitable architecture. The communication module142may include an ALU for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The one or more processors may be configured to implement various instructions. For example, the one or more processors may be configured to execute software instructions to implement the functions disclosed herein, such as some or all of those described with respect toFIGS.1-6. In some embodiments, the function described herein is implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry.

The vehicle component controller144may include a hardware processing circuitry configured to control vehicle components146. The vehicle component controller144may be associated with the vehicle control subsystem448described inFIG.4. The vehicle components146may be any of the components in the vehicle control subsystem448described inFIG.4. For example, the vehicle components146may include a human machine interface, a break unit, a power distribution unit, a camera array, a microphone array, a speaker array, sensors446, among others. Each vehicle component146may be configured to perform its respective operations as described herein inFIG.4. The human machine interface may be configured to provide support audio, visual, and/or message-based communication. The human machine interface may be configured to support one or two-way communication. Using the human machine interface, a person may be able to communicate via network110with another device (e.g., the oversight server170) and/or with a remote operator. The power distribution unit may be implemented in hardware and/or software, and configured to distribute power to the components of the autonomous vehicle402. The power distribution unit may be a component in the power source442edescribed inFIG.1. The components of the vehicle components boundary domain104are operably coupled to each other through wires and/or wireless communication.

The authentication component148may include a hardware processor, memory, and/or circuitry (not explicitly shown), and is generally configured to authenticate components and communication among components of the vehicle component boundary domain104. For example, a software application designed using software code may be stored in the memory and executed by the processor to perform the functions of the authentication component148. Examples of the authentication component148may include a near-field communication (NFC) device, a mobile phone (e.g., smartphone), a laptop, a computing device, and the like. The authentication component148is configured to communicate with other components of the vehicle components boundary domain104via wires and/or wireless communication.

Security Boundary Domain

The security boundary domain106may include a third set of components configured to facilitate authentication/authorization of any component in the control device450, authentication of any message140received from an external device (e.g., oversight server170, etc.) and messages140received from an internal component with respect to the control device450(e.g., any component of the control device450to another component). The security boundary domain106may include a memory152. The memory152may be one of the data storages490described inFIG.4. The memory152may be volatile or non-volatile and may comprise ROM, RAM, TCAM, DRAM, and SRAM. The memory152may include one or more of a local database, cloud database, NAS, etc. The memory152may store any of the information described inFIGS.1-6along with any other data, instructions, logic, rules, or code operable to implement the function(s) described herein when executed by the gateway processor120and/or any of the ADC units122a-c. For example, the memory152may store authentication/authorization instructions154, security keys156, access management158, and/or any other data/instructions. The software instructions128include code that when executed by the gateway processor120and/or ADC unit122a-ccause the control device450to perform the functions described herein, such as some or all of those described inFIGS.1-6. The memory152comprises one or more disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device to store programs when such programs are selected for execution and to store instructions and data that are read during program execution.

The authentication/authorization instructions154include code that when executed by the gateway processor120and/or ADC unit122a-cand/or a processor in the security boundary domain106cause the control device450to perform the functions described herein, such as authenticating a component of the control device450that initiates to communicate with another component, upon authenticating the component, authorizing the communication, distribute a security key156to each component of the control device450to be used for authenticating each component.

The security keys156may include a plurality of security keys, security code, and the like used for authenticating each component of the control device450. The security keys156may also be used to establish secured communication paths between any two combinations of components in one or more boundary domains102,104, and106. The security keys156may also be used to establish secured communication paths between any two combinations of boundary domains102,104, and106. For example, the control device450and/or the gateway processor120(e.g., by executing the authentication/authorization instructions154) may establish a trusted communication path between the autonomous vehicle components boundary domain102and the vehicle components boundary domain104by receiving an initial private security key156from the oversight server170, sharing the initial private security key156with the communication module142, receiving a request from the communication module142to communicate a message140, e.g., to the gateway processor120, where the request includes the message140and a private security key156, receiving a private security key156from the communication module142, and comparing the received private security key156with the initial private security key156(received from the oversight server170). If it is determined that the received private security key156corresponds to or matches the initial private security key156, the control device450may determine that the communication module142is authenticated and authorized to communicate the message140, e.g., to the gateway processor120. The control device450may perform a similar operation for establishing a trusted communication path between any two domains or components of the control device450.

The access management158may include records of access to security keys156(e.g., records of components associated with particular security keys156), and historical records of access to the security keys156, among others. The access management158may indicate which component(s) is authorized to initiate a communication, i.e., trusted. The access management158may also indicate which component(s) is not authorized to access message(s)140.

Oversight Server

Oversight server170may include one or more processing devices and is generally configured to oversee the operations of the autonomous vehicle402while they are in transit and oversee the traveling of the autonomous vehicle402and while they are at a terminal. The oversight server170may provide software and/or hardware resources (e.g., map data134, routing plans136, messages140, recommendations, feedback from a remote operator on autonomous vehicle navigation, etc.) to the autonomous vehicles402. The oversight server170may comprise a processor172, a network interface174, a user interface176, and a memory178. The components of the oversight server170are operably coupled with each other. The processor172may include one or more processing units that perform various functions of the oversight server170. The memory178may store any data and/or instructions used by the processor172to perform its functions. For example, the memory178may store software instructions180that when executed by the processor172cause the oversight server170to perform one or more functions described herein. The oversight server170may be configured as shown or in any other suitable configuration.

In one embodiment, the oversight server170may be implemented by a cluster of computing devices that may serve to oversee the operations of the autonomous vehicle402. For example, the oversight server170may be implemented by a plurality of computing devices using distributed computing and/or cloud computing systems. In another example, the oversight server170may be implemented by a plurality of computing devices in one or more data centers. As such, in one embodiment, the oversight server170may include more processing power than the control device450. The oversight server170is in signal communication with the autonomous vehicle402and its components (e.g., the control device450).

Processor172comprises one or more processors. The processor172may be any electronic circuitry, including state machines, one or more CPU chips, logic units, cores (e.g., a multi-core processor), FPGAs, ASICs, or DSPs. The processor172may be a programmable logic device, a microcontroller, a microprocessor, or any suitable combination of the preceding. The processor172may be communicatively coupled to and in signal communication with the network interface174, user interface176, and memory178. The one or more processors are configured to process data and may be implemented in hardware or software. For example, the processor172may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor172may include an ALU for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The one or more processors are configured to implement various instructions. For example, the one or more processors are configured to execute software instructions180to implement the functions disclosed herein, such as some or all of those described with respect toFIGS.1-6. In some embodiments, the function described herein may be implemented using logic units, FPGAs, ASICs, DSPs, or any other suitable hardware or electronic circuitry.

Network interface174may be configured to enable wired and/or wireless communications of the oversight server170. The network interface174may be configured to communicate data between the oversight server170and other devices, servers, autonomous vehicles402, systems, or domains. For example, the network interface174may comprise an NFC interface, a Bluetooth interface, a Zigbee interface, a Z-wave interface, an RFID interface, a WIFI interface, a LAN interface, a WAN interface, a PAN interface, a modem, a switch, and/or a router. The processor172may be configured to send and receive data using the network interface174. The network interface174may be configured to use any suitable type of communication protocol as would be appreciated by one of ordinary skill in the art.

User interfaces176may include one or more user interfaces that are configured to interact with users, such as a remote operator. The remote operator may access the oversight server170via a communication path. In certain embodiments, the user interfaces176may include peripherals of the oversight server170, such as monitors, keyboards, mouse, trackpads, touchpads, microphones, webcams, speakers, and the like. In certain embodiments, the user interface176may include a graphical user interface, a software application, or a web application. The remote operator may use the user interfaces176to access the memory178to review any data stored in the memory178. The remote operator may confirm, update, and/or override the routing plan136, messages140, map data134, and/or any other data stored in memory178.

Memory178may be volatile or non-volatile and may comprise ROM, RAM, TCAM, DRAM, and SRAM. The memory178may include one or more of a local database, cloud database, NAS, etc. Memory178may store any of the information described inFIGS.1-6along with any other data, instructions, logic, rules, or code operable to implement the function(s) described herein when executed by processor172. For example, the memory178may store software instructions150, sensor data130, object detection machine learning module132, map data134, routing plan136, driving instructions138, messages140, and/or any other data/instructions. The software instructions180may include code that when executed by the processor172causes the oversight server170to perform the functions described herein, such as some or all of those described inFIGS.1-6. The memory178comprises one or more disks, tape drives, or solid-state drives, and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution.

Operational Flow for Facilitating Secured Communication for the Autonomous Vehicle

FIG.2illustrates an example operational flow200of system100ofFIG.1for facilitating secured communication for the autonomous vehicle402. The gateway processor120may be configured to coordinate communications between the autonomous vehicle402and external devices, systems, such as the oversight server170, other autonomous vehicles402, and the like. The gateway processor120may also be configured to coordinate communications between the internal components of the control device450, including components in the autonomous vehicle components boundary domain102, the vehicle components boundary domain104, and security boundary domain106, similar to that described inFIG.1. The gateway processor120may also be configured to coordinate communications between any combination of the autonomous vehicle components boundary domain102, the vehicle components boundary domain104, the security boundary domain106, and the oversight server170.

The operational flow200may begin when the gateway processor120receives a message140from the oversight server170, e.g., via the network (110inFIG.1). Examples of the message140may include a command to engage the autonomous operations, the map data134, an autonomous software algorithm update, a minimal risk maneuver command that comprises instructions to pull the autonomous vehicle over to a side of a road or stop the autonomous vehicle without hindering the traffic, security software instruction updates, autonomous vehicle configuration files, access management updates, diagnostics data, ADS unit122a-clog data, ADC unit122a-cconfiguration data, event triggers, human machine interface audio (e.g., when a communication path is established between with a device at the autonomous vehicle402and a remote operator at the oversight server170such that the remote operator can be heard from the device), human machine interface video (e.g., when a communication path is established between with a device at the autonomous vehicle402and a remote operator at the oversight server170such that the remote operator can be seen from the device), and any other suitable data/instruction that can be transmitted to the control device450. The message140may be associated with one of the domains102,104,106. For example, the message140may be designated to be received by a particular component in one of the domains102,104,106. The message140may be in form of a particular data structure/format, data object fields, or software code that can be evaluated by the gateway processor120, such as a structured data package.

Determining a Priority Level and Destination of the Message

The gateway processor120may evaluate the message140to extract information from it, for example, to determine a priority level210, domain data212, and destination data214associated with the message140. To this end, the gateway processor120may parse the message140. The priority level210associated with the message140may indicate the scheduling requirement associated with the message140.

In certain embodiments, the priority level210associated with the message140may indicate whether the priority, e.g., for executing and/or routing the message140to a respective destination is low, medium, or high. In certain embodiments, determining the priority level210associated with the message140may include determining that the message140is associated with a priority level tag data that indicates the priority level210of the message140. For example, the priority level210may be indicated by priority flag bits or priority data field in the message140. For example, if the priority flag bits are “11”, it may be determined that the priority level210is high, if the priority flag bits are “01”, it may be determined that the priority level210is medium, and if the priority flag bits are “00”, it may be determined that the priority level210is low. Other levels of priority levels210may also be possible.

In certain embodiments, the priority level210may be indicated by a value. for example, the priority level210may be determined to be high when it is more than a threshold value (e.g., more than 8 out of 10), the priority level210may be determined to be medium when it is determined to be between two threshold values (e.g., between 4 and 8 out of 10), the priority level210may be determined to be low when it is determined to be less than a threshold value (e.g., less than 4 out of 10).

In certain embodiments, determining the priority level210associated with the message140may include determining that the message140is associated with a particular internet protocol (IP) address that is associated with the priority level210of the message140. For example, different IP addresses may be used to transmit messages140with different priority levels210. The control device540may be provided, by the oversight server170, with a table or list of IP addresses each used to transmit messages140with a particular priority level210.

In response to determining that the message140is associated with a first IP address (that is preset to be used for transmitting messages140with high priority levels based on the table of IP addresses), the gateway processor120may determine that the priority level210of the message140is a high priority level. In response to determining that the message140is associated with a second IP address (that is preset to be used for transmitting messages140with medium priority levels based on the table of IP addresses), the gateway processor120may determine that the priority level210of the message140is a medium priority level. In response to determining that the message140is associated with a third IP address (that is preset to be used for transmitting messages140with high priority levels based on the table of IP addresses), the gateway processor120may determine that the priority level210of the message140is a low priority level.

The gateway processor120may also determine a domain tag data212associated with the message140. The domain tag data212may indicate that the message140is associated with or designated to a particular domain from the domains102,104,106. In certain embodiments, determining the domain tag data212may be in response to determining a domain data (e.g., domain flag bits or domain data field) transmitted along or included in the message140, where the domain data may indicate the particular domain for which the message is designated.

The gateway processor120may also identify destination data214associated with the message140. The destination data214may indicate that the message140is designated to a particular component216within the particular domain102,104,106identified from the domain tag data212. In some cases, the particular component216may be an internal software or hardware component with respect to the gateway processor120. In some cases, the particular component216may be an external software or hardware component with respect to the gateway processor120, such as any of the ADS units122a-c, PPS123, memory126, memory152, communication module142, vehicle component controller144, vehicle component146, or any other component described inFIGS.1,4-6.

Scheduling the Message According to the Priority Level to the Determined Destination

In the scheduling and routing operation, the gateway processor120may schedule the message140to be transmitted to the particular domain102,104,106identified from the domain tag data212based on the priority level210, the domain tag data212, and the destination data214.

In certain embodiments, in cases where the priority level210associated with the message140is high (as indicated by priority tag data or priority data field included in the message140, similar to that described above), scheduling the message140to be transmitted to particular domain102,104,106based on the priority level210and the identified domain tag data212may include moving or routing the message140to the top of a scheduling queue that may comprise a plurality of messages associated with various priority levels.

In certain embodiments, in a case where the priority level210associated with the message140is high (as indicated by priority tag data or priority data field included in the message140, similar to that described above), scheduling the message140to be transmitted to particular domain102,104,106based on the priority level210and the identified domain tag data212may include moving or routing the message140to a particular scheduling queue that is dedicated for messages with high priority levels (e.g., messages with priority levels with more than a threshold value, such as more than 8 out 10).

In certain embodiments, in a case where the priority level210associated with the message140is medium (as indicated by priority tag data or priority data field included in the message140, similar to that described above), scheduling the message140to be transmitted to particular domain102,104,106based on the priority level210and the identified domain tag data212may include moving or routing the message140to a particular scheduling queue that is dedicated for messages with medium priority levels (e.g., messages with priority levels within two thresholds, such as between 4 and 8 out of 10).

In certain embodiments, in a case where the priority level210associated with the message140is low (as indicated by priority tag data or priority data field included in the message140, similar to that described above), scheduling the message140to be transmitted to particular domain102,104,106based on the priority level210and the identified domain tag data212may include moving or routing the message140to a particular scheduling queue that is dedicated for messages with low priority levels (e.g., messages with priority levels with less than a threshold value, such as less than 4 out 10).

In certain embodiments, the messages140communicated to a component within the security boundary domain106may not be shared with other components in the other domains102and104. For example, if the message140is designated to be transmitted to the security boundary domain106, the gateway processor120may route the message140to the security boundary domain106such that the message140is not shared with other domains102,104. This is because the message140may include private security keys156for particular components. Thus, the message140is not shared with other domains102,104.

The gateway processor120may route the message140to the particular component216based on the priority level210, domain data212, and destination data214. The particular component216may receive the message140and execute or act upon the message140according to the information/instructions included in the message140. For example, if the message140includes instructions to engage the autonomous functions (i.e., initiate autonomous driving of the autonomous vehicle402), the particular component216may execute particular autonomous driving algorithms (and optionally instruct other related components) to engage the autonomous functions of the autonomous vehicle402. In another example, if the message140includes a command to perform a minimal risk condition maneuver (e.g., pull over or stop the autonomous vehicle402), the particular component216may execute particular minimal risk condition maneuver instructions (and optionally instruct other related components) to perform the minimal risk condition maneuver. In another example, if the message140includes the updated map data134, the particular component216may use the updated map data134for traveling of the autonomous vehicle402. In this manner, the gateway processor120may receive, process, schedule, route, and act upon an incoming message140.

In contain embodiments, the gateway processor120may perform similar operations for an outgoing message140. The outgoing message140may include a request to engage the autonomous functions for the autonomous vehicle402, a location coordinate of the autonomous vehicle402, autonomous vehicle health data, sensor data130captured by the sensors464, information received from the oversight server170, information received from another autonomous vehicle402, and/or any other data/instructions/requests. The outgoing message140may be communicated to the oversight server170, other autonomous vehicles402, a device associated with an authorized person who is attempting to access the autonomous vehicle402, or information associated with the autonomous vehicle402, among others. For example, for communicating an outgoing message140, the gateway processor120may process the outgoing message140, in response determine a priority level210and destination data214associated with the outgoing message140, schedule the outgoing message140(in a particular scheduling queue based on the determined priority level210, similar to that described above with respect to an incoming message140), and transmit the outgoing message140to the destination component216defined in the destination data214.

Example Method for Facilitating Secured Communication for the Autonomous Vehicle

FIG.3illustrates an example flowchart of a method300for facilitating secured communication for autonomous vehicles402. Modifications, additions, or omissions may be made to method300. Method300may include more, fewer, or other operations. For example, operations may be performed in parallel or in any suitable order. While at times discussed as the system100, autonomous vehicle402, control device450, the gateway processor120, or components of any of thereof performing operations, any suitable system or components of the system may perform one or more operations of the method300. For example, one or more operations of method300may be implemented, at least in part, in the form of software instructions128and processing instructions480, respectively, fromFIGS.1and4, stored on non-transitory, tangible, machine-readable media (e.g., memory126and data storage490, respectively, fromFIGS.1and4) that when run by one or more processors (e.g., gateway processor120and processor470, respectively, fromFIGS.1and4) may cause the one or more processors to perform operations302-314.

At operation302, the control device450(e.g., via the gateway processor120) coordinates secured communication between the autonomous vehicle boundary domain102, the vehicle components boundary domain104, the security boundary domain106, and the oversight server170. In this process, the control device450(e.g., via the gateway processor120) may share security keys156among the boundary domains102,104,106, where the security keys156are received from the oversight server170. The control device450(e.g., gateway processor120) may establish secured communication paths between any combination of the boundary domains102,104,106, and oversight server170by sharing and/or evaluating the security keys156, similar to that described inFIGS.1and2.

At operation304, the control device450(e.g., via the gateway processor120) determines whether a message140is received. For example, the control device450(e.g., via the gateway processor120) may determine whether the message140is received from the oversight server170or another autonomous vehicle402. If it is determined that the message140is received, the method300may proceed to operation306. Otherwise, the method300may remain at operation304until a message140is received.

At operation306, the control device450(e.g., via the gateway processor120) determines a priority level210associated with the message140. To this end, the control device450(e.g., via the gateway processor120) may process the message140, for example, by extracting various data fields and/or information from the message140, such as the priority level210, domain tag data212, and destination data214, similar to that described inFIG.2. For example, the control device450(e.g., via the gateway processor120) may determine the priority level210associated with the message140from priority tag data, priority data field, and/or IP address associated with the message140, that is used to transmit the message140, similar to that described inFIG.2.

At operation308, the control device450(e.g., via the gateway processor120) identifies a domain tag data212associated with the message140, where the domain tag data212indicates a particular domain102,104,106. In this process, the control device450(e.g., via the gateway processor120) may identify the domain tag data212that is included in the message140, similar to that described inFIG.2.

At operation310, the control device450(e.g., via the gateway processor120) identifies a destination data214associated with the message140, where the destination data214indicates that the message140is designated to a particular component216, similar to that described inFIG.2.

At operation312, the control device450(e.g., via the gateway processor120) schedules the message140to be transmitted to the particular domain102,104,106. In this process, the control device450(e.g., via the gateway processor120) may route the message140to a particular scheduling queue based on the determined priority level210, the domain tag data212, and the destination data214associated with the message140, similar to that described inFIG.2. At operation314, the control device450(e.g., via the gateway processor120) routes the message140to the particular component216. Upon receipt of the message140, the particular component216may act upon the message140, e.g., execute commands or instructions included in the message140.

Example Autonomous Vehicle and its Operation

FIG.4shows a block diagram of an example vehicle ecosystem400in which autonomous driving operations can be determined. As shown inFIG.4, the autonomous vehicle402may be a semi-trailer truck. The vehicle ecosystem400may include several systems and components that can generate and/or deliver one or more sources of information/data and related services to the in-vehicle control computer450that may be located in an autonomous vehicle402. The in-vehicle control computer450can be in data communication with a plurality of vehicle subsystems440, all of which can be resident in the autonomous vehicle402. A vehicle subsystem interface460may be provided to facilitate data communication between the in-vehicle control computer450and the plurality of vehicle subsystems440. In some embodiments, the vehicle subsystem interface460can include a controller area network (CAN) controller to communicate with devices in the vehicle subsystems440.

The autonomous vehicle402may include various vehicle subsystems that support the operation of the autonomous vehicle402. The vehicle subsystems440may include a vehicle drive subsystem442, a vehicle sensor subsystem444, a vehicle control subsystem448, and/or network communication subsystem492. The components or devices of the vehicle drive subsystem442, the vehicle sensor subsystem444, and the vehicle control subsystem448shown inFIG.4are examples. The autonomous vehicle402may be configured as shown or any other configurations.

The vehicle drive subsystem442may include components operable to provide powered motion for the autonomous vehicle402. In an example embodiment, the vehicle drive subsystem442may include an engine/motor442a, wheels/tires442b, a transmission442c, an electrical subsystem442d, and a power source442e.

The vehicle sensor subsystem444may include a number of sensors446configured to sense information about an environment or condition of the autonomous vehicle402. The vehicle sensor subsystem444may include one or more cameras446aor image capture devices, a radar unit446b, one or more thermal sensors446c, a wireless communication unit446d(e.g., a cellular communication transceiver), an inertial measurement unit (IMU)446e, a laser range finder/LiDAR unit446f, a Global Positioning System (GPS) transceiver446g, a wiper control system446h. The vehicle sensor subsystem444may also include sensors configured to monitor internal systems of the autonomous vehicle402(e.g., an O2monitor, a fuel gauge, an engine oil temperature, etc.).

The IMU446emay include any combination of sensors (e.g., accelerometers and gyroscopes) configured to sense position and orientation changes of the autonomous vehicle402based on inertial acceleration. The GPS transceiver446gmay be any sensor configured to estimate a geographic location of the autonomous vehicle402. For this purpose, the GPS transceiver446gmay include a receiver/transmitter operable to provide information regarding the position of the autonomous vehicle402with respect to the Earth. The radar unit446bmay represent a system that utilizes radio signals to sense objects within the local environment of the autonomous vehicle402. In some embodiments, in addition to sensing the objects, the radar unit446bmay additionally be configured to sense the speed and the heading of the objects proximate to the autonomous vehicle402. The laser range finder or LiDAR unit446fmay be any sensor configured to use lasers to sense objects in the environment in which the autonomous vehicle402is located. The cameras446amay include one or more devices configured to capture a plurality of images of the environment of the autonomous vehicle402. The cameras446amay be still image cameras or motion video cameras.

Cameras446amay be rear-facing and front-facing so that pedestrians, and any hand signals made by them, or signs held by pedestrians, may be observed from all around the autonomous vehicle. These cameras446amay include video cameras, cameras with filters for specific wavelengths, as well as any other cameras suitable to detect hand signals, hand-held traffic signs, or both hand signals and hand-held traffic signs. A sound detection array, such as a microphone or array of microphones, may be included in the vehicle sensor subsystem444. The microphones of the sound detection array may be configured to receive audio indications of the presence of, or instructions from, authorities, including sirens and commands such as “Pull over.” These microphones are mounted, or located, on the external portion of the vehicle, specifically on the outside of the tractor portion of an autonomous vehicle. Microphones used may be any suitable type, mounted such that they are effective both when the autonomous vehicle is at rest, as well as when it is moving at normal driving speeds.

The vehicle control subsystem448may be configured to control the operation of the autonomous vehicle402and its components. Accordingly, the vehicle control subsystem448may include various elements such as a throttle and gear selector448a, a brake unit448b, a navigation unit448c, a steering system448d, and/or an autonomous control unit448e. The throttle and gear selector448amay be configured to control, for instance, the operating speed of the engine and, in turn, control the speed of the autonomous vehicle402. The throttle and gear selector448amay be configured to control the gear selection of the transmission. The brake unit448bcan include any combination of mechanisms configured to decelerate the autonomous vehicle402. The brake unit448bcan slow the autonomous vehicle402in a standard manner, including by using friction to slow the wheels or engine braking. The brake unit448bmay include an anti-lock brake system (ABS) that can prevent the brakes from locking up when the brakes are applied. The navigation unit448cmay be any system configured to determine a driving path or route for the autonomous vehicle402. The navigation unit448cmay additionally be configured to update the driving path dynamically while the autonomous vehicle402is in operation. In some embodiments, the navigation unit448cmay be configured to incorporate data from the GPS transceiver446gand one or more predetermined maps so as to determine the driving path for the autonomous vehicle402. The steering system448dmay represent any combination of mechanisms that may be operable to adjust the heading of autonomous vehicle402in an autonomous mode or in a driver-controlled mode.

The autonomous control unit448emay represent a control system configured to identify, evaluate, and avoid or otherwise negotiate potential obstacles or obstructions in the environment of the autonomous vehicle402. In general, the autonomous control unit448emay be configured to control the autonomous vehicle402for operation without a driver or to provide driver assistance in controlling the autonomous vehicle402. In some embodiments, the autonomous control unit448emay be configured to incorporate data from the GPS transceiver446g, the radar unit446b, the LiDAR unit446f, the cameras446a, and/or other vehicle subsystems to determine the driving path or trajectory for the autonomous vehicle402.

The network communication subsystem492may comprise network interfaces, such as routers, switches, modems, and/or the like. The network communication subsystem492may be configured to establish communication between the autonomous vehicle402and other systems, servers, etc. The network communication subsystem492may be further configured to send and receive data from and to other systems.

Many or all of the functions of the autonomous vehicle402can be controlled by the in-vehicle control computer450. The in-vehicle control computer450may include at least one data processor470(which can include at least one microprocessor) that executes processing instructions480stored in a non-transitory computer-readable medium, such as the data storage device490or memory. The in-vehicle control computer450may also represent a plurality of computing devices that may serve to control individual components or subsystems of the autonomous vehicle402in a distributed fashion. In some embodiments, the data storage device490may contain processing instructions480(e.g., program logic) executable by the data processor470to perform various methods and/or functions of the autonomous vehicle402, including those described with respect toFIGS.1-6.

The data storage device490may contain additional instructions as well, including instructions to transmit data to, receive data from, interact with, or control one or more of the vehicle drive subsystems442, the vehicle sensor subsystem444, and the vehicle control subsystem448. The in-vehicle control computer450can be configured to include a data processor470and a data storage device490. The in-vehicle control computer450may control the function of the autonomous vehicle402based on inputs received from various vehicle subsystems (e.g., the vehicle drive subsystem442, the vehicle sensor subsystem444, and the vehicle control subsystem448).

FIG.5shows an exemplary system500for providing precise autonomous driving operations. The system500may include several modules that can operate in the in-vehicle control computer450, as described inFIG.4. The in-vehicle control computer450may include a sensor fusion module502shown in the top left corner ofFIG.5, where the sensor fusion module502may perform at least four image or signal processing operations. The sensor fusion module502can obtain images from cameras located on an autonomous vehicle to perform image segmentation504to detect the presence of moving objects (e.g., other vehicles, pedestrians, etc.) and/or static obstacles (e.g., stop sign, speed bump, terrain, etc.) located around the autonomous vehicle. The sensor fusion module502can obtain LiDAR point cloud data item from LiDAR sensors located on the autonomous vehicle to perform LiDAR segmentation506to detect the presence of objects and/or obstacles located around the autonomous vehicle.

The sensor fusion module502can perform instance segmentation508on image and/or point cloud data items to identify an outline (e.g., boxes) around the objects and/or obstacles located around the autonomous vehicle. The sensor fusion module502can perform temporal fusion510where objects and/or obstacles from one image and/or one frame of point cloud data item are correlated with or associated with objects and/or obstacles from one or more images or frames subsequently received in time.

The sensor fusion module502can fuse the objects and/or obstacles from the images obtained from the camera and/or point cloud data item obtained from the LiDAR sensors. For example, the sensor fusion module502may determine based on a location of two cameras that an image from one of the cameras comprising one half of a vehicle located in front of the autonomous vehicle is the same as the vehicle captured by another camera. The sensor fusion module502may send the fused object information to the tracking or prediction module546and the fused obstacle information to the occupancy grid module560. The in-vehicle control computer may include the occupancy grid module560which can retrieve landmarks from a map database558stored in the in-vehicle control computer. The occupancy grid module560can determine drivable areas and/or obstacles from the fused obstacles obtained from the sensor fusion module502and the landmarks stored in the map database558. For example, the occupancy grid module560can determine that a drivable area may include a speed bump obstacle.

Below the sensor fusion module502, the in-vehicle control computer450may include a LiDAR-based object detection module512that can perform object detection516based on point cloud data item obtained from the LiDAR sensors514located on the autonomous vehicle. The object detection516technique can provide a location (e.g., in3D world coordinates) of objects from the point cloud data item. Below the LiDAR-based object detection module512, the in-vehicle control computer may include an image-based object detection module518that can perform object detection524based on images obtained from cameras520located on the autonomous vehicle. For example, the object detection518technique can employ a deep image based object detection524(e.g., a machine learning technique) to provide a location (e.g., in3D world coordinates) of objects from the image provided by the camera520.

The radar556on the autonomous vehicle can scan an area in front of the autonomous vehicle or an area towards which the autonomous vehicle is driven. The radar data may be sent to the sensor fusion module502that can use the radar data to correlate the objects and/or obstacles detected by the radar556with the objects and/or obstacles detected from both the LiDAR point cloud data item and the camera image. The radar data also may be sent to the tracking or prediction module546that can perform data processing on the radar data to track objects by object tracking module548as further described below.

The in-vehicle control computer450(as shown inFIGS.1and4) may include a tracking or prediction module546that receives the locations of the objects from the point cloud and the objects from the image, and the fused objects from the sensor fusion module502. The tracking or prediction module546also receives the radar data with which the tracking or prediction module546can track objects by object tracking module548from one point cloud data item and one image obtained at one time instance to another (or the next) point cloud data item and another image obtained at another subsequent time instance.

The tracking or prediction module546may perform object attribute estimation550to estimate one or more attributes of an object detected in an image or point cloud data item. The one or more attributes of the object may include a type of object (e.g., a pedestrian, a car, a truck, etc.). The tracking or prediction module546may perform behavior prediction552to estimate or predict the motion pattern of an object detected in an image and/or a point cloud. The behavior prediction552can be performed to detect a location of an object in a set of images received at different points in time (e.g., sequential images) or in a set of point cloud data items received at different points in time (e.g., sequential point cloud data items). In some embodiments, the behavior prediction552can be performed for each image received from a camera and/or each point cloud data item received from the LiDAR sensor. In some embodiments, the tracking or prediction module546can be performed (e.g., run or executed) to reduce computational load by performing behavior prediction552on every other or after every pre-determined number of images received from a camera or point cloud data item received from the LiDAR sensor (e.g., after every two images or after every three-point cloud data items).

The behavior prediction552feature may determine the speed and direction of the objects that surround the autonomous vehicle from the radar data, where the speed and direction information can be used to predict or determine motion patterns of objects. A motion pattern may comprise a predicted trajectory information of an object over a pre-determined length of time in the future after an image is received from a camera. Based on the motion pattern predicted, the tracking or prediction module546may assign motion pattern situational tags to the objects (e.g., “located at coordinates (x,y),” “stopped,” “driving at 50 mph,” “speeding up” or “slowing down”). The situational tags can describe the motion pattern of the object. The tracking or prediction module546may send the one or more object attributes (e.g., types of the objects) and motion pattern situational tags to the planning module562. The tracking or prediction module546may perform an environment analysis554using any information acquired by the system500and any number and combination of its components.

The in-vehicle control computer may include the planning module562that receives the object attributes and motion pattern situational tags from the tracking or prediction module546, the drivable area and/or obstacles, and the vehicle location and pose information from a fused localization module526(further described below).

The planning module562can perform navigation planning564to determine a set of trajectories on which the autonomous vehicle can be driven. The set of trajectories can be determined based on the drivable area information, the one or more object attributes of objects, the motion pattern situational tags of the objects, location of the obstacles, and the drivable area information. In some embodiments, the navigation planning564may include determining an area next to the road where the autonomous vehicle can be safely parked in a case of emergencies. The planning module562may include behavioral decision making566to determine driving actions (e.g., steering, braking, throttle) in response to determining changing conditions on the road (e.g., traffic light turned yellow, or the autonomous vehicle is in an unsafe driving condition because another vehicle drove in front of the autonomous vehicle and in a region within a pre-determined safe distance of the location of the autonomous vehicle). The planning module562performs trajectory generation568and selects a trajectory from the set of trajectories determined by the navigation planning operation564. The selected trajectory information may be sent by the planning module562to the control module570.

The in-vehicle control computer may include a control module570that receives the proposed trajectory from the planning module562and the autonomous vehicle location and pose from the fused localization module526. The control module570may include a system identifier572. The control module570can perform a model-based trajectory refinement574to refine the proposed trajectory. For example, the control module570can apply filtering (e.g., Kalman filter) to make the proposed trajectory data smooth and/or to minimize noise. The control module570may perform the robust control576by determining, based on the refined proposed trajectory information and current location and/or pose of the autonomous vehicle, an amount of brake pressure to apply, a steering angle, a throttle amount to control the speed of the vehicle, and/or a transmission gear. The control module570can send the determined brake pressure, steering angle, throttle amount, and/or transmission gear to one or more devices in the autonomous vehicle to control and facilitate precise driving operations of the autonomous vehicle.

The deep image-based object detection524performed by the image-based object detection module518can also be used to detect landmarks (e.g., stop signs, speed bumps, etc.) on the road. The fused localization module526obtains information about landmarks detected from images, the landmarks obtained from a map database536stored on the in-vehicle control computer, the landmarks detected from the point cloud data item by the LiDAR-based object detection module512. The fused localization module526also obtains information about the speed and displacement from the odometer sensor544, or a rotary encoder, and the estimated location of the autonomous vehicle from the GPS/IMU sensor538(i.e., GPS sensor540and IMU sensor542) located on or in the autonomous vehicle. Based on this information, the fused localization module526can perform a localization operation528to determine a location of the autonomous vehicle, which can be sent to the planning module562and the control module570.

The fused localization module526can estimate pose530of the autonomous vehicle based on the GPS and/or IMU sensors538. The pose of the autonomous vehicle can be sent to the planning module562and the control module570. The fused localization module526can also estimate status (e.g., location, possible angle of movement) of the trailer unit based on (e.g., trailer status estimation534), for example, the information provided by the IMU sensor542(e.g., angular rate and/or linear velocity). The fused localization module526may also check the map content532.

FIG.6shows an exemplary block diagram of an in-vehicle control computer450included in an autonomous vehicle402. The in-vehicle control computer450may include at least one processor604and a memory602having instructions stored thereupon (e.g., software instructions128and processing instructions480inFIGS.1and4, respectively). The instructions, upon execution by the processor604, configure the in-vehicle control computer450and/or the various modules of the in-vehicle control computer450to perform the operations described inFIGS.1-6. The transmitter606may transmit or send information or data to one or more devices in the autonomous vehicle. For example, the transmitter606can send an instruction to one or more motors of the steering wheel to steer the autonomous vehicle. The receiver608receives information or data transmitted or sent by one or more devices. For example, the receiver608receives a status of the current speed from the odometer sensor or the current transmission gear from the transmission. The transmitter606and receiver608also may be configured to communicate with the plurality of vehicle subsystems440and the in-vehicle control computer450described above inFIGS.4and5.

Implementations of the disclosure can be described in view of the following clauses, the features of which can be combined in any reasonable manner.

Clause 1. A system comprising:

a memory configured to store a first message; and

a gateway processor operably coupled with the memory, and configured to:coordinate communications among an autonomous vehicle components boundary domain, a vehicle components boundary domain, and an oversight server;receive the first message from the oversight server, wherein:the first message is associated with one of the autonomous vehicle components boundary domain, the vehicle components boundary domain, or a security domain; andthe security domain comprises a third set of components configured to facilitate authentication of received messages;determine a priority level associated with the first message, wherein the priority level associated with the first message indicates a scheduling requirement associated with the first message;identify a domain tag data associated with the first message, wherein the domain tag data indicates that the first message is associated with a particular domain from among the autonomous vehicle components boundary domain, the vehicle components boundary domain, or the security domain;identify a destination data associated with the first message, wherein the destination data indicates that the first message is designated to a particular component within the particular domain;schedule the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data; androute the first message to the particular component based at least in part upon the destination data.

Clause 2. The system of Clause 1, wherein the particular component is an internal software component with respect to the gateway processor.

Clause 3. The system of Clause 1, wherein the particular component is an external hardware component with respect to the gateway processor.

Clause 4. The system of Clause 1, wherein the priority level of the first message further indicates whether a priority of the first message is low, medium, or high.

Clause 5. The system of Clause 4, wherein scheduling the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data comprises moving the first message to a top of a scheduling queue comprising a plurality of messages associated with various priority levels in response to determining that the priority level associated with the first message is high, wherein the priority level is determined to be high when the priority level is more than a threshold value.

Clause 6. The system of Clause 4, wherein scheduling the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data comprises moving the first message to a scheduling queue dedicated for messages with high priority levels in response to determining that the priority level associated with the first message is high, wherein the priority level is determined to be high when the priority level is more than a threshold value.

Clause 7. The system of Clause 4, wherein scheduling the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data comprises moving the first message to a scheduling queue dedicated for messages with medium priority levels in response to determining that the priority level associated with the first message is medium, wherein the priority level is determined to be medium when the priority level is between a first threshold value and a second threshold value.

Clause 8. The system of Clause 4, wherein scheduling the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data comprises moving the first message to a scheduling queue dedicated for messages with low priority levels in response to determining that the priority level associated with the first message is low, wherein the priority level is determined to be low when the priority level is less than a threshold value.

Clause 9. A method comprising:

coordinating communication between an autonomous vehicle components boundary domain, a vehicle components boundary domain, and an oversight server;

receiving a first message from the oversight server, wherein:the first message is associated with one of the autonomous vehicle components boundary domain, the vehicle components boundary domain, or a security domain; andthe security domain comprises a third set of components configured to facilitate authentication of received messages;

determining a priority level associated with the first message, wherein the priority level associated with the first message indicates a scheduling requirement associated with the first message;

identifying a domain tag data associated with the first message, wherein the domain tag data indicates that the first message is associated with a particular domain from among the autonomous vehicle components boundary domain, the vehicle components boundary domain, or the security domain;

identifying a destination data associated with the first message, wherein the destination data indicates that the first message is designated to a particular component within the particular domain;

scheduling the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data;

and routing the first message to the particular component based at least in part upon the destination data.

Clause 10. The method of Clause 9, wherein:

the autonomous vehicle components boundary domain comprises a first set of components configured to facilitate autonomous operations of an autonomous vehicle;

the first set of components comprises at least one of:one or more autonomous drive compute units;a memory associated with the autonomous vehicle;a controller area network controller; ora timing synchronization component.

Clause 11. The method of Clause 9, wherein:

the vehicle components boundary domain comprises a second set of components configured to facilitate non-autonomous operations of an autonomous vehicle; and

the second set of components comprises at least one of:a modem;vehicle component controller;a human machine interface;a break unit;a power distribution unit;a camera array;a microphone array; ora speaker array.

Clause 12. The method of Clause 9, wherein the third set of components comprises at least one of:

an authentication software component; or

one or more security keys used to establish secured communication paths between any two combination of the autonomous vehicle components boundary domain, the vehicle components boundary domain, and the security domain.

Clause 13. The method of Clause 9, further comprising establishing a trusted communication path between the autonomous vehicle components boundary domain and the vehicle components boundary domain.

Clause 14. The method of Clause 13, wherein establishing the trusted communication path between the autonomous vehicle components boundary domain and the vehicle components boundary domain comprises:

receiving an initial private security key from the oversight server;

sharing the initial private security key with a communication processor associated with the vehicle components boundary domain;

receiving a request from the communication processor to communicate a second message, wherein the request comprises the second message and a private security key; and

determining that the received private security key corresponds to the initial private security key.

Clause 15. A non-transitory computer-readable medium storing instructions that when executed by a processor causes the processor to:

coordinate communications between an autonomous vehicle components boundary domain, a vehicle components boundary domain, and an oversight server;

receive a first message from the oversight server, wherein:the first message is associated with one of the autonomous vehicle components boundary domain, the vehicle components boundary domain, or a security domain; andthe security domain comprises a third set of components configured to facilitate authentication of received messages;

determine a priority level associated with the first message, wherein the priority level associated with the first message indicates a scheduling requirement associated with the first message;

identify a domain tag data associated with the first message, wherein the domain tag data indicates that the first message is associated with a particular domain from among the autonomous vehicle components boundary domain, the vehicle components boundary domain, or the security domain;

identify a destination data associated with the first message, wherein the destination data indicates that the first message is designated to a particular component within the particular domain;

schedule the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data; and

route the first message to the particular component based at least in part upon the destination data.

Clause 16. The non-transitory computer-readable medium of Clause 15, wherein scheduling the first message to be transmitted to the particular domain based at least in part upon the priority level associated with the first message and the identified domain tag data comprises:

determining that the particular domain is the security domain; and

in response to determining that the particular domain is the security domain, routing the first message to the security domain.

Clause 17. The non-transitory computer-readable medium of Clause 15, wherein determining the priority level associated with the first message comprises determining that the first message is associated with a priority level tag data that indicates the priority level associated with the first message.

Clause 18. The non-transitory computer-readable medium of Clause 15, wherein determining the priority level associated with the first message comprises determining that the first message is associated with a particular internet protocol (IP) address that is associated with the priority level associated with the first message.

Clause 19. The non-transitory computer-readable medium of Clause 15, wherein: in response to determining that the first message is associated with a first IP address, determine that the priority level associated with the first message is a high priority level;

in response to determining that the first message is associated with a second IP address, determine that the priority level associated with the first message is a medium priority level; and

in response to determining that the first message is associated with a third IP address, determine that the priority level associated with the first message is a low priority level.

Clause 20. The non-transitory computer-readable medium of Clause 15, wherein the first message comprises one of:

a command to engage the autonomous operations;

map data that comprises a virtual map of an area where an autonomous vehicle is traveling;

an autonomous software algorithm update; or

a minimal risk maneuver command that comprises instructions to pull the autonomous vehicle over or stop the autonomous vehicle.

Clause 21. The system of any of Clauses 1-8, wherein the processor is further configured to perform one or more operations of a method according to any of Clauses 9-14.

Clause 22. The system of any of Clauses 1-8, wherein the processor is further configured to perform one or more operations according to any of Clauses 15-20.

Clause 23. An apparatus comprising means for performing a method according to any of Clauses 9-14.

Clause 24. An apparatus comprising means for performing one or more instructions according to any of Clauses 15-20.

Clause 25. The non-transitory computer-readable medium of any of Clauses 15-20 storing instructions that when executed by the one or more processors further cause the one or more processors to perform one or more operations of a method according to any of Clauses 9-14 when run on a system.