Black Box Data Recorder with Artificial Intelligence Processor in Autonomous Driving Vehicle

A black box controller determines whether or not vehicle sensor data of an autonomous vehicle is to be recorded in a non-volatile storage device, based on a result from an artificial intelligence (AI) processor. For example, the AI processor analyzes the sensor data to detect an event of interest that is about to take place and/or has taken place, such as an imminent or impending collision or nearby collision involving the respective vehicle or other vehicle. In response to the AI processor detecting an event, the AI processor instructs the black box controller to store the vehicle sensor data to the non-volatile storage; and the sensor data collected for a time period leading to the event and/or during the event is selected and stored in the non-volatile storage. As a result, there are less writing operations to the non-volatile storage device that may have relatively limited endurance.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate to autonomous vehicle technology and more particularly, but not limited to, black box data recorders in autonomous vehicles.

BACKGROUND

Autonomous driving vehicles (ADV) typically include many sensors to perform autonomous/driverless operations. In the case of an accident, collision, or near collision involving such a vehicle, reviewing the sensor data recorded just prior to and/or during the accident may assist the determination of the cause of the accident, and/or whether there may have been a vehicle failure.

A storage device used in a black box for recording such data may has limited endurance. Thus, recording may be done in a low resolution, with a limited time of recording, a limited number of frames, and the data may be compressed.

A typical storage device does not contain a secure mechanism for the verification of the authenticity of the data stored in the storage device. As a result, the data stored on the black box storage device may be vulnerable to tempering.

DETAILED DESCRIPTION

At least some embodiments disclosed herein provide improved black box data recorders in autonomous driving vehicles (AVD). Automakers may want to record vehicle sensor data of autonomous vehicles. However, extended recordings of data in non-volatile storage can be expensive; and many non-volatile storage technologies have limited endurance. Embodiments described herein provide an improved solution for recording vehicle sensor data generated immediately prior to, and possibly during, an event of interest (e.g., collision or near collision involving the respective vehicle or nearby vehicle).

In one embodiment, a volatile memory is provided within a vehicle to temporarily hold vehicle sensors data. A black box controller determines when the vehicle sensor data in the volatile memory is to be recorded in a non-volatile storage device. In one embodiment, an artificial intelligence (AI) processor analyzes the sensor data stored in the memory to detect an event of interest (e.g., imminent or impending collision or nearby collision involving the respective vehicle or other vehicle) that is about to take place and/or has taken place. In response to the AI processor detecting an event of interest, the AI processor instructs the black box controller to store the sensor data from the volatile memory to the non-volatile storage. As a result, there are less writing operations to the non-volatile storage device that may have relatively limited endurance.

The techniques described herein can also be used in areas beyond AVDs, including vehicles using driver assistance technologies, video-surveillance, and monitoring of harsh environments.

FIG. 1illustrates an embodiment of an improved black box data recorder in an autonomous driving vehicle (AVD). InFIG. 1, a black box controller102receives a data stream of sensor data from multiple vehicle sensors104(e.g., surrounding cameras and other sensors). The vehicle sensor data may further include, but is not limited to camera data, radar data, lidar data, sonar data, laser measurements, tire pressure monitoring, and vehicle operation system data. The sensor data as referenced herein may also include vehicle operation data such as GPS data, inertial sensor data, autonomous vehicle computer signals and health status, etc.

The received data stream of the sensor data is initially held in a memory106. In one embodiment, the memory106is volatile, such as a dynamic random access memory (DRAM), which requires continual power in order to refresh or maintain the data in the memory.

Alternatively or in combination, the sensor data can be held in a cyclic buffer implemented with volatile memory. The vehicle sensor data stored in the volatile buffer would not be compressed, or in an alternative embodiment could be compressed using a lossless compression or a less lossy compression—to preserve data quality.

When using a non-volatile cyclic buffer, the buffer may have a relatively smaller capacity (because of its high cost per memory unit), and be used to buffer a smaller, lesser quality version of the sensor data (lossy compression or a more lossy compression) to reduce the size of the data.

The cyclic buffers as referenced herein may also be circular buffers, circular queues, or ring buffers that include a data structure that use a single, fixed-size buffer as if it were connected end-to-end. The buffer structure readily buffers data streams. The reference to cyclic buffers as used herein references how the buffers are used. For example, the cyclic buffers are designed to be full as new data overwrites old data in a cyclic manner, ensuring that the buffer holds the latest set of data. The actual size of the first and/or the second cyclic buffers can vary within the scope of the invention. The actual number of cyclic buffers provided within a black box recorder may also vary within the scope of the invention.

Movement/status sensors108of the ADV send signals to the Black Box Controller, when the movement/status sensors108detect a variation, or are triggered or activated. The movement/status sensors108may include one or more of inertia sensors, acceleration sensors, sudden activation of the braking system, failure of the engine or other components within the ADV, signals from the Advanced driver assistance systems (ADAS) or an autonomous computer that indicates an accident/collision or near collision. In response to the data from the sensors (104) and/or the movement/status sensors108, an artificial intelligence (AI) processor110analyzes the data stream of sensor data stored in the memory106to determine if an event of interest is about to happen and/or has occurred; and in response to the identification of the event based on the analysis of the AP processor110, the black box controller102transfers the sensor data to a non-volatile storage device112for storage. For example, the AI processor110analyzes the sensor data stored in the memory106to determine if the sensor data indicates and/or predicts a collision or near collision involving the respective vehicle or nearby vehicle is imminent or impending.

In response to the AI processor110determining the data stream of sensor data indicates such an event is about to occur and/or has occurred, the AI processor110signals the Black Box Controller102to have sensor data generated/received immediately prior to, and possibly during the event, copied from the system memory106to the non-volatile storage device112. With the data stream of sensor data relevant to the event stored in the non-volatile storage device112, the sensor data can be later retrieved in an analysis of a possible cause of the event.

By using the AI processor110to analyze the sensor data and thus selectively determine when sensor data should be stored in the non-volatile storage device112, endurance issues related to non-volatile storage are at least partially addressed by resulting in less write operations to the non-volatile storage device112. With less write operations, higher quality sensor data (e.g., increasing the resolution, the frame rate) can be stored in the storage device112.

In one embodiment, the AI processor110includes a neural network trained to understand and/or determine whether the sensor data indicates an imminent or impending event of interest, such as a collision, nearby collision, or impact involving the vehicle or a nearby vehicle, and whether the event warrants having the sensor data written in the non-volatile storage device112. For example, using machine learning techniques and/or pattern recognition techniques, an AI model can be established, e.g., from training examples and used by the AI processor110to perform some tasks, such as making a decision of whether or not to record the sensor data in the storage device112.

In one embodiment, the AI processor110utilizes an AI model (e.g., an artificial neural network) in making its decisions of whether to have the sensor data saved in the storage device112. The AI model can be created based on known decisions for prior sensor data. For example, sensor data may be used as input to train an AI model that has parameters (e.g., the weights of the neurons in the artificial neural network, the connectivity of the network, the response functions of the neurons); and the parameters are adjusted during the training so that decisions/predictions made using sensor data best matches with the known recording decisions. The AI model and the parameters as a whole can be used in the AI processor110to make the recording decisions (or predictions of events of interest) based on the real time sensor data.

For example, the sensor data used as input to train the AI model can be the recorded data from past road trips; and the known decisions may be obtained from human reviewers of the recorded data in identifying events of interest and/or the decisions to record data leading to the events of interest. For example, recorded camera and sensor data may be played back to a human operator, which indicates when the human operator would want to have stored the sensor data. For example, when a car is running in a test mode, a safety driver may generate real time input to indicate when to record (e.g., when the safety driver sees an event of interest for recording). Past recordings that lead to accidents can also be used (e.g., the sensor data that was seen before an actual accident).

In some instances, the AI processor may make decisions in real-time based on a set of rules. For example, the AI processor110may recognizes objects and in response use predetermined rules to make recording decisions, where the rules specifies what spatial and motion relations among the objects and the vehicle should trigger the recording of the sensor data in storage device112(and when not to save the sensor data from the memory (106) to the storage device112).

FIG. 2illustrates another embodiment of an improved black box data recorder in an autonomous driving vehicle (AVD), wherein the AI processor110is integrated within the non-volatile storage device112, providing a sub-system within the improved black box data recorder.

In some implementations, the storage device112ofFIG. 1orFIG. 2can not only record the sensor data, but also create authenticity verification data that can be used to verify the authenticity of the sensor data stored in the storage device112. For example, a secret key can be used to sign a hash of the stored sensor data to generate a signature. To verify whether the sensor data in the storage device112has been altered, the secret key can be applied to a hash of the current data stored in the storage device112to generate a current signature for comparison with the signature created at the time of the recordation of the sensor data. If the signatures do not agree with each other, the recorded sensor data has been altered.

In general, a symmetric cryptographic approach or an asymmetric cryptographic approach can be used to verify the authenticity of the sensor data stored in the storage device112.

The symmetric approach uses the same secret key for the creating of the signature and during the verification of the signature. The secret key can be shared among authorized parties in a factory environment and/or self-generated by using a physical unclonable function (PUF) of the storage device112in conjunction with a freshness mechanism and a Message Authenticated Code (MAC) algorithm. To verify the signature, an authorized party uses the same MAC algorithm with the secret key.

The asymmetric approach uses a pair of public key and private key of the storage device112, which key pair can be generated using a physical unclonable function (PUF) of the storage device112, in conjunction with a freshness mechanism and a digital signature algorithm to produce a signature. Examples of digital signature algorithms include the digital standard signature (DSS) or one of its variants like the Elliptic Curve Digital Signature Algorithm (ECDSA) as defined in a National Institute of Standards and Technology (NIST) standard (e.g., Federal Information Processing Standards (FIPS) 186-4). The storage device112uses the private key to create a digital signature on data set stored in the storage device112; and the authenticity of the digital signature can be verified using the public key.

Examples of the indicators used in the freshness mechanism include the result of a monotonic counter that is increased each time the signature output is generated, a time stamp, a nonce, etc.

Further details and examples of techniques to generate a unique key using PUF can be found in U.S. patent application Ser. No. 15/853,498, filed Dec. 22, 2017 and entitled “Physical Unclonable Function using Message Authentication Code”, the disclosure of which is hereby incorporated herein by reference.

FIG. 3illustrates a flow diagram of an improved black box data recorder in an autonomous driving vehicle (AVD). As described above, initially a data stream of sensor data is received302. The vehicle sensor data may include, but is not limited to camera data, radar data, lidar data, sonar data, laser measurements, tire pressure monitoring, and vehicle operation system data. The vehicle sensor data as referenced herein may also include vehicle operation data such as GPS data, inertial sensor data, autonomous vehicle computer signals and health status, etc.

The data stream of sensor data is analyzed by the AI processor (as trained via a neural network) to determine if a relevant event is imminent or impending304. For example, the AI processor110analyzes the sensor data stored in the memory106to determine if the sensor data indicates whether or not a collision or nearby collision is imminent/impending. The sensor data analyzed may include data from the movement/status sensors108, such as one or more of inertia sensors, acceleration sensors, sudden activation of the braking system, failure of the engine or other components within the ADV, signals from the Advanced driver assistance systems (ADAS) or an autonomous computer that indicates an accident/collision or near collision. In an alternative embodiment, the AI processor may analyze the data from the sensors104without the data from the movement/status sensors108.

If the AI processor determines that no relevant event is imminent or about to occur, no signal is sent to the black box controller306and/or the storage device112for the recording of recent sensor data from the sensors104and/or108. If the AI processor determines a relevant event is imminent (e.g., a collision or nearby collision is likely), the black box controller102and/or the storage device112receives a notification to store the data stream of sensor data in the non-volatile storage device112and write operations to the non-volatile storage device begin308.

In one embodiment, the volatile memory106could be implemented as memory of an onboard computer system installed in the vehicle, outside of a black box recorder that includes the storage device112. The black box recorder may be configured with ports to be connected to the onboard computer. The black box controller102may or may not be in the recorder. In one embodiment, a CPU or processor of an onboard computer may serve as the black box controller, and/or the AI processor110can be implemented via software on the CPU with hardware acceleration (e.g., using a graphical processing unit (GPU) or other AI specific hardware). In some implementations, the black box recorder for intelligent recording functions and the onboard computer for autonomous driving functions may share software and/or hardware in processing the sensor data, such as camera and/or video data from the sensors104.

The Non-volatile storage device112may be implemented via various techniques, such as memory cells in an integrated circuit. The storage media of the storage device112is non-volatile in that no power is required to maintain the data/information stored in the non-volatile storage media, which data/information can be retrieved after the non-volatile storage media is powered off and then powered on again. The memory cells may be implemented using various memory/storage technologies, such as NAND gate based flash memory, phase-change memory (PCM), magnetic memory (MRAM), resistive random-access memory, and 3D XPoint™, such that the storage device112is non-volatile and can retain data stored therein without power for days, months, and/or years.

In one embodiment, the storage device102can include cross point storage (e.g., 3D XPoint™ memory). A cross point memory device uses transistor-less memory elements, each of which has a memory cell and a selector that are stacked together as a column. Memory element columns are connected via two perpendicular lays of wires, where one lay is above the memory element columns and the other lay below the memory element columns. Each memory element can be individually selected at a cross point of one wire on each of the two layers. Cross point memory devices are fast and non-volatile and can be used as a unified memory pool for processing and storage.

The storage device112communications with the black box controller102may occur using a communication channel, which may use a PCIe protocol, NVMe protocol, or other communication protocols. The black box controller102and the storage device112can be configured to communicate with each other using data storage management and usage commands.

The black box controller102and/or a separate controller of the storage device112may run firmware to perform operations responsive to communications. Firmware in general is a type of computer program that provides control, monitoring and data manipulation of engineered computing devices. The firmware of the storage device or the black box controller may control the operations of operating the storage device, such as storing and accessing data, performing power management tasks, etc.

The memory106may use volatile Dynamic Random-Access Memory (DRAM) for the holding of data stream sensor data and possibly instructions used by the controller102to improve the computation performance of the controller102DRAM is volatile in that it requires power to maintain the data/information stored therein, which data/information is lost immediately or rapidly when the power is interrupted.

Volatile DRAM typically has less latency than non-volatile storage media, but loses its data quickly when power is removed. Thus, it is advantageous to use the volatile DRAM to temporarily store instructions and data used for the controller102in its current computing task to improve performance. In some instances, the volatile DRAM can be replaced with volatile Static Random-Access Memory (SRAM) that uses less power than DRAM in some applications.

Examples of computer-readable storage media include, but are not limited to, recordable and non-recordable type media such as volatile and non-volatile memory devices, read only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, magnetic disk storage media, and optical storage media (e.g., Compact Disk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.), among others. The instructions may be embodied in a transitory medium, such as electrical, optical, acoustical or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, etc. A transitory medium is typically used to transmit instructions, but not viewed as capable of storing the instructions.