DRIVING REPORT GENERATION USING A DEEP LEARNING DEVICE

Methods, systems, and devices for driving report generation using a deep learning device are described. In some cases, a vehicle may use sensor data and a deep learning device to provide a report to a driver of the vehicle. The vehicle may collect data from vehicle sensors and store a set of inputs received from the sensors in a volatile memory device. One or more processing units coupled with the memory system of the vehicle system may generate a model associated with the environment of the vehicle using the stored sensory inputs. The vehicle may identify events using the model, the sensory inputs or both. In some examples, the vehicle may employ a deep learning device to generate an event report using a machine learning model and the set of inputs.

TECHNICAL FIELD

The following relates to one or more systems for memory, including driving report generation using a deep learning device.

BACKGROUND

Memory devices are widely used to store information in various electronic devices such as computers, user devices, wireless communication devices, cameras, digital displays, and the like. Information is stored by programming memory cells within a memory device to various states. For example, binary memory cells may be programmed to one of two supported states, often corresponding to a logic 1 or a logic 0. In some examples, a single memory cell may support more than two possible states, any one of which may be stored by the memory cell. To access information stored by a memory device, a component may read (e.g., sense, detect, retrieve, identify, determine, evaluate) the state of one or more memory cells within the memory device. To store information, a component may write (e.g., program, set, assign) one or more memory cells within the memory device to corresponding states.

DETAILED DESCRIPTION

Some vehicle systems may implement an advanced driver assistance system (ADAS) to collect and share information to improve travel safety (e.g., to prevent collisions). As ADAS capability and availability increase. more driver behavior data may be collected. However, current vehicles may provide limited performance information to the driver associated with trip meta data and metrics. Additional information may be useful to the driver and others. For example, a new driver may benefit from specific feedback on driving decisions and suggestions for how to improve. In other examples, vehicle insurers may desire information about driver performance (e.g., for determining insurance rates). In other examples, taxi services and fleet coordinators may utilize reports on driver performance (e.g., for ensuring accountability for their drivers). In some examples, such as in response to a collision that occurs during a trip, trip information may be used by law enforcement in documenting the collision, determining fault or both. In these and other examples, more driver and trip information may be desirable.

In some implementations, a vehicle may use sensor data and a deep learning device to provide a report to a driver of the vehicle. For example, the vehicle may collect data from vehicle sensors and store a set of inputs received from the sensors in a volatile memory device. One or more processing units coupled with the memory system of the vehicle system may generate a model associated with the environment of the vehicle using the stored sensory inputs. The vehicle may identify events (e.g., a sharp turn, a stop sign, a collision, etc.) using the model, the sensory inputs or both. In some examples, the vehicle may employ a deep learning device to generate an event report using a machine learning model and the set of inputs. The deep leaming may be integrated with a memory system of the vehicle. In some examples, the vehicle may transmit the event report to an output device and store the event report in a non-volatile memory device of the memory system. In some examples, the deep learning device may provide feedback or suggestions during the trip (e.g., while the driver is driving), for example using a head up display (HUD). The driver or other interested party may use the generated report to improve driving safety and accountability.

Features of the disclosure are initially described in the context of systems, devices, and circuits with reference toFIG.1. Features of the disclosure are described in the context of a system and process flows with reference toFIGS.2through4. These and other features of the disclosure are further illustrated by and described in the context of an apparatus diagram and flowchart that relate to driving report generation using a deep learning device with reference toFIGS.5through6.

FIG.1illustrates an example of a system100that supports driving report generation using a deep learning device in accordance with examples as disclosed herein. The system100includes a host system105coupled with a memory system110.

The memory system controller115may also include a local memory120. In some cases, the local memory120may include read-only memory (ROM) or other memory that may store operating code (e.g., executable instructions) executable by the memory system controller115to perform functions ascribed herein to the memory system controller115. In some cases, the local memory120may additionally, or alternatively, include static random access memory (SRAM) or other memory that may be used by the memory system controller115for internal storage or calculations, for example, related to the functions ascribed herein to the memory system controller115. Additionally, or alternatively, the local memory120may serve as a cache for the memory system controller115. For example, data may be stored in the local memory120if read from or written to a memory device130, and the data may be available within the local memory120for subsequent retrieval for or manipulation (e.g., updating) by the host system105(e.g., with reduced latency relative to a memory device130) in accordance with a cache policy.

In some cases, planes165may refer to groups of blocks170, and in some cases, concurrent operations may be performed on different planes165. For example, concurrent operations may be performed on memory cells within different blocks170so long as the different blocks170are in different planes165. In some cases, an individual block170may be referred to as a physical block, and a virtual block180may refer to a group of blocks170within which concurrent operations may occur. For example, concurrent operations may be performed on blocks170-a,170-b,170-c, and170-dthat are within planes165-a,165-b,165-c, and165-d, respectively, and blocks170-a,170-b,170-c, and170-dmay be collectively referred to as a virtual block180. In some cases, a virtual block may include blocks170from different memory devices130(e.g., including blocks in one or more planes of memory device130-aand memory device130-b). In some cases, the blocks170within a virtual block may have the same block address within their respective planes165(e.g., block170-amay be “block 0” of plane165-a, block170-bmay be “block 0” of plane165-b, and so on). In some cases, performing concurrent operations in different planes165may be subject to one or more restrictions, such as concurrent operations being performed on memory cells within different pages175that have the same page address within their respective planes165(e.g., related to command decoding, page address decoding circuitry, or other circuitry being shared across planes165).

The system100may include any quantity of non-transitory computer readable media that support driving report generation using a deep learning device. For example, the host system105(e.g., a host system controller106), the memory system110(e.g., a memory system controller115), or a memory device130(e.g., a local controller135) may include or otherwise may access one or more non-transitory computer readable media storing instructions (e.g., firmware, logic, code) for performing the functions ascribed herein to the host system105, the memory system110, or a memory device130. For example, such instructions, if executed by the host system105(e.g., by a host system controller106), by the memory system110(e.g., by a memory system controller115), or by a memory device130(e.g., by a local controller135), may cause the host system105, the memory system110, or the memory device130to perform associated functions as described herein.

In some cases, a vehicle may implement the system100, which may use sensor data and a deep learning device to provide a report to a driver of the vehicle. For example, the vehicle may collect data from vehicle sensors and store a set of inputs received from the sensors in a volatile memory device. One or more processing units coupled with the memory system of the vehicle system may generate a model associated with the environment of the vehicle using the stored sensory inputs. The vehicle may identify events (e.g., a sharp turn, a stop sign, a collision, etc.) using the model, the sensory inputs or both. In some examples, the vehicle may employ a deep learning device to generate an event report using a machine learning model and the set of inputs. In some examples, the vehicle may transmit the event report to an output device and store the event report in a non-volatile memory device of the memory system. In some examples. the deep learning device may provide feedback or suggestions during the trip (e.g., while the driver is driving), for example using a head up display (HUD). The driver or other interested party may use the generated report to improve driving safety and accountability.

FIG.2illustrates an example of a system200that supports driving report generation using a deep learning device in accordance with examples as disclosed herein. The system200may implement or be implemented by aspects of the system100described with reference toFIG.1. For example, the system200may depict operation of a zonal computing system of a vehicle205that includes various components, such as central processors210, gateway processors215, memory systems, devices230, and DLAs235which may be examples of corresponding devices described with reference toFIG.1. In some cases, devices230may include devices such as sensors or actuators for the system200. Additionally, the system200may support the utilization of machine learning processes via the DLAs235to enable tasks such as autonomous driving, machine vision, voice recognition, and natural language processing, among other tasks.

The vehicle205may implement a zonal computing system to manage various devices that may be included in the vehicle205. For example, the vehicle205may include a zonal computing system in which different groups of components of the vehicle205are divided into various zones and managed in accordance with the zones. The zonal computing system may include one or more central processors210that are configured to communicate with a remote server225. For example, the zonal computing system may include a central processor210-aand a central processor210-bthat may each be configured to communicate with the remote server225. In some examples, the remote server225may provide the vehicle205access to a network, and the vehicle205may receive data from the network via the remote server225. In some examples, the remote server225may be an example of a cloud server. The central processors210may communicate with the remote server225wirelessly, for example, using one or more antennas of the vehicle205in accordance with one or more radio access technologies.

The central processors210may additionally be configured to communicate with various zones of the zonal computing system. For example, the zonal computing system may include: gateway processors215; and devices230, which may include actuators that are configured to control (e.g., trigger, cause, or perform actions with) a subsystem of the vehicle205or sensors that are configured to measure a physical property associated with the vehicle205or an environment associated with the vehicle205(e.g., a motion sensor, a camera, a radar sensor, a speedometer, a gas meter, a fuel temperature sensor, an oxygen sensor, a light detection and ranging (LIDAR) sensor, or some other sensor that may be included in the vehicle205); among other computing components that may be included in the zonal computing system. Each of the gateway processors215and devices230may be associated with a respective zone of the zonal computing system. The gateway processors215may be coupled with at least one of the central processors210(e.g., directly or via one or more other gateway processors215) and with one or more devices230, or a combination thereof. Additionally, the gateway processors215may be configured to route communications between the at least one central processor210and the respective devices230with which the gateway processors215are coupled. Accordingly, the central processors210may be configured to communicate with devices230of a zone via one or more gateway processors215associated with the zone. In some examples, a zone may include a communication path coupled with one or more gateway processors215.

In the example ofFIG.2, the zonal computing system may include a gateway processor215-a, a gateway processor215-b, a gateway processor215-c, a gateway processor215-d, a gateway processor215-e, a gateway processor215-f, and a gateway processor215-g, although any quantity of gateway processors215may be included in the zonal computing system of the vehicle205. In some examples, each gateway processor215may be associated with a different zone of the zonal computing system. For example, the gateway processor215-amay be associated with a first zone of the zonal computing system, the gateway processor215-bmay be associated with a second zone of the zonal computing system, the gateway processor215-cmay be associated with a third zone of the zonal computing system, and so on. In other examples, multiple gateway processors215may be associated a single zone of the zonal computing system. For example, the gateway processor215-a, the gateway processor215-b, and the gateway processors215-cmay be associated with the first zone; the gateway processor215-dand the gateway processor215-emay be associated with the second zone, and the gateway processor215-fand the gateway processor215-gmay be associated with the third zone. In some examples, the central processors210may be coupled with one or more devices230. For example, the central processor210-amay be coupled with a device230-h, and the central processor210-bmay be coupled with a device230-i. In some examples, the device230-hand230-imay be associated with different zones of the zonal computing system or with one of the zones with which the central processors210are configured to communicate (e.g., one of the zones associated with a gateway processor215).

The devices230included in the vehicle205may be associated with the respective zones of the gateway processors215with which they are coupled. For example, in the example ofFIG.2, the gateway processor215-amay be coupled with a device230-a, the gateway processor215-bmay be coupled with a device230-b, the gateway processor215-cmay be coupled with a device230-c, the gateway processor215-dmay be coupled with a device230-d, the gateway processor215-emay be coupled with a device230-e, the gateway processor215-fmay be coupled with a device230-f, and the gateway processor215-gmay be coupled with a device230-g. Each of the devices230-athrough230-gmay be associated with (e.g., included in) the zone with which the corresponding gateway processor215is associated (e.g., the first zone through a seventh zone, respectively).

The components of the zonal computing system may communicate according to various communication protocols. For example, the central processors210and the gateway processors215may be coupled over various signal buses240that operate according to a first communication protocol. For instance, the central processor210-aand the central processor210-bmay communicate over a signal bus240-a. The central processor210-bmay communicate with the gateway processor215-a, the gateway processor215-b, and the gateway processor215-cover a signal bus240-b. The central processor210-amay communicate with the gateway processor215-d, the gateway processor215-e, the gateway processor215-f, and the gateway processor215-gover a signal bus240-c. In some examples, the central processors210may communicate with the gateway processors215directly or indirectly over the signal buses240. For example. the central processor210-bmay be directly coupled with the gateway processor215-aand the gateway processor215-cover the signal bus240-band indirectly coupled with the gateway processor215-bover the signal bus240-bvia the gateway processor215-a, the gateway processor215-c, or both. Thus, communications between the central processor210-band the gateway processor215-bmay be routed through the gateway processor215-a, the gateway processor215-c, or both. Additionally, the central processor210-amay be directly coupled with the gateway processor215-dand the gateway processor215-eover the signal bus240-cand indirectly coupled with the gateway processor215-fand the gateway processor215-gover the signal bus240-c. In some examples, the signal buses240may be examples of ethernet cables and the first communication protocol may be an ethernet communication protocol according to which the central processors210and the gateway processors215may communicate.

Additionally, the devices230may be coupled with respective gateway processors215or central processors210over various signal buses245that operate according to one or more different communication protocols. In some examples, the one or more different communication protocols may be lower capacity or bandwidth communication protocols with respect to the first communication protocol, such as a serial communication protocol. The gateway processors215may be configured to translate information between the first communication protocol (e.g., used to communicate information between the gateway processors215and the central processors210) and the one or more different communication protocols (e.g., used between the gateway processors215and the devices230). For example, the gateway processor215-amay translate information that is communicated from the central processor210-bto the device230-afrom the first communication protocol to a second communication protocol. Additionally, the gateway processor215-amay translate information that is communicated from the device230-ato the central processor210-bfrom the second communication protocol to the first communication protocol. As such, the central processors210may communicate information with the devices230to control various operations and functions of the vehicle205(e.g., such as operations related to autonomous driving, alert notifications, etc.).

The zonal computing system of the vehicle205may include one or more DLAs235configured to perform operations of the components of the zonal computing system by utilizing one or more neural networks. In some cases, the use of neural networks may help to reduce power consumption and reduce latency, among other performance operations. The DLAs235may include machine learning processes and other advanced computing techniques that may be utilized by the components of the zonal computing system. For example, a processor of the vehicle205(e.g., a central processor210, a gateway processor215) may transmit information to a DLA235, which the DLA235may use as input into one or more neural networks. The DLA235may transmit responsive information to the processor that is output by the one or more neural networks based on the information received from the processor. For instance, the processor may transmit information gathered from one or more devices230to the DLA235, and the DLA235may input the information into one or more neural networks, for example, for the purposes of supporting data analytics or autonomous driving, among other operations of the vehicle205supported by the processor. The DLA235may transmit outputs of the one or neural networks to the processor, which the processor may use in performing, for example, the data analytics, autonomous driving, etc.

A DLA235may be included in (e.g., embedded in) or coupled with a central processor210or a gateway processor215. For example, in the example ofFIG.2, the central processor210-amay be coupled with a DLA235-a, the central processor210-bmay include (e.g., be embedded with) a DLA235-b, the gateway processor215-amay be coupled with a DLA235-c, the gateway processor215-emay include a DLA235-d, or a combination thereof. It is noted, however, thatFIG.2depicts an example configuration of DLAs235within the vehicle205and that any combination of components of the zonal computing system may include or be coupled with a respective DLA235.

A memory system may be included in (e.g., embedded in) or coupled with a central processor210or a gateway processor215and coupled with a DLA235. For example, in the example ofFIG.2, a first memory system may be coupled with the central processor210-aand the DLA235-a, a second memory system may be included (e.g., embedded) in the central processor210-band coupled with the DLA235-b, a third memory system may be coupled with the gateway processor215-aand the DLA235-c, a fourth memory system may be included in the gateway processor215-eand coupled with the DLA235-d, or a combination thereof. It is noted, however, thatFIG.2depicts an example configuration of memory systems within the vehicle205and that any combination of components of the zonal computing system may include or be coupled with a respective memory system.

In some cases, a DLA235may be directly coupled with a non-volatile memory system, a volatile memory system, or both within a single package. For example, a DLA235may be arranged or implemented on a first die, and a volatile memory system, such as a DRAM device, may be arranged or implemented on a second die. The first die and the second die may be included in a same package, which may support a high-bandwidth connection between the DLA235and the volatile memory system. Accordingly, the DLA235may use the volatile memory system as a buffer (e.g., a cycle buffer) as part of performing machine-learning processes. Additionally or alternatively, the DLA235may include on-board volatile memory, such as SRAM or other memory types which may be implemented in a same die as the DLA235. Accordingly, the DLA235may be included in a single die package. In such cases, the DLA235may use the on-board memory as a buffer (e.g., a cycle buffer) as part of performing machine-learning processes.

In some cases, the DLA235may directly couple with a non-volatile memory system, such as a NAND device. For example, the non-volatile memory device may be positioned above or below the DLA235in a vertical to support hybrid bonding between the non-volatile memory system and the DLA235. Such bonding of the DLA235with the non-volatile memory system may support high-bandwidth data transmission between the DLA235and the non-volatile memory system.

In some cases, the system may use the devices230to measure or record various environmental factors or events. For example, the devices230may include or may be an example of cameras (e.g., rear view cameras, side view cameras) which may record one or more video streams of the environment in the vicinity of the vehicle205, such as videos streams of the rear of the vehicle205. Additionally, the vehicle205may include one or more LiDAR sensors, which may be configured to detect objects and determine a distance to the objects (e.g., a distance between the vehicle205and a detected object). The vehicle205may also include one or more radar sensors, which may be configured to detect objects, determine distances to detects objects, determine velocity of detected objects, or any combination thereof. The vehicle205may also include one or more sound navigation and ranging (sonar) sensors, which may be configured to use ultrasonic sound waves to detect positions of one or more objects (e.g., relative to the vehicle205). The vehicle205may include one or more inertial measurement units (IMUs), which may measure and report forces experienced by the vehicle205.

The system200may use video streams and, in some cases, inputs from additional sensors (e.g., speedometers, thermometers, engine monitoring sensors, weather monitoring sensors) to detect events and generate associated reports. For example, the devices230(e.g., cameras, LiDAR, radar, sonar, and other sensors) may provide the video streams and additional inputs to the one or more DLAs235or other computing platforms, such as one or more central processors210, of the vehicle205.

The one or more DLAs235, one or more central processors210, or both may process the video streams and additional inputs to identify sections of a trip (e.g., critical driving sections of a session of operating the vehicle205) and provide a report to a driver of the vehicle205. For example, the vehicle205may collect data from devices230and store a set of inputs received from the device230in a volatile memory device. One or more processing units, such as one or more central processors210, one or more gateway processor215, one or more DLAs235, or a combination thereof coupled with the memory system of the vehicle205system may generate a model associated with the environment of the vehicle205using the stored sensory inputs. The vehicle205may identify events (e.g., a sharp turn, a stop sign, a collision, etc.) using the model, the sensory inputs or both. In some examples, the vehicle205may employ a deep learning device (e.g., a DLA235) to generate an event report using a machine learning model and the set of inputs. In some examples, the vehicle205may transmit the event report to an output device and store the event report in a non-volatile memory device of the memory system. In some examples. the deep learning device may provide feedback or suggestions during the trip (e.g., while the driver is driving), for example using a head up display (HUD). The driver or other interested party may use the generated report to improve driving safety and accountability.

FIG.3illustrates an example of a process flow300that supports driving report generation using a deep learning device in accordance with examples as disclosed herein. The process flow300may implement or be implemented to realize aspects of the system100or the system200. For example, aspects of a vehicle (e.g., a vehicle205), such as one or more central processors210, one or more gateway processors215, one or more devices230, one or more DLAs235, or a combination thereof may implement the process flow300. In the following description of process flow300, the operations may be performed in a different order than the order shown. For example, specific operations may also be left out of process flow300, or other operations may be added to process flow300.

The process flow300may illustrate a process to generate an event report of a section of a trip of the vehicle (e.g., a critical driving section). For example, at305, a memory system of the vehicle may receive a set of inputs from one or more sensors of the vehicle. The one or more sensors may include one or more cameras, one or more LIDAR sensors, one or more radar sensors, one or more sonar sensors, an IMU, a speedometer, an accelerometer, one or more infrared light detectors, or any combination thereof. The set of inputs may include a video stream captured by the one or more cameras, distance information associated with one or more objects included in the video stream (e.g., distances from the vehicle to other nearby vehicles), location information associated with the one or more objects, a speed of the vehicle, a respective speed of one or more of the one or more objects, an acceleration of the vehicle, infrared light information associated with an environment of the vehicle, or any combination thereof. The memory system may store the set of inputs in a memory device, such as a volatile memory device, in response to receiving the set of inputs.

At310, one or more processing units of the vehicle (e.g., one or more central processors210) may generate a model of the physical environment in the vicinity of the vehicle using the set of inputs. The model may include a physical representation of the vehicle with respect to the environment (e.g., a wireframe, a point-cloud, or a combination thereof). The model may also capture various forces experienced by the vehicle or passengers in the vehicle. For example, the model may use inputs from the IMU to model forces experienced by the vehicle. Generating the model may include identifying objects in the environment of the vehicle205. For example, the one or more processing units may identify one or more second vehicles included in a video stream included in the set of inputs (e.g., using machine vision). In some examples, the one or more processing units may identify respective speeds of the one or more second vehicles using parameters, such as LiDAR information, radar information, sonar information, or a combination thereof included in the set of inputs.

At315, the one or more processing units may identify an event associated with the vehicle using the set of inputs received by the memory system at305, the model generated at310, or both. In some examples, the one or more processing units may flag sections of the trip (e.g., critical driving sections of a session of operating the vehicle205). In some cases, the event may include the vehicle transitioning between a first lane and a second lane included in a video stream (e.g., a lane change), executing a turn, or a second vehicle being within a threshold distance of the vehicle. Identifying the event may include determining, using the set of inputs, whether the vehicle is transitioning between a first lane and a second lane, whether the vehicle is executing turn, or whether a second vehicle is within a threshold distance of the vehicle. In some examples, the one or more processing units may share inputs from the set of inputs. For example, a first processing unit may transmit data from a first sensor, such as a video stream, to a second compute unit associated with a second sensor, such as a LIDAR, to associated a distance to an object detected by the LiDAR to a vehicle included in the video stream. In some examples, the one or more processing units may communicate using an optical interface, such as a photonic interconnect.

In some examples, a deep leaming device, such as a DLA235, may generate an indication of a first action associated with the event. For example, the deep learning device may, using one or more machine learning models, determine a driving action (e.g., modify speed, change direction) using an autonomous driving model. Because such an action may correspond to an action which would be taken by the vehicle if the autonomous driving model were operating the vehicle, the action may be referred to as a recommended action. In some examples, a deep learning device may generate an indication of the recommended action associated with the event and display the recommended action on a display component of the vehicle, such as a head-up display (HUD). In some examples, the DLA235may be incorporated with a memory system. In such examples, the DLA235may analyze data as it is stored in a volatile memory device or a non-volatile memory device of the memory system. The close-coupling of the DLA235with the memory system may enable the DLA235to analyze the sensor data with greater efficiency because of reductions to path lengths the data may take.

At320, the memory system may generate an event report associated with the event using a machine learning model and the set of inputs received by the memory system at305. For example, the machine learning model may be one or more neural networks and the deep learning device may be or include one or more DLAs235. In some examples. the event report may include an evaluation of a second action executed by the vehicle205using the indication of the first action associated with the event. For example, the memory system may compare the action taken by the vehicle with the recommended action, and determine a score based on how closely the action taken by the vehicle matches the recommended action. In some examples, the event report may include the indication of the recommended action associated with the event generated by the machine learning model, a comparison between the action taken by the vehicle and the recommended action, a comparison between the action taken by the vehicle and a similar action taken by another driver, or any combination thereof. The event report may also include video feeds from the trip (e.g., from critical driving sections of a session of operating the vehicle205) and instructions on how the driver may improve their driving.

At325, the memory system may transmit the event report to an output device associated with the vehicle. For example, the memory system may transmit the report to a remote server (e.g., a cloud server) associated with the vehicle. Additionally, the memory system may store the event report in a non-volatile memory device of the vehicle.

Aspects of the process flow300may be implemented by a controller, among other components. Additionally or alternatively, aspects of the process flow300may be implemented as instructions stored in memory (e.g., firmware stored in a memory coupled with the host system or the memory system). For example, the instructions, when executed by a controller, may cause the controller to perform the operations of the process flow300.

FIG.4illustrates an example of a process flow400that supports driving report generation using a deep learning device in accordance with examples as disclosed herein. The process flow400may implement or be implemented to realize aspects of the system100or the system200. For example, aspects of a vehicle (e.g., a vehicle205), such as one or more central processors210, one or more gateway processors215, one or more devices230, one or more DLAs235, or a combination thereof may implement the process flow400. In the following description of process flow400, the operations may be performed in a different order than the order shown. For example, specific operations may also be left out of process flow400, or other operations may be added to process flow400.

The process flow400may illustrate a process to generate a report of an accident associated with the vehicle (e.g., a collision) and securely store or provide the report to a driver of the vehicle or other entities, such as law enforcement, insurance companies, or both. The entities may use the report to obtain a summary of causes or other aspects of the accident. For example, at405, a memory system of the vehicle may receive a set of inputs from one or more sensors of the vehicle. The one or more sensors may include one or more cameras, one or more light detection and ranging sensors, one or more radar sensors, one or more sonar sensors, an inertial measurement unit, a speedometer, an accelerometer, one or more infrared light detectors, or any combination thereof. The set of inputs may include a video stream, distance information associated with one or more objects included in the video stream, location information associated with the one or more objects, a speed of the vehicle, a respective speed of one or more of the one or more objects, an acceleration of the vehicle, infrared light information associated with an environment of the vehicle. or any combination thereof.

The memory system may, at410process the input data to generate a model of the physical environment in the vicinity of the vehicle. The model may include a physical representation of the vehicle with respect to the environment (e.g., a wireframe, a point-cloud, or a combination thereof). The model may also capture various forces experienced by the vehicle or passengers in the vehicle. For example, the model may use inputs from the IMU to model forces experienced by the vehicle. Generating the model may include identifying objects in the environment of the vehicle205. For example, the one or more processing units may identify one or more second vehicles included in a video stream included in the set of inputs (e.g., using machine vision). In some examples, the one or more processing units may identify respective speeds of the one or more second vehicles using parameters, such as LiDAR information, radar information, sonar information, or a combination thereof included in the set of inputs. In some examples, the memory system may process the data prior to storing the inputs in a non-volatile memory device at415(e.g., may process the data stored in a volatile memory device, such as a large-cycle buffer).

Additionally or alternatively, the memory system may process the data after storing the inputs in a non-volatile memory device. For example, at420, hardware accelerators may compress the data for uploading to a cloud service via a dedicated high-speed internet connection. Compressed data uploaded to a cloud service may allow users to access data in cases where the vehicle sustains severe damage (e.g., in a severe accident).

At425, the one or more processing units may identify an event associated with the vehicle using the set of inputs, the model generated at410, the model generated at420, or a combination thereof. For example, the one or more processing units may identify a collision or a near-collision with another vehicle. In some cases, identifying the collision may include determining whether a second vehicle is within a threshold distance of the vehicle. In some examples, the one or more processing units may share inputs from the set of inputs. For example, a first processing unit may transmit data from a first sensor, such as a video stream, to a second compute unit associated with a second sensor, such as a LiDAR, to associated a distance to an object detected by the LiDAR to a vehicle included in the video stream. In some examples, the one or more processing units may communicate using an optical interface, such as a photonic interconnect.

At430, the memory system may identify and extract metadata associated with the event. For example, the memory system may identify: a speed of the vehicle; speeds of other nearby vehicles; forces experienced by the vehicle (e.g., using and IMU); environmental conditions (e.g., weather conditions, road conditions); or a combination thereof. In some examples, the memory system may generate a report that may include the extracted metadata.

At435, the memory system may generate an interactive event report. For example, the memory system may use the model generated at410, at420, or both to create a visual, three-dimensional representation of the event (e.g., a representation of the vehicle and the environment surrounding the vehicle). The interactive event report may allow a user to examine a physical representation of the event from one or more angles or from one or more magnifications.

At440, the one or more processing units may encrypt the report generated at435, for example using one or more encryption keys. At445, the one or more processing units may provide an encryption key of the one or more encryption keys, to the user. For example, the encryption key may be a secure password shared with the user. In some examples, the interactive event report generated at435may also be encrypted. The one or more reports may include the metadata extracted at430.

At450, the one or more reports (e.g., the interactive report, the encrypted report, or both) may be transmitted to the user. For example, the one or more reports may be transmitted over a mobile or web app. In some examples, the one or more reports may be optionally shared to a secure cloud server or other remote server associated with the vehicle In some examples, the one or more reports may be transmitted and stored without user intervention such that authorities may be granted access to unaltered or untampered data (e.g., in the event of an accident).

Aspects of the process flow400may be implemented by a controller, among other components. Additionally or alternatively, aspects of the process flow400may be implemented as instructions stored in memory (e.g., firmware stored in a memory coupled with the host system or the memory system). For example, the instructions, when executed by a controller, may cause the controller to perform the operations of the process flow400.

FIG.5illustrates a block diagram500of a memory device520that supports driving report generation using a deep learning device in accordance with examples as disclosed herein. The memory device520may be an example of aspects of a memory device as described with reference toFIGS.1through4. The memory device520, or various components thereof, may be an example of means for performing various aspects of driving report generation using a deep learning device as described herein. For example, the memory device520may include a sensor reception component525, a storage component530, an event identification component535, an event report generation component540, a transmission component545, a model component550, or any combination thereof. Each of these components may communicate, directly or indirectly, with one another (e.g., via one or more buses).

The sensor reception component525may be configured as or otherwise support a means for receiving, at a memory system of a vehicle, a set of inputs from one or more sensors of the vehicle. The storage component530may be configured as or otherwise support a means for storing the set of inputs in a volatile memory device of the memory system based at least in part on receiving the set of inputs. The event identification component535may be configured as or otherwise support a means for identifying, by one or more processing units of the vehicle, an event associated with the vehicle based at least in part on the set of inputs. The event report generation component540may be configured as or otherwise support a means for generating, by a deep learning device directly coupled with the memory system of the vehicle, an event report associated with the event, the deep learning device for performing one or more operations using a machine leaming model and the set of inputs. The transmission component545may be configured as or otherwise support a means for transmitting the event report to an output device associated with the vehicle. In some examples, the storage component530may be configured as or otherwise support a means for storing the event report in a non-volatile memory device of the memory system.

In some examples, the model component550may be configured as or otherwise support a means for generating, by the one or more processing units, a model associated with an environment of the vehicle using the set of inputs, where identifying the event is further based at least in part on the model.

In some examples, the event report generation component540may be configured as or otherwise support a means for generating, by the deep learning device, an indication of a first action associated with the event. In some examples, the event report generation component540may be configured as or otherwise support a means for determining an evaluation of a second action executed by the vehicle based at least in part on the indication of the first action, where the event report includes the evaluation.

In some examples, the event report generation component540may be configured as or otherwise support a means for generating, by the machine learning model, an indication of a recommended action associated with the event. In some examples, the transmission component545may be configured as or otherwise support a means for transmitting the indication of the recommended action to a display component of the vehicle.

In some examples, to support generating the model, the sensor reception component525may be configured as or otherwise support a means for identifying one or more second vehicles included in a video stream, the video stream included in the set of inputs. In some examples, to support generating the model, the sensor reception component525may be configured as or otherwise support a means for identifying respective speeds of the one or more second vehicles using one or more parameters included in the set of inputs.

In some examples, the model component550may be configured as or otherwise support a means for identifying metadata of a model associated with an environment of the vehicle based at least in part on identifying the event. In some examples, the transmission component545may be configured as or otherwise support a means for transmitting the metadata to a remote server associated with the vehicle.

In some examples, the model component550may be configured as or otherwise support a means for encrypting the metadata using the one or more processing units, where transmitting the metadata includes transmitting the encrypted metadata to the remote server.

In some examples, to support identifying the event, the sensor reception component525may be configured as or otherwise support a means for determining, based at least in part on the set of inputs, whether a second vehicle is within a threshold distance of the vehicle.

In some examples, to support identifying the event, the sensor reception component525may be configured as or otherwise support a means for determining, based at least in part on the set of inputs, whether the vehicle is transitioning between a first lane and a second lane included in a video stream of the set of inputs.

In some examples, to support identifying the event, the sensor reception component525may be configured as or otherwise support a means for determining, based at least in part on the set of inputs, whether the vehicle is executing a turn.

In some examples, the transmission component545may be configured as or otherwise support a means for transmitting the event report to a remote server associated with the vehicle.

In some examples, the transmission component545may be configured as or otherwise support a means for transmitting data associated with the set of inputs from a first processing unit of the one or more processing units to a second processing unit of the one or more processing units using an optical interconnect between the first processing unit and the second processing unit, where identifying the event is based at least in part on transmitting the data.

In some examples, the one or more sensors include one or more cameras, one or more light detection and ranging sensors, one or more radar sensors, one or more sonar sensors, an inertial measurement unit, a speedometer, an accelerometer, one or more infrared light detectors, or a combination thereof, and the set of inputs include a video stream, distance information associated with one or more objects included in the video stream, location information associated with the one or more objects, a speed of the vehicle, a respective speed of one or more of the one or more objects, an acceleration of the vehicle, infrared light information associated with an environment of the vehicle, or a combination thereof.

FIG.6illustrates a flowchart showing a method600that supports driving report generation using a deep learning device in accordance with examples as disclosed herein. The operations of method600may be implemented by a memory device or its components as described herein. For example, the operations of method600may be performed by a memory device as described with reference toFIGS.1through5. In some examples, a memory device may execute a set of instructions to control the functional elements of the device to perform the described functions. Additionally, or alternatively, the memory device may perform aspects of the described functions using special-purpose hardware.

At605, the method may include receiving, at a memory system of a vehicle, a set of inputs from one or more sensors of the vehicle. The operations of605may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of605may be performed by a sensor reception component525as described with reference toFIG.5.

At610, the method may include storing the set of inputs in a volatile memory device of the memory system based at least in part on receiving the set of inputs. The operations of610may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of610may be performed by a storage component530as described with reference toFIG.5.

At615, the method may include identifying. by one or more processing units of the vehicle, an event associated with the vehicle based at least in part on the set of inputs. The operations of615may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of615may be performed by an event identification component535as described with reference toFIG.5.

At620, the method may include generating, by a deep learning device directly coupled with the memory system of the vehicle, an event report associated with the event, the deep learning device for performing one or more operations using a machine leaming model and the set of inputs. The operations of620may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of620may be performed by an event report generation component540as described with reference toFIG.5.

At625, the method may include transmitting the event report to an output device associated with the vehicle. The operations of625may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of625may be performed by a transmission component545as described with reference toFIG.5.

At630, the method may include storing the event report in a non-volatile memory device of the memory system. The operations of630may be performed in accordance with examples as disclosed herein. In some examples, aspects of the operations of630may be performed by a storage component530as described with reference toFIG.5.

Aspect 1: A method, apparatus, or non-transitory computer-readable medium including operations, features, circuitry, logic, means, or instructions, or any combination thereof for receiving, at a memory system of a vehicle, a set of inputs from one or more sensors of the vehicle; storing the set of inputs in a volatile memory device of the memory system based at least in part on receiving the set of inputs; identifying, by one or more processing units of the vehicle, an event associated with the vehicle based at least in part on the set of inputs; generating, by a deep learning device directly coupled with the memory system of the vehicle, an event report associated with the event, the deep learning device for performing one or more operations using a machine leaming model and the set of inputs; transmitting the event report to an output device associated with the vehicle; and storing the event report in a non-volatile memory device of the memory system.

Aspect 2: The method, apparatus, or non-transitory computer-readable medium of aspect 1, further including operations, features, circuitry, logic, means, or instructions, or any combination thereof for generating, by the one or more processing units, a model associated with an environment of the vehicle using the set of inputs, where identifying the event is further based at least in part on the model.

Aspect 3: The method, apparatus, or non-transitory computer-readable medium of aspect 2, further including operations, features, circuitry, logic, means, or instructions, or any combination thereof for generating, by the deep learning device, an indication of a first action associated with the event and determining an evaluation of a second action executed by the vehicle based at least in part on the indication of the first action, where the event report includes the evaluation.

Aspect 4: The method, apparatus, or non-transitory computer-readable medium of any of aspects 2 through 3, further including operations, features, circuitry, logic, means, or instructions, or any combination thereof for generating, by the machine learning model, an indication of a recommended action associated with the event and transmitting the indication of the recommended action to a display component of the vehicle.

Aspect 5: The method, apparatus, or non-transitory computer-readable medium of any of aspects 2 through 4, where generating the model includes operations, features, circuitry, logic, means, or instructions, or any combination thereof for identifying one or more second vehicles included in a video stream, the video stream included in the set of inputs and identifying respective speeds of the one or more second vehicles using one or more parameters included in the set of inputs.

Aspect 6: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 5, further including operations, features, circuitry, logic, means, or instructions, or any combination thereof for identifying metadata of a model associated with an environment of the vehicle based at least in part on identifying the event and transmitting the metadata to a remote server associated with the vehicle.

Aspect 7: The method, apparatus, or non-transitory computer-readable medium of aspect 6, further including operations, features, circuitry, logic, means, or instructions, or any combination thereof for encrypting the metadata using the one or more processing units, where transmitting the metadata includes transmitting the encrypted metadata to the remote server.

Aspect 8: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 7, where identifying the event includes operations, features, circuitry, logic, means, or instructions, or any combination thereof for determining, based at least in part on the set of inputs, whether a second vehicle is within a threshold distance of the vehicle.

Aspect 9: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 8, where identifying the event includes operations, features, circuitry, logic, means, or instructions, or any combination thereof for determining, based at least in part on the set of inputs, whether the vehicle is transitioning between a first lane and a second lane included in a video stream of the set of inputs.

Aspect 10: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 9, where identifying the event includes operations, features, circuitry, logic, means, or instructions, or any combination thereof for determining, based at least in part on the set of inputs, whether the vehicle is executing a turn.

Aspect 11: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 10, further including operations. features, circuitry, logic, means, or instructions, or any combination thereof for transmitting the event report to a remote server associated with the vehicle.

Aspect 12: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 11, further including operations, features, circuitry, logic, means, or instructions, or any combination thereof for transmitting data associated with the set of inputs from a first processing unit of the one or more processing units to a second processing unit of the one or more processing units using an optical interconnect between the first processing unit and the second processing unit, where identifying the event is based at least in part on transmitting the data.

Aspect 13: The method, apparatus, or non-transitory computer-readable medium of any of aspects 1 through 12, where the one or more sensors include one or more cameras, one or more light detection and ranging sensors, one or more radar sensors, one or more sonar sensors, an inertial measurement unit, a speedometer, an accelerometer, one or more infrared light detectors, or a combination thereof, and the set of inputs include a video stream, distance information associated with one or more objects included in the video stream, location information associated with the one or more objects, a speed of the vehicle, a respective speed of one or more of the one or more objects, an acceleration of the vehicle, infrared light information associated with an environment of the vehicle, or a combination thereof.