Roadway scene re-creation using video analytics and metadata

A system comprising a capture device and a database. The capture device may be configured to capture video, perform video analysis on the captured video to identify characteristics about objects in the captured video and determine an absolute location of the objects, store metadata separate from the captured video in response to the video analysis and communicate with a wireless communication device. The database may be configured to communicate with the wireless communication device, store the metadata received from the wireless communication device, re-create a scene in response to the characteristics and absolute location of the objects in the metadata and generate an animation based on the scene using the metadata received over time. The metadata may be communicated without the captured video. The animation may provide a visualization similar to the captured video using only the metadata.

FIELD OF THE INVENTION

The invention relates to video capture generally and, more particularly, to a method and/or apparatus for implementing roadway scene re-creation using video analytics and metadata.

BACKGROUND

Installing a dashboard camera on a vehicle is becoming increasingly common. The dashboard cameras capture detailed scenes of roadway events using high definition or even ultra-high-definition video. The encoded high or ultra definition video data are typically stored together with time synchronization to an onboard storage medium such as an SD card, CF card, internal memory, etc. In the event of an incident, the stored data can be exported and the video can be reviewed for post-incident analysis. Collisions involving vehicles are expensive, especially when death or injury occurs, with substantial financial repercussions to the individuals, companies and associated insurance providers involved.

Storage of video data generally occupies a large amount of storage capacity. For a dashboard camera video, continually capturing video results is a large storage burden (i.e., storing every frame of video captured even if processed with high efficiency encoding techniques such as H.265). Removable storage media, such as consumer-level SD cards might have a storage capacity range from 32 GB-1 TB. Even a 1 TB storage capacity will reach capacity when continually recording. Most dashboard cameras implement a circular buffer that effectively erases the oldest data with new incoming data periodically in order to ensure storage capacity for the latest recordings. If an event is flagged (either automatically, or through driver action, such as pushing an “event” button), the system can allocate a predetermined amount of capacity for the data before and after the event to preserve a video recording outside of the circular buffer.

In some scenarios, significant events may not be flagged (or understood by the system to be significant) until long after the video recording has been discarded. For example, a delivery company may be contacted by an irate customer with a complaint that the delivery truck had backed over a flower garden the previous day when the owner was not home. If the driver had no recollection of such an event and the system was not aware of any damage to the vehicle or property, then the event might not have been flagged for preservation. The truck may have had several cameras that recorded various points-of-view that could exonerate the company and driver, but the video and sensor data is no longer available.

In a vehicle fleet scenario, (i.e., a taxi service, a delivery service, a utility service, etc.) there is a preference to avoid accidents or other incidents altogether through an activity called driver coaching. Risky or accident-prone driver behavior can be flagged by the system and either preserved locally at the on-board storage or sent via RF transmission for analysis by a remotely located driving coach. The driving coach can review the incident and provide feedback to the driver on how to eliminate the risky behavior. Uploading large amounts of video data, and performing a manual, visual review of the driver behavior to flag undesired activity is time-consuming. The coach is unable to immediately notify the driver with corrective action, particularly in a high-risk scenario. For example, distracted driving is well known to severely heighten the risk of an accident. If the driver is found to be smoking, eating or texting during driving the fleet operation center would want to be notified immediately to contact the driver without delay to stop the activity before the driver can continue.

Sending video data in real-time during the event is burdensome at best, as even state of the art encoded video represents an enormous amount of data. The time and cost to transmit the video data over limited bandwidth networks available on the roadways is prohibitive. Wireless networks, especially outside of a home or corporate domain are notoriously bandwidth constrained and data bandwidth is expensive. Uploading minutes of HD or Ultra-HD video data is expensive and time consuming.

In the event of an automobile accident, providing accurate details as quickly as possible to the relevant stakeholders is important (i.e., correct description of the accident, insurance information, value of damages, etc.). Stakeholders may include local emergency services, police, insurance companies, among others. Even if the video data of the accident is available, manually flagging and communicating the video data is cumbersome.

It would be desirable to implement roadway scene re-creation using video analytics and metadata.

SUMMARY

The invention concerns a system comprising a capture device and a database. The capture device may be configured to capture video, perform video analysis on the captured video to identify characteristics about objects in the captured video and determine an absolute location of the objects, store metadata separate from the captured video in response to the video analysis and communicate with a wireless communication device. The database may be configured to communicate with the wireless communication device, store the metadata received from the wireless communication device, re-create a scene in response to the characteristics and absolute location of the objects in the metadata and generate an animation based on the scene using the metadata received over time. The metadata may be communicated without the captured video. The animation may provide a visualization similar to the captured video using only the metadata.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention include providing roadway scene re-creation using video analytics and metadata that may (i) implement vehicle-mounted cameras configured to perform local video analytics, (ii) generate metadata corresponding to captured video, (iii) store metadata about multiple vehicles, (iv) provide a searchable database of video metadata, (v) use a current location to determine an absolute location of objects in video, (vi) transfer metadata from a vehicle-mounted camera to a database via a user communication device without communicating video data, (vii) generate an animation representing information in video data, (viii) re-create a scene from captured video after the source video file is no longer available, (ix) provide alternate views in a re-created scene other than the point of view of the vehicle-mounted camera, (x) preserve space on a storage medium for metadata and/or (xi) be implemented as one or more integrated circuits.

Embodiments of the present invention may be configured to generate metadata in response to analyzing and detecting objects in captured video. The captured video may be generated using one or more camera systems mounted to a vehicle. In an example, a dashboard camera may be implemented as a forward facing camera that may capture a scene as high-quality video from a perspective similar to the perspective of a driver. In some embodiments, multiple cameras may be installed. In one example, the cameras may be configured with cabin viewpoints, which allow the occupants of the vehicle and the behavior of the occupants to be captured simultaneously with the forward viewpoint. In another example, the cameras may be installed at the sides of the vehicle, inside a cargo bay of a vehicle and at the back of a vehicle capturing the rearward view. Audio recording devices may also be employed to capture important auditory details inside or outside the cabin of the vehicle.

Embodiments of the present invention may be configured to capture data from an array of sensors (e.g., a sensor cluster) implemented in the camera system. Data from the sensor cluster may be recorded along with video data (e.g., HD video and/or Ultra HD video). In an example, the camera system may comprise sensors such as GPS/GNSS for real time location data, a magnetometer for direction, an accelerometer and gyroscope for positional data, rotational data, and acceleration data of the vehicle, etc. In some embodiments, the camera systems may be connected to the vehicle communication bus (e.g., a CAN bus) and capture state information from the vehicle (e.g., steering wheel position and movement, gas and brake pedal activation, service codes related to the vehicle engine state, tire inflation details, seatbelt engagement status, door ajar sensor warnings, etc.). The data from the sensor cluster, the vehicle communication bus input, and the encoded high-definition audio and video may be stored.

Embodiments of the present invention may be configured to perform real time video object detection and recognition. The object detection and/or recognition may be configured to determine relative and/or absolute positions of the objects detected. Characteristics of the objects may be determined (e.g., make/model/year of a vehicle, identifying features, a status of street lights, etc.). The identified objects, the positions of the objects and/or the characteristics of the objects may be stored as metadata separate from the video data. The metadata may be configured to provide a description of the information in the video data. The resulting data packets of the metadata may be used to replace the encoded video with codes that may adequately describe the object along with relative positions of the object with respect to the subject (e.g., ego) vehicle. The video data may be converted into objects, descriptions, and/or distances in the metadata. The metadata may be uploaded to a server and/or a remote computing device. The amount of data transferred by only transferring metadata may be less than the amount of data for transferring the video data. The remote computing device may analyze the metadata in order to generate an animation of the video data described by the metadata.

The camera systems may implement an edge device. The camera systems may be configured to perform real time video analytics at the edge (e.g., locally on the camera system instead of uploading to a cloud computing service for the video analytics). On-board computer processing may be configured to enable trained computer vision models to identify objects from video frames as part of the video processing pipeline. The objects may be identified before the video is encoded and/or stored. The information about the objects (e.g., the absolute locations accurately calculated with respect to the camera lens, the orientations, the characteristics, the statuses, etc.) may provide sufficient information about the objects such that the video data may not be necessary in order to re-create the events that occurred in the captured video in great detail at a later time. For example, the encoded pixel data may occupy a large amount of storage space and may include unnecessary details (e.g., details irrelevant to determining where vehicles are located, determining a cause of a crash, determining property damage, etc.) captured over a large number of video frames. The metadata may provide the important information about the objects captured in the metadata. In an example, the captured metadata may provide all important information concerning the objects from the scene using a much smaller storage space than storing video data of the scene.

Using the metadata generated, many details in the field of view of the camera systems may be identified and known without the need for the source video feed to be archived onboard the camera system and/or uploaded to a storage service. The video data may not need to be uploaded and/or recorded in real-time. The video data may not need to be transferred to a storage medium (e.g., when the vehicle arrives at a destination such as a fleet headquarters). The details about the objects determined from the video data, information in the encoded audio, the sensor data, and the vehicle state data may be recorded in the metadata. The metadata may be input into a system configured to reproduce the scene as an animation for review based on the metadata alone.

Embodiments of the present invention may be configured to generate the metadata with sufficient information to re-create the captured scene such that encoded video may no longer need to be archived. Since the encoded video data may occupy a larger capacity of memory than the metadata, by not storing and/or transmitting the video data, the cost of the system may be reduced (e.g., cost savings on bandwidth costs for transferring data and cost savings on data storage). For the camera system, a storage medium (e.g., an SD card) may be implemented with a smaller capacity (e.g., lower cost) if the encoded video data does not need to be stored. For example, a large capacity SD card may be used to archive large amounts of encoded video data. However, the metadata may provide a condensed version of the details of the objects in the video data. In one example, the smaller sized metadata may be transmitted in real time using internal buffers. In another example, the smaller sized metadata may be stored using a smaller sized SD card since less video data may be stored (or none at all).

Since the metadata may provide a condensed (or compressed or minimized) version of the information about the content in the video data, the bandwidth consumed for uploading flagged events or the entire data feed may be possible. On roadways in localities where wireless communication infrastructure is not ideal, the lesser bandwidth consumed by transmitting the metadata without the video data may facilitate communication. Since bandwidth is generally expensive and costs increase as the amount of data transmitted increases, transmitting the metadata alone may limit costs significantly.

The reduction in bandwidth resulting from embodiments of the present invention may enable the camera systems to be implemented without communication modules that implement expensive data communication protocols (e.g., LTE/5G radio components or even Wi-Fi). The amount of data in the metadata may be small enough that less expensive (e.g., slower speed) wireless components may be implemented (e.g., Bluetooth communication to a smartphone may communicate the metadata and the smartphone, which may have the faster communication protocols may forward the metadata to the remote computing device). In another example, a wired connection (e.g., USB) may be implemented to transmit the metadata directly to a smartphone. Once the data is transmitted to the smartphone, the smartphone and may archive the metadata and/or communicate the metadata to a remote computing device (e.g., cloud storage).

Since the amount of data in the metadata may be small, the metadata may be constantly uploaded to the remote computing device. Constantly uploading the metadata may ensure all the data is uploaded in the case of data loss on the camera system. For example, the storage medium (e.g., an SD card, an SSD, an HDD, onboard memory) may become corrupted or destroyed resulting in data loss. By uploading the small amount of data in real time, data preservation on the local storage medium may not be critical.

The video data may be discarded after being processed into the metadata. The metadata may comprise a smaller coding for the objects, positions, sensor cluster data and/or vehicle state information than the video data. In some embodiments, the camera systems may be implemented without a video encoder (e.g., further reducing costs).

Referring toFIG.1, a diagram illustrating an example system50in accordance with an embodiment of the present invention is shown. The system50may comprise a block (or circuit)60, blocks (or circuits)100a-100n, blocks (or circuits)110a-110n, blocks (or circuits)112a-112nand/or a block (or circuit)120. The block60may be a network. The blocks100a-100nmay implement video capture devices. The blocks110a-110nmay implement subscriber devices (e.g., subscriber communication devices). The blocks112a-112nmay implement user devices (e.g., user communication devices). The block120may implement a server computer. The system50may comprise other components (not shown). The number and/or types of components implemented by the system50may be varied according to the design criteria of a particular implementation.

The network60may enable communication between the various components of the system50. In an example, the network60may be the internet and/or a wide area network. Some of the components of the system50may communicate with the internet60wirelessly. Some of the components of the system50may communicate with the internet via a hard-wired connection.

The subscriber devices110a-110nand/or the user communication devices112a-112nmay be configured to execute computer readable instructions (e.g., executable programs, apps, binaries, etc.). For example, the subscriber devices110a-110nand/or the user communication devices112a-112nmay be implemented as a desktop computer, a laptop computer, a netbook computer, a tablet computing device, a phablet computing device, a smartphone, a smartwatch, smart clothing (e.g., clothing with LTE communication built in), human implantable devices (e.g., a computer chip embedded under the skin), etc. In an example, the subscriber devices110a-110nand/or the user communication devices112a-112nmay be implemented as a vehicle capable of 3G/4G/LTE/5G communication (e.g., a vehicle with a touchscreen infotainment system). Generally, the subscriber devices110a-110nand/or the user communication devices112a-112nmay be a device capable of data transmission to the network60and may comprise a display, a processor, a memory, an input (e.g., mouse, keyboard, touchscreen, voice recognition, etc.) and/or an output (e.g., a display, haptic feedback, a speaker, etc.). In some embodiments, the subscriber devices110a-110nand/or the user communication devices112a-112nmay have similar implementations. For example, the user communication devices112a-112nmay be wireless communication devices. The type and/or features of the subscriber devices110a-110nand/or the user communication devices112a-112nmay be varied according to the design criteria of a particular implementation.

Each of the subscriber devices110a-110nmay be configured to connect to the network60(e.g., hard-wired, Wi-Fi, etc.). The subscriber devices110a-110nmay be configured to store and/or execute the computer readable instructions114a-114n. Executing the computer readable instructions114a-114nmay enable the subscriber devices110a-110nto display various interfaces, generate output and/or receive input. For example, the app114amay be a front-end for interacting with the system50.

Each of the capture devices100a-100nmay be configured to capture video data. In some embodiments, the capture devices100a-100nmay be implemented as vehicle-mounted cameras (e.g., dashcams) to record video while a user drives the vehicle. In one example, each vehicle may be equipped with one of the capture devices100a-100n. In another example, one vehicle may be equipped with more than one of the capture devices100a-100n(e.g., to capture multiple fields of view and/or perspectives from the vehicle). In some embodiments, the capture devices100a-100nmay be cameras mounted at stationary locations (e.g., security cameras mounted on buildings). For example, the stationary camera may generate metadata used to determine roadway data. The implementation of the capture devices100a-100nmay be varied according to the design criteria of a particular implementation.

In some embodiments, the capture devices100a-100nmay be configured to communicate directly with the network60. For example, the capture devices100a-100nmay comprise components implementing Wi-Fi communication and/or 3G/4G/LTE/5G (e.g., cellular) communication. In some embodiments, the capture devices100a-100nmay be configured to communicate indirectly with the network60. For example, the capture devices100a-100nmay comprise short-range communication such as Bluetooth and/or Wi-Fi (e.g., short-range communication to a tethered device such as a smartphone). A cost of manufacturing the capture devices100a-100nmay be reduced if no 3G/4G/LTE/5G is implemented. A 3G/4G/LTE/5G connection further adds costs for the user since 3G/4G/LTE/5G generally involves a subscription to a carrier (and potential data usage penalties). For example, a cost of the capture devices100a-100nmay be lower when Bluetooth alone and/or Bluetooth/Wi-Fi is implemented compared to a camera that implements 3G/4G/LTE/5G hardware. Implementing the capture devices100a-100nwith a low cost may enable users to buy more than one of the capture devices100a-100nand/or provide a larger user base. When the system50has more of the capture devices100a-100navailable to capture video data and/or provide metadata, more data points may be available for analysis. Generally, having more data points enables more useful analytical results generated by the system50. In some embodiments, the capture devices100a-100nmay comprise a display and/or an input interface. For example, the capture devices100a-100nmay be configured to run apps (e.g., the computer executable instructions114a-114n). In another example, the capture devices100a-100nmay be implemented as smartphones configured as cameras.

Each of the user communication devices112a-112nmay be configured to connect to the network60and/or the capture devices100a-100n. In one example, the user communication devices112a-112nmay implement wireless communication devices. The user communication devices112a-112nmay comprise components configured to implement a wide area network connection (e.g., Wi-Fi) and/or local, device-to-device connections (e.g., Bluetooth, ZigBee, Z-Wave, etc.). For example, the user communication devices112a-112nmay implement a Wi-Fi and/or 3G/4G/LTE/5G connection to the internet60and a Bluetooth and/or Wi-Fi connection to one or more of the capture devices100a-100n. In some embodiments, the user communication devices112a-112nmay be configured to send/receive data to/from the internet60. For example, the user communication devices112a-112nmay receive data (e.g., video data, metadata, etc.) from one or more of the capture devices100a-100nand transmit the data to the internet60. In another example, the user communication devices112a-112nmay receive data (e.g., data requests, interrupt requests, firmware updates, etc.) from the internet60and transmit the data to the capture devices100a-100n.

Generally, the user communication devices112a-112nare implemented as portable devices (e.g., carried by a person, mounted in a vehicle, battery powered, etc.). The user communication devices112a-112nmay be configured to execute the computer readable instructions114a-114n. In one example, the subscriber devices110a-110nmay store and/or execute one version (e.g., a subscriber version) of an app (e.g., the computer readable instructions114a) and the user communication devices112a-112nmay store and/or execute another version (e.g., a provider version) of an app (e.g., the computer readable instructions114b). One of the user communication devices112a-112nmay be configured to communicate with more than one of the capture devices100a-100n(e.g., one smartphone may connect to multiple vehicle-mounted cameras in the same vehicle). In the example shown, the smartphone112acommunicates with the capture device100aand the capture device100b(e.g., the driver may have the smartphone and the vehicle may have two vehicle-mounted cameras). The connections between the user communication devices112a-112nand/or the capture devices100a-100nmay be varied according to the design criteria of a particular implementation.

Each of the capture devices100a-100nmay comprise a respective block (or circuit)102a-102n. The circuits102a-102nmay implement video processor functionality. In some embodiments, the circuits102a-102nmay be a system-on-chip (SoC). For example, the circuits102a-102nmay comprise input/output, a memory, processors, etc. The components and/or functionality of the circuits102a-102nmay be varied according to the design criteria of a particular implementation.

The circuits102a-102nmay be configured to record, encode, decode, transmit and/or store video data. The circuits102a-102nmay be configured to perform video analysis and/or video analytics. For example, the circuits102a-102nmay process video, identify patterns in the video data and/or recognize objects captured by the video data. Heuristics, templates and/or comparisons may be performed by the circuits102a-102nto recognize and/or identify objects in captured video frames (e.g., video data) as objects that can be perceived by humans. In one example, the circuits102a-102nmay identify an object as a vehicle (or part of a vehicle such as a hood, a license plate, etc.). In another example, the circuits102a-102nmay identify text, shapes and/or colors. In yet another example, the circuits102a-102nmay identify objects (e.g., signs, pedestrians, street lights, etc.). The video data captured by the circuits102a-102nmay be stored by the capture devices100a-100n. In one example, the circuits102a-102nmay implement a memory. In another example, the circuits102a-102nmay connect to an external memory (e.g., the capture devices100a-100nmay be configured to receive a flash memory such as a SD card, a microSD card, NAND memory, Compact Flash (CF) and/or an XD card).

The circuits102a-102nmay be configured to extract metadata from the captured video frames. The metadata may comprise information about objects detected in the video frames by the video analysis. The metadata may comprise information about the video data (e.g., a time the video was recorded, the model number of the camera recording the video, a location that the video was captured based on GPS information, an altitude that the video was recorded, a direction of the capture device while capturing the video data, etc.). For example, the circuits102a-102nmay comprise a GPS unit to determine GPS coordinates, a magnetometer to determine direction of travel and/or a real time clock (RTC) circuit to determine time.

The circuits102a-102nmay be configured to detect license plates. License plates may be detected, and the alphanumeric characters and/or symbols on the license plate may be identified. The license plate alphanumeric characters and/or symbols may be extracted from the captured video data as the metadata. For example, the circuits102a-102nmay detect, and store as metadata, the license plate alphanumeric characters and/or symbols, and store the license plate alphanumeric characters and/or symbols, the time the video was recorded, the longitude coordinate, the latitude coordinate, an altitude value, time of capture and/or the direction of the capture devices102a-102nwhen the video was recorded. The circuits102a-102nmay be further configured to analyze the video data to extract metadata such as a make of a vehicle, a model of a vehicle, color(s) of a vehicle, number of pedestrians, number of vehicles and/or roadway characteristics (e.g., road condition, weather condition, traffic signals present, state of traffic signals, road signs present, amount of traffic, flow of traffic, etc.). The metadata may be associated with the corresponding video data. For example, each video clip captured may be assigned an identification number and the metadata associated with the video clip may have the identification number.

The server120may be configured to store data, retrieve and transmit stored data, process data and/or communicate with other devices. In an example, the server120may implement a metadata server. The server120may be implemented as part of a cloud computing platform (e.g., distributed computing). In an example, the server120may be implemented as a group of cloud-based, scalable server computers. By implementing a number of scalable servers, additional resources (e.g., power, processing capability, memory, etc.) may be available to process and/or store variable amounts of data. For example, the server120may be configured to scale (e.g., provision resources) based on demand. The server120may implement scalable computing (e.g., cloud computing). The scalable computing may be available as a service to allow access to processing and/or storage resources without having to build infrastructure (e.g., the provider of the system50may not have to build the infrastructure of the server120).

The server120may be configured to execute computer readable instructions. In an example, the server120may process HTML, CSS, Javascript, PHP, SQL, AJAX applications, APIs, etc. The server120may be configured to distribute apps (e.g., one or more of the computer readable instructions114a-114n) to the subscriber devices110a-110nand/or the user communication devices112a-112n. The server120may be configured to generate interfaces (e.g., graphical user interfaces) based on stored data for the subscriber devices110a-110n. For example, the server120may generate data to implement an interface, the data may be sent to the subscriber devices110a-110n, the subscriber devices110a-110nmay interpret the data to generate a user interface, the user may interact with the user interface to provide requests, the subscriber devices110a-110nmay transmit the requests to the server120and the server may process the requests. Similarly, the capture devices100a-100nand/or the user communication devices112a-112nmay interpret data from the server120to implement an interface. The processing capabilities and/or functionality of the server120may be varied according to the design criteria of a particular implementation.

The server120may comprise a block (or circuit)120. The circuit120may implement a database (e.g., a remote database). The database130may store data and/or filter the stored data in response to search parameters. Details of the database130may be described in association withFIG.2. Generally, the database130may store data provided by the capture devices100a-100n. In an example, the database130may store the metadata. Search parameters may be transmitted by the subscriber devices110a-110nand the database130may be searched based on the search parameters. For example, the database130may enable the metadata to be associated with the video data stored by (and located on) the capture devices100a-100n.

The system50may be configured to provide a searchable, real time database of roadway video. In an example, the system50may be implemented to assist in time-critical challenges (e.g., AMBER alerts, roadway crime, asset recovery, auto insurance investigation, etc.). The system50may implement a “Big Data” approach to providing and/or searching captured video and/or metadata.

The system50may be implemented using inexpensive cameras100a-100nto program participants (e.g., the users and/or the data producers). The data producers may install the capture devices100a-100n. For example, the data producers may install the capture devices100a-100non vehicles as dashcams. The capture devices100a-100nmay provide the benefits of a security camera and/or a dashboard camera to the data producers (e.g., security, video evidence, video data for uploading to video services such as YouTube, etc.). In some embodiments, the system50may determine an amount of video recorded and provide rewards (e.g., perks) to the data producers (e.g., discounts on the capture devices100a-100n).

The data producers may use the capture devices100a-100nto collect and/or upload video metadata to the server120(e.g., for storage in the database130). For example, the video metadata may be uploaded via the user communication devices112a-112n. The data producers may provide the recorded video to the server120on demand. The data producers may be compensated on an ongoing basis for providing the video metadata and/or the recorded video. In one example, the data producer may receive a payment for providing and/or making a pre-determined amount of recorded video available. In another example, the data producer may receive a payment each time one of the video recordings is requested.

The video metadata may be accumulated in the remote database130. For example, the database130may be curated. The video metadata may be made available through a web interface to subscribers (e.g., the data consumers). The subscribers may use the subscriber devices110a-110nto access the database130. The database130and/or the server120may enable the subscribers to search the database130using search parameters. In one example, the interface may provide a map overlay (e.g., based on data presented by the server120) that the subscriber may interact with on the subscriber devices110a-110nto provide the search parameters. In another example, the subscriber may specify search parameters such as a location, a time of an incident and/or license plate data. The database130may perform a search of the metadata to determine whether any of the video metadata matches the search parameters.

The database130may provide the search results. The interface generated on the subscriber devices110a-110nmay provide the subscriber with a list of videos that match the search results communicated by the server120. The subscriber may request available recorded video files for a given event. If a subscriber requests one of the video files, a request may be sent to the server120. The server120and/or the database130may determine which of the capture devices100a-100ncaptured the video based on the video metadata (e.g., the metadata may comprise an ID of a camera and/or user that captured the video). The server120may send a request to the user communication devices112a-112nand/or the capture devices100a-100nto upload the recorded video. If the capture devices100a-100nstill have the requested video stored, a video upload may be initiated. The recorded video may be trickled (e.g., uploaded as a low priority data transfer) from the corresponding one of the capture devices100a-100n, through the corresponding one of the user communication devices112a-112nand to the internet60. In some embodiments, the recorded video may be buffered on one of the user communication devices112a-112nuntil particular conditions are met for uploading the video recording (e.g., until a Wi-Fi connection is available). The server120may notify the subscriber that the video is available for download. An account of the data producer that uploaded the video may be credited in response to the video upload.

The system50may enable data provider users to access the database130. The data provider users may feed the database130in real time with video metadata. The system50may enable the subscriber users to search the database130. When the database130determines there is a hit for a search request, the system50may allow the subscriber to get access to the video metadata and/or the video recording.

The capture devices100a-100nmay be implemented with inexpensive hardware powerful enough to perform video analytics (e.g., license plate recognition (LPR)). The video analytics may be performed in real time, while capturing the video data. In one example, the capture devices100a-100nmay be sold with a low margin to encourage wide adoption of the device so that many users may be the data providers to capture large amounts of video data for the system50. Since data providers may be compensated for providing the video data and/or video metadata, the data providers may have the ability to turn the capture devices100a-100ninto a money making tool. For example, in the system50the drivers may own the capture devices100a-100nand use them to make money (e.g., similar to how an Uber, Lyft or other ridesharing service drivers own a vehicle and use the vehicle to make money).

The database130may be implemented to receive video metadata, index the metadata and/or provide responses to search requests in real time. In some embodiments, the database130may store video recordings. Generally, the video metadata (e.g., plate number, GPS coordinates, time, etc.) is uploaded via the user communication devices112a-112nwithout the corresponding recorded video (e.g., the metadata may be uploaded before the video data is uploaded). If one of the subscriber users requests a recorded video file corresponding to the video metadata, the system50may enable the video data to be uploaded to the metadata server120(e.g., data may be uploaded as a low-priority data transfer). The recorded video data may have a limited time frame of availability. In one example, the capture devices100a-100nmay be configured to overwrite stored video as new video data is captured (e.g., a loop recording). In an example of 40 hours of driving per week with the capture devices100a-100nimplementing a 128 GB SD card and recording at 10 Mbit/s, the recorded video may be overwritten in approximately 3.5 days. When a video expires (e.g., is overwritten), the video metadata stored in the database130may still provide useful information to the subscriber user (e.g., which vehicles were in a particular area at a particular time).

The video metadata and/or the recorded video may provide useful information to the subscriber users. In one example, the system50may be helpful in an AMBER Alert situation. In another example, video evidence may be provided to insurance companies involved with a given auto accident. Different viewpoints and/or camera angles may be used to determine the circumstances that led up to an accident. In yet another example, the system50may save many government agencies (e.g., Department of Transportation) a tremendous amount of money in planning infrastructure (e.g., to limit infrastructure development and/or expand infrastructure development based on driving habits). In still another example, the system50may provide investigative departments (e.g., Department of Justice, local police, highway patrol, homeland security, etc.) with more information (e.g., post-mortem incident investigation).

Generally, the provider of the system50may not capture any video data. The video data may be captured by the data providers that have purchased (or otherwise received) the capture devices100a-100n. The provider of the system50may curate the resulting data generated by the data providers.

Referring toFIG.2, a diagram illustrating a block diagram representing example data sets stored in the database130is shown. The database130may comprise a block (or circuit)150, a block (or circuit)152, a block (or circuit)154, a block (or circuit)156and/or a block (or circuit)158. The block150may comprise a subscriber list. The block152may comprise video storage. The block154may comprise an authentication module. The block156may comprise user profiles. The block158may comprise plate and/or object metadata. The database130may comprise other blocks (or data sets). The implementation of the database130may be varied according to the design criteria of a particular implementation.

The subscriber list150may be configured to store information about the subscriber users. The subscriber list150may provide an account for each subscriber user. For example, a log in with password may be implemented by the app114ato enable the subscriber user to access the database130from the subscriber device110a. The subscriber list150may enable the system50to accept payment from subscriber users that request video data (e.g., store payment information, process payment information, etc.). The subscriber list150may implement individual settings, configurations and/or notifications for each of the subscriber users.

The video storage152may store recorded video data. In some embodiments, the data providers may upload the recorded video data to the database130when requested by the subscriber users. The database130may provide storage (e.g., temporary hosting) of the recorded video data to enable the subscriber user to download the requested video data. In some embodiments, peer-to-peer data transfers may be implemented to share the recorded video data (e.g., the database130may not store the recorded video data). Generally, the recorded video uploaded from the capture devices100a-100nmay be stored by the server120.

The authentication module154may be configured to provide security for the data stored in the database130. The authentication module154may be configured to prevent unauthorized access to the database130. In one example, the authentication module154may be implemented as a username and password. For example, the user communication devices112a-112nmay provide credentials to the database130to upload the video metadata and/or the recorded video. In another example, two-factor authentication may be implemented by the authentication module154. For example, the subscriber user may log in using the subscriber devices110a-110nby providing a username, a password, and an additional key (e.g., a text message with a passcode provided to the smartphone110a). The implementation of the authentication module154may be varied according to the design criteria of a particular implementation.

For example, users on the subscriber list150may be authorized users of the database130. Generally, not all users have access to the database130. The authentication module154may implement a heavy layer of security for the subscriber users and/or the data provider users to log onto the system50. Since the database130may store privacy information (e.g., license plate data, location information, credit card information, banking information, etc.) the database130may be secured with a traditional approach and then have a second layer of security added. Security may be provided even if the implementation of the authentication module154adds inconvenience to the users.

The user profiles156may store data corresponding to the data provider users. The user profiles156may comprise blocks (or circuits)170a-170n. The blocks170a-170nmay comprise the data provider profiles. Each of the data provider profiles170a-170nmay store information corresponding to an individual data provider. Each of the data provider profiles170a-170nmay comprise blocks (or circuits)172a-172n. The blocks172a-172nmay be configured to store data sets for the data providers170a-170n.

The data sets172a-172nmay facilitate access to the database130for each of the data provider users. In an example, the data set172amay store a video list. The video list172amay comprise a list of videos that have been recorded by a particular data provider. For example, the video list172amay be used to send a request to the capture devices100a-100nand/or the user communication devices112a-112nto upload the recorded video data. In another example, the video list172amay be used to provide a payment to the particular data provider that captured the requested video data. In an example, the data set172bmay store payment information. The payment information172bmay associate credit card, electronic payment (e.g., PayPal, Bitcoin, Apple Pay, Google Wallet, etc.) and/or bank information with a particular one of the data provider users. The payment information172bmay be used to facilitate payments to the data provider that has uploaded a requested recorded video.

In some embodiments, the data provider may receive one of the capture devices100a-100nin exchange for providing the payment information172band if enough video data is provided (e.g., provided on a consistent basis) the data collector user may not be charged. If the capture device is not used enough (or not used regularly, a charge may be incurred (e.g., due to inactivity). In an example, the data provider may receive one of the capture devices100a-100nfree of charge and be able to use the camera as a regular dashcam as well as for providing data for the system50. In one example, the data provider may not be charged for one of the capture devices100a-100nfor the first 90 days and if data is provided to the system50during the first 90 days no charge will be incurred. To avoid a situation where a freeloader receives the camera for free and uses the camera with the system50for a minimum threshold amount to avoid a fee and then stops providing data to the system50, the payment information172bmay be stored to charge a penalty fee to encourage the data provider to provide data.

In an example, the data set172nmay comprise user settings. The user settings172nmay provide a configuration and/or preferences for each of the data providers170a-170n. The data sets172a-172nmay store other information (e.g., a user name, a profile picture, a data usage plan of the data provider, etc.). In an example, the data usage plan may provide details of a carrier (e.g., 3G/4G/LTE/5G provider) to manage data transmission (e.g., prefer transferring large files over Wi-Fi instead of a limited data plan). The amount and/or type of data stored in the data sets172a-172nof each of the data provider profiles170a-170nmay be varied according to the design criteria of a particular implementation.

The metadata158may store information about various captured videos (e.g., video clips)180a-180n. The video clips180a-180nmay not be the actual video data (e.g., the actual video data may be stored with the video storage152). The video clips180a-180nmay comprise information about the video clips and/or information about objects detected in the video clips by the capture devices100a-100n.

In some embodiments, each video clip metadata180a-180nthat is captured and/or uploaded by the capture devices100a-100nmay be stored by the database130. The video data associated with the video clip metadata180a-180nmay be stored in the video storage152. Generally, the video data in the video storage152may be stored temporarily. For example, video data may have a relatively large file size and storing video data indefinitely may be impractical. However, the video clip metadata180a-180nmay have a much smaller file size than the video data. The video clip metadata180a-180nmay be configured to provide sufficient information about the video data to re-create the positioning of various objects in the video data even after the video data is no longer available.

Every time video data is uploaded to the database130, the corresponding video metadata180a-180nmay be stored by the database130. The video metadata180a-180nmay be stored long-term compared to the video data. In some embodiments, the metadata158may be organized by detected objects instead of the video clip metadata180a-180n. In one example, where the objects detected are license plates, each time a license plate is detected a new metadata entry may be created. For example, each of the license plate entries may comprise the associated clips180a-180n. The clips180a-180nmay comprise metadata entries for each time a license plate has been read by one of the capture devices100a-100n. For example, each time a license plate is detected, a new one of the clips180a-180nmay be appended to the corresponding one of the license plate entries. Similarly, if the objects detected are vehicles then the video clips180a-180nmay be associated with a particular vehicle. The arrangement of the metadata158may be varied according to the design criteria of a particular implementation.

Each video clip180a-180nis shown comprising metadata182a-182n. The clip metadata182a-182nmay comprise the data extracted by the capture devices100a-100nfrom the video recorded by the capture devices100a-100nand/or data associated with the video recorded by the capture devices100a-100n. The video clip metadata182a-182nmay be configured to provide useful information about the video clips that have been uploaded.

In one example, the video clip metadata182amay comprise a time. The time182amay indicate a date and/or time of day when the corresponding video was recorded (e.g., a timestamp). The time182amay be used to find recorded video that occurred at a particular time. In another example, the video metadata182bmay comprise an expiration flag. The expiration flag182bmay indicate whether or not the recorded video is still available (e.g., stored in the memory of the capture device, stored in the video storage152, has not been overwritten, etc.). For example, the expiration flag182bmay have a particular value (e.g., a logical one value) if the video has been overwritten. If the expiration flag182bindicates that the recorded video is no longer available, the video clip metadata180a-180nmay still provide useful information. The video clip metadata182cmay provide a file ID. The file ID182cmay be used to associate the video clip metadata180a-180nto a particular stored video file (e.g., either in the video storage152and/or in the memory of the cameras100a-100n). For example, if the expiration flag182bindicates the video data is still available then the file ID182cmay be used to retrieve the video data. The video clip metadata182nmay provide a camera ID. The camera ID182nmay be used to associate the video clip metadata180a-180nto a particular one of the cameras100a-100n(e.g., the camera that captured the video data associated with the metadata). The camera ID182nmay enable the video data to be retrieved from the capture devices100a-100n(if the video is still available) and/or to enable the data provider to be contacted for more information (or provide payment to the data provider user). The number and/or type of video clip metadata182a-182navailable may be varied according to the design criteria of a particular implementation.

The video clip metadata180a-180nmay comprise a number of objects184a-184n. The objects184a-184nmay correspond to each object detected using the video analysis performed by the capture devices100a-100n. In one example, the object184amay be a particular vehicle detected in the video data. In another example, the object184bmay be a particular pedestrian detected in the video data. In yet another example, the object184cmay be a license plate detected in the video data. In still another example, the object184nmay be a particular sign and/or landmark detected in the video data. The number and/or types of objects184a-184nstored with the video clip metadata180a-180nmay be varied according to the design criteria of a particular implementation.

Each of the objects184a-184nmay have associated object information186a-186n. In an example, the object information186amay correspond to an object type (e.g., a person, a vehicle, a building, a sign, a billboard, a license plate, etc.). The object type186amay provide details about the associated objects184a-184n. In one example, if the object is a vehicle, the object type186amay indicate the make, model, year, color, license plate, number of passengers, distinctive markings, etc. The object information186bmay correspond to a location. The location186bmay comprise GPS coordinates corresponding to the object in the recorded video. The location186bmay be used to find recorded video that was captured at a particular location (e.g., at an intersection at a particular time). In some embodiments, the location186bmay comprise an absolute location of the objects184a-184n. For example, the absolute location186bmay be determined by the video analysis performed by the capture devices100a-100nto determine the actual coordinates of the objects detected instead of merely the GPS coordinates of the capture devices100a-100n. In some embodiments, the location186bmay be the location of the object within the video frame (e.g., the distance of the object from the camera lens determined by the capture devices100a-100nusing video analysis).

The object information186cmay comprise a direction. In some embodiments, the direction186cmay indicate the direction of travel of the objects184a-184n(or if the objects184a-184nare stationary). For example, the direction186cmay be determined by the capture devices100a-100nanalyzing a sequence of video frames to determine where the object is moving over time. In some embodiments, the direction186cmay be the direction that the capture device186a-186nwas facing when the video data was captured. For example, the information from the location186band the direction186cmay be combined to determine the absolute location coordinates of the objects184a-184n. Other types of metadata186nmay be stored about the objects184a-184n. The types and/or amount of object information186a-186nmay be varied according to the design criteria of a particular implementation.

The information stored about the video clips180a-180nmay be used to identify vehicles, times, locations and/or other data about the recorded videos. The video clip metadata180a-180nmay be the data checked by the database130to determine results for a search query from the subscriber users. The video metadata180a-180nmay be used to approximate what may be recorded and/or visible when viewing the corresponding recorded video. The storage format for the metadata158may be implemented to enable re-enact and/or re-creating a scene (e.g., the vehicle locations) after the video data has expired (e.g., re-creating the arrangement of the objects when the video data is no longer available).

The circuits102a-102nmay be configured to perform object detection and/or video analysis to determine and/or recognize details of an object (e.g., of objects other than license plates). For example, in some video scenes, license plates may not be visible (e.g., the license plate is obstructed and/or not clear enough for optical character recognition). The circuits102a-102nmay be configured to determine roadway data in real time. In one example, the object information186a-186nmay store information corresponding to a type of vehicle detected (e.g., color of a car, make of a vehicle, model of a vehicle, year of a vehicle, speed of a vehicle, etc.). In another example, the object information186a-186nmay comprise roadway data (e.g., a lamp post detected, a street sign detected, a shape of a roadway detected, conditions of the road detected, etc.).

The database130may be configured to index the video metadata and/or associate new video metadata with license plate numbers and/or objects in real time. The database130may arrange the data to enable efficient filtering of information to provide fast search results for the subscriber users. In the example shown, the metadata158is arranged according to the video clips180a-180n. In another example, the metadata158may be arranged based on a time, a location, a camera ID, etc.). The arrangement of the storage of the data in the database130may be varied according to the design criteria of a particular implementation.

The database130may be configured to create a database entry for each incoming video clip. In one example, the video metadata182a-182nfor the clip180amay comprise information such as id=“1”, lp=“5SAM333”, date=“20170307”, time=“14:30”, alt=“141.46354”, lat=“37.804440” and/or lng=“−122.422874”. In another example, the video metadata182a-182nfor the clip180imay comprise information such as id=“2”, lp=“5SAM333”, date=“20170307”, time=“14:32”, alt=“142.13576”, lat=“37.804643” and/or lng=“−122.420899”. The database130may receive a large amount of data collected from various data provider users in a short amount of time. The database130may be constantly (e.g., continually, regularly, periodically, etc.) sorting the received data in order to serve up results to the subscriber users on the web interface. For example, the database130may implement one file for each license plate to avoid parsing all stored data in order to filter out license plate results in real time.

Referring toFIG.3, a diagram illustrating capturing video data from vehicle-mounted cameras is shown. A vehicle200is shown. The vehicle200may be a vehicle of a data provider (e.g., a data provider vehicle). The vehicle200may comprise a number of the capture devices100a-100n. In the example shown, the capture device100amay be installed facing the direction of travel of the vehicle200, the capture device100bmay be installed directed away from a passenger side of the vehicle200, the capture device100cmay be installed directed away from a driver side of the vehicle200and the capture device100dmay be installed directed facing opposite from the direction of travel of the vehicle200.

The user communication device112ais shown in the vehicle200. In the example shown, the user communication device112amay be a smartphone communicating to the network60(e.g., via a 3G/4G/LTE/5G wireless connection). For example, each of the installed cameras100a-100dmay communicate with the smartphone112a(e.g., creating a local network) and the smartphone112amay communicate with the external network60. In the example shown, the capture devices100a-100dmay be positioned on the windows of the front, side and back of the vehicle200(e.g., suction cupped from the inside of the vehicle200). The number, installation and/or locations of the capture devices100a-100nin a vehicle may be varied according to the design criteria of a particular implementation and/or a preference of the data provider.

A line202aand a line204aare shown extending from the capture device100a. The line202aand the line204amay represent a field of view captured by the capture device100a. The field of view of the capture device100amay record video of a view from the front of the vehicle200(e.g., from a perspective of a front of the vehicle200). A line202band a line204bare shown extending from the capture device100b. The line202band the line204bmay represent a field of view captured by the capture device100b. The field of view of the capture device100bmay record video of the view from the right of the vehicle200(e.g., from a perspective of a passenger side of the vehicle200). A line202cand a line204care shown extending from the capture device100c. The line202cand the line204cmay represent a field of view captured by the capture device100c. The field of view of the capture device100cmay record video of the view from the left of the vehicle200(e.g., from a perspective of a driver side of the vehicle200). A line202dand a line204dare shown extending from the capture device100d. The line202dand the line204dmay represent a field of view captured by the capture device100d. The field of view of the capture device100dmay record video of the view from the rear of the vehicle200(e.g., from a perspective of a back of the vehicle200).

The vehicle200may have a number of the capture devices100a-100ninstalled. In the example shown, four of the capture devices100a-100nmay be installed. For example, the cameras may be directed for a “drive mode” (e.g., the camera100adirected forward, and the camera100ddirected backwards) and the cameras may be directed for a “trawl mode” (e.g., the camera100band the camera100ceach directed sideways). For example, the trawl mode may be useful when in parking lots. The number of the capture devices100a-100ninstalled on the vehicle200may be varied according to the design criteria of a particular implementation.

Referring toFIG.4, a diagram illustrating multiple vehicles capturing video footage of an event220is shown. The event220may be a collision at an intersection of a road222and a road224. A vehicle250aand a vehicle250bare shown colliding. The drivers of the vehicle250aand the vehicle250bmay use the system50as subscriber users. For example, the subscriber users that drive the vehicle250aand the vehicle250b(or insurance companies representing the drivers of the vehicle250aand/or the vehicle250bto determine fault) may want video evidence from different viewpoints of the collision (e.g., to aid in resolving insurance claims that may arise as a result of the event220).

The vehicle200amay have the capture device100ainstalled, the vehicle200bmay have the capture device100binstalled, the vehicle200cmay have the capture device100cinstalled and/or the vehicle200dmay have the capture device100dinstalled. The drivers of the vehicle200a, the vehicle200b, the vehicle200cand/or the vehicle200dmay be data providers. A building230aand a building230bare shown. The building230amay have the capture device100einstalled and the building230bmay have the capture device100finstalled as a security camera. In some embodiments, one or more of the capture devices100a-100nmay be implemented as stationary cameras. The owner of the building230aand the owner of the building230bmay be data providers for the system50(e.g., capture video of the event220).

The capture device100amay capture one field of view (e.g., the line202aand the line204a) from the viewpoint of the vehicle200a. The capture device100bmay capture one field of view (e.g., the line202band the line204b) from the viewpoint of the vehicle200b. The capture device100cmay capture one field of view (e.g., the line202cand the line204c) from the viewpoint of the vehicle200c. The capture device100dmay capture one field of view (e.g., the line202dand the line204d) from the viewpoint of the vehicle200d. The capture device100emay capture one field of view (e.g., the line202eand the line204e) from the viewpoint of the building230a. The capture device100fmay capture one field of view (e.g., the line202fand the line204f) from the viewpoint of the building230b. The various fields of view may provide video metadata and/or video recordings from different viewpoints.

The database130may receive metadata corresponding to the video data captured by the capture devices100a-100fof the event220. For example, the database130may receive six different user IDs of the data provider users. The database130may receive six slightly different GPS coordinates corresponding to the different locations of the cameras100a-100f. In some embodiments, the database130may receive the same timestamp from each of the capture device100a-100f. In some embodiments, the timestamp may be slightly different because the video files may be stored as video clips having a pre-determined time (e.g., 2 minute video clips) and the start and end time of each video clip may depend on when the cameras100a-100fwere booted up (e.g., when the vehicles200a-200dwere started). In some embodiments, system50may be configured to synchronize the time on each of the capture devices100a-100n(e.g., to ensure the timestamp for a start and end time of each video clip matches between the capture devices100a-100n). The database130may receive up to 6 different direction metadata information. In some embodiments, multiple clips180a-180nwith metadata information182a-182nand/or object information186a-186nfor each object detected may be received (e.g., depending on the length of the event220).

The video metadata from each of the cameras100a-100fmay be uploaded to the database130. Since two vehicles (e.g.,250a-250b) are in the car accident, the database130may associate at least the objects184a-184bwith two license plates and/or vehicle type entries (e.g.,184afor the vehicle250aand184bfor the vehicle250b). Depending on the fields of view, some of the cameras100a-100nmay not capture both license plates and/or vehicles (e.g., the field of view of the camera100bmay capture the license plate of the vehicle250abut not capture the license plate of the vehicle250b). License plate entries may be made for the data provider vehicles200a-200d(e.g., the capture device100amay capture the license plate of the vehicle200d). Additional license plate and/or vehicle entries may be made for other objects and/or vehicles captured (e.g., the capture device100bmay capture the colliding vehicles250a-250bas well as the vehicle200a).

The metadata158may be extracted from the video data captured by each of the capture devices100a-100n. The video metadata158may be associated with the video clips180a-180ncorresponding to each field of view captured. In one example, the video clip180amay correspond to the metadata associated with the video data captured by the capture device100a. In another example, the video clip180bmay correspond to the metadata associated with the video data generated by the capture device100b. In one example, the object184amay correspond with the vehicle250acaptured by the capture device100aand associated with the video clip180aand the object184bmay correspond to the vehicle250bcaptured by the capture device100aand associated with the video clip180a. In another example, the object184amay correspond with the vehicle250bcaptured by the capture device100band associated with the video clip180band the object184bmay correspond to the vehicle250acaptured by the capture device100band associated with the video clip180b. The subscriber users may use the subscriber devices110a-110n(e.g., via the app and/or web interface114a) to search the metadata184a-184nto view any videos that may have been captured of the event220.

In some embodiments, the capture devices100a-100nmay implement Wi-Fi communication (e.g., to transmit the metadata and/or the recorded videos to the network60). Implementing the Bluetooth communication to transfer data between the capture devices100a-100nand the user communication devices112a-112nmay be useful in the vehicles (e.g.,200a-200d) to forward metadata and/or recorded videos to the network60. In some embodiments, the capture devices100a-100nmay implement Wi-Fi functionality to connect to access points that may be fixed (e.g., Wi-Fi hotspots, home networks, business networks, etc.). For example, if someone had a storefront or house that overlooked the intersection (or any field of view where license plate recognition and/or other type of roadway metadata extraction could be performed on passing vehicles), the cameras100a-100ncould be placed to face out a window of a home and/or business (e.g., similar to a consumer IP camera). Connecting the capture devices100a-100nto a Wi-Fi access point may enable the cameras100a-100nto operate like a consumer IP Camera but additionally provide the video clip metadata180a-180n(e.g., by receiving payments from the system50in exchange for providing the video metadata, the cost of a security camera for the business may be subsidized). Similarly, a business operating a fleet of vehicles (e.g., taxi drivers, delivery drivers, drivers of a car-sharing company, etc.) may install the capture devices100a-100nin an existing fleet of vehicles and make a small profit over time by receiving payments from the system50by being a data provider (and use the video data like an ordinary dash camera for post mortem analysis of any crash involving a vehicle from the fleet).

If the subscriber user (e.g., the driver of the vehicle250a, the driver of the vehicle250b, an insurance company representing the drivers of the vehicles250a-250b, and/or another user) searches the video clips180a-180nin the database130, a list of videos of the event220may be provided. If the subscriber user decides to request a video recording of the event, the server120may send a request to one or more of the capture devices100a-100n(or the associated user communication device112a-112n). The circuits102a-102nmay set a flag for the requested video in response to the request from the server120. Setting the flag for the requested video may be used to preserve the requested video recording (e.g., prevent the video from being over-written). The capture devices100a-100nand/or the user communication devices112a-112nmay upload the video data to the server120. In some embodiments, the associated video may be preserved for all data points that may be returned as search results of a search query. For example, the server120may send the request to one or more of the capture devices100a-100n(or the associated user communication device112a-112n) to preserve the video data associated with the search results so that the video data may be preserved in order to be made available if the subscriber user later decides to request the video data. The flag may be set for a pre-determined amount of time before the video is unflagged (e.g., to prevent storage of video data indefinitely).

In some embodiments, the video recording may be flagged for preservation in response to a request from a subscriber user. In some embodiments, the system50may send a request to all capture devices100a-100nthat have video data from a particular time and a particular location. For example, an event may occur that may be considered important (e.g., a VIP such as a celebrity is at a location, a crime has been committed, a gunshot was reported, a blast was reported, etc.). When a particular event has been determined to have occurred, all cameras100a-100nin the vicinity may start preserving data. In one example, if the VIP event is happening presently, the video data being currently recorded may be preserved. In another example, the video data currently residing in the memory of the capture devices100a-100nassociated with the time of the event may be preserved (e.g., a crime is reported after the event occurs and later an interrupt request is provided to the capture devices100a-100nto preserve potential evidence).

In some embodiments, the video metadata182a-182band/or the object information186a-186nassociated with each of the objects184a-184nmay enable the event220to be re-created, even if the video data is no longer available (e.g., expired, over-written, etc.). For example, the object information186a-186nmay be used for each of the objects184a-184nof each of the video clips180a-180nassociated with the event220(e.g., based on a timestamp and/or general location) to determine the absolute location of each of the vehicles200a-200dand/or the colliding vehicles250a-250b. Even if the video data is no longer available, the object information186a-186bnmay provide detailed absolute coordinate information over a sequence of time. For example, the detailed absolute coordinate information may be used to indicate where each of the colliding vehicles250a-250bwere before, during and after the event220. For example, by analyzing the absolute location coordinates over a sequence of time, the speed, direction of travel, the timing of the event220and/or other information about the colliding vehicles250a-250bmay be determined to re-create the event220.

Referring toFIG.5, a block diagram illustrating an example implementation of the circuit102aconfigured to perform the functionality of a vehicle-mounted camera100a. The capture device100amay be a representative example of one of the capture devices100a-100n. The capture device100amay comprise the circuit102a. In some embodiments, the circuit102amay be a main printed circuit board for the capture device100a. In some embodiments, the circuit102amay comprise various circuit boards, daughter boards and/or modules. For example, the circuit102amay comprise multiple printed circuit boards connected using cables. In another example, the circuit102amay comprise input/output slots configured to connect to drop in components (e.g., off-the-shelf components). Generally, the circuit102amay be designed to fit a space and/or heat dissipation specification for the capture device100a.

The circuit102amay comprise a block (or circuit)300, a block (or circuit)302, a block (or circuit)304, a block (or circuit)306, a block (or circuit)308, a block (or circuit)310, a block (or circuit)312and/or a block (or circuit)314. The block300may implement a wireless communication module. The block302may implement a processor. The block304may implement a memory. The block306may implement a lens module. The block308may implement a camera sensor module. The block310may implement a microphone. The block312may implement an audio processor. The block314may implement a sensor cluster. The circuit102amay comprise other components (not shown). For example, the circuit102amay comprise a firmware, input/output ports, memory expansion slots, a battery, a power adapter (for connection to an external power supply), etc. The components of the circuit102amay be varied according to the design criteria of a particular implementation.

The circuit102ais shown presenting a signal (e.g., META) and a signal (e.g., VIDEO). The circuit102ais shown receiving a signal (e.g., CAN). The circuit102amay present the signal META to a corresponding one of the user communication devices112a(e.g., a paired wireless communication device). The circuit102amay present the signal VIDEO to a video processing pipeline. The circuit102amay receive the signal CAN from a communication bus of the vehicle200a(e.g., a CAN bus).

The wireless communication module300may be configured to implement one or more wireless communication protocols. The wireless communication module300may be configured to transmit data and/or receive data. For example, the wireless communication module300may enable the circuit102ato communicate with one or more of the user communication devices112a-112nand/or the network60. In the example shown, the wireless communication module300may comprise a block (or circuit)320and/or a block (or circuit)322. The block320may implement a Bluetooth module. The block322may implement a Wi-Fi module. The wireless communication module300may comprise other components (not shown). For example, the wireless communication module300may comprise a module configured to implement 3G/4G/LTE/5G communication. The implementation of the wireless communication module300may be varied according to the design criteria of a particular implementation.

The Bluetooth module320may be configured to implement Bluetooth communication. For example, the Bluetooth module320may be configured to pair the circuit102awith the user communication device112a. The Bluetooth module320may enable a transfer of the video data and/or the metadata from the circuit102ato the user communication device112a. The Bluetooth module320may enable a short-range and/or low-power communication.

The Wi-Fi module322may be configured to implement Wi-Fi communication. In some embodiments, the Wi-Fi module322may be configured to connect to a local and/or wide area network. In some embodiments, the Wi-Fi module322may be configured to create a local network and other devices (e.g., the user communication device112a) may connect to the Wi-Fi module322. The Wi-Fi module322may enable wireless communication of the video data and/or the metadata. In some embodiments, the Wi-Fi module322may be configured implement a mesh network with other Wi-Fi modules. For example, the Wi-Fi module322implemented in the capture device100amay form a mesh network with other nearby Wi-Fi module(s)322in the capture devices100b-100n. Implementing a mesh network may enable the circuits102a-102nto communicate using an ad-hoc Wi-Fi network as vehicles move around. For example, one or more of the data provider users may have a desired connectivity to the network60(e.g., sufficient upload bandwidth, sufficient data quota with the service provider, an unlimited data plan, etc.). Other data provider users that do not have the desired connectivity to the network60may use the mesh network to utilize the connectivity to the network60of data provider users that do have the desired connectivity to the network60. The mesh network may help more of the data provider users to have reliable access to the network60.

In some embodiments, the circuit102amay be implemented without the Wi-Fi module322. For example, since the signal META may be transmitted to the user communication device112awithout the video data, the circuit102amay have a low bandwidth requirement. The small amount of data communicated in the signal META may enable a lower cost communication protocol (e.g., the Bluetooth module320) to be implemented. The circuit102amay rely on the user communication device112ato communicate with the network60. Using a low cost communication protocol to communicate with the user communication device112aand relying on the user communication device112ato communicate with the network60may enable the circuit102ato be implemented at a low cost.

The processor302may be configured to read, write and/or process data, perform calculations and/or execute computer readable instructions. The processor302may implement one or more processing cores to enable parallel processing. The processor302may be implemented using various architectures (e.g., x86/x64, ARM, RISC-V, etc.). In some embodiments, the processor302may be a Tensor processing unit configured for deep machine learning. The processor302may be configured to perform various types of operations (e.g., a general purpose processor). In some embodiments, the processor302may implement various hardware modules configured to implement specific tasks. In the example shown, the processor302may comprise a block (or circuit)330a block (or circuit)332and/or a block (or circuit)334. The block330may implement a video analytics module. The block332may implement an input buffer. The block332may comprise an input buffer. The block334may implement a detection libraries module. The processor302may comprise other types of modules such as a video encoding module, an image encoding module and/or other modules (not shown). For example, the processor302may further comprise an encryption module. The implementation of the processor302may be varied according to the design criteria of a particular implementation.

The video analytics module330may be configured to perform real-time video analytics on the captured video data. The video analytics module330may be configured to perform the real-time video analytics on the video data captured by multiple installed camera systems in parallel (e.g., all of the camera systems100a-100dconfigured to provide an all-around view as shown in association withFIG.3). The video analytics module330may be configured to extract the metadata in response to the video analytics. The video analytics module330may be configured to scan visual data present in each video frame captured by the capture device100a. In some embodiments, the video analytics module330may implement optical character recognition. In some embodiments, the video analytics module330may be configured to detect and/or classify objects detected in the video data. For example, the video analytics module330may compare features of the captured video frames to known features (e.g., templates) to identify an object captured in the video frame as a particular type of object (e.g., identify a vehicle as a particular make, model and year of a vehicle). The type of video analytics performed by the video analytics module330may be varied according to the design criteria of a particular implementation.

In some embodiments, the analytics performed by the video analytics module330may comprise determining characteristics of roadway features and/or objects. The metadata generated by the video analytic module330may be used as training data sets for machine learning. In one example, the capture devices100a-100nmay be implemented as dashboard cameras (e.g., dashcams) and the images captured may comprise various roadway data. The video analytics module330may analyze many images of various roadways and/or obstacles (e.g., curves, lane markings, curbs, street signs, traffic lights, license plate styles, road markings, pedestrians, vehicle types, vehicle makes, vehicle models, road hazards (e.g., snow, ice, flooding, chemical spills, etc.), construction barriers, etc.). The large amounts of metadata produced by the video analytics module330may provide many sample images and/or metadata of roadway data and/or object data. The data generated by the video analytics module330may be used to generate feature templates to improve detection when video analytics is performed (e.g., provide a greater data set to compare against). In an example, the metadata generated by the video analytics module330may be combined with data from other sensors (e.g., LIDAR, GPS and/or other sensors used to gather roadway data) for machine learning for autonomous driving.

In some embodiments, the analytics performed by the video analytics module330may comprise determining a distance of various detected objects from the lens306. The video analytics module330may be configured to compare the sizes of objects detected in the video data with known object sizes. The video analytics module330may be configured to determine a region of a license plate, and compare the detected plate characters with the size of the license plate characters for the detected region (e.g., or vehicle type for vehicles such as motorcycles that have smaller plates). The video analytics module330may be configured to detect the corners of objects (e.g., such as the corners of a license plate) to determine a size of the detected object.

In some embodiments, the analytics performed by the video analytics module330may be configured to account for (e.g., normalize) objects that are at an angle to determine the size of the detected objects. For example, relationships between various portions of an object may be analyzed to determine an orientation of an object (e.g., rotated, at an angle, slanted, skewed, etc.). The video analytics module330may be configured to correct distortion caused by the lens306. For example, de-warping may be performed to correct distortions caused by a wide (e.g., fisheye) lens.

The video analytics module330may utilize a number of libraries provided by the detection libraries module334. Each of the libraries may be configured to recognize various types of objects. Each of the libraries may be specialized and/or customized for detecting specific types of objects. In one example, one of the libraries may be configured to detect and/or identify vehicles in the captured video frames. In another example, one of the libraries may be configured to recognize street signs. In yet another example, one of the libraries may be configured to perform OCR for license plate reading. The libraries may be third party modules for recognizing objects that may be divided into discrete tasks. Different libraries may be provided by different vendors (e.g., each vendor may specialize in a particular area of object detection). The different libraries may run sequentially or in parallel and operate on different parts of the video frame stored in the input buffer332.

The input buffer332may be a memory configured to provide temporary storage for newly captured video data. The video data may be in the input buffer332while the video analytics module330performs the video analysis in real time and/or video encoding is performed to store the video data in the memory304. The input buffer332may be configured to limit the amount of spatial data used from an input video frame as input to the buffer used for video analytics. By limiting the amount of spatial data input to the working buffer the inherently limited system resources of the real time system may allocated more efficiently.

The detection libraries module334may comprise a computer vision model comprising features of various objects for the video analytics module330to detect. For example, the various objects detected by the video analytics module330may be determined in response to a comparison with the data in the detection libraries module334. The detection libraries module334may be updateable. For example, as various feature libraries are loaded into the detection libraries module334, the video analytics module330may detect different types of objects and/or characteristics. In an example, one detection library may provide features for identifying vehicles, another detection library may provide features for identifying pedestrians, another detection library may provide features for identifying traffic signals, etc. Details of various library data implemented by the detection libraries module334may be described in U.S. Application No. 16,156,254, filed on Oct. 10, 2018 and U.S. application Ser. No. 17/568,746, filed on Jan. 5, 2022, appropriate portions of which are incorporated by reference.

The library (or libraries) implemented by the detection libraries module334for each of the camera systems100a-100nmay be determined based on a type of vehicle used or the type of service provided by the vehicle (e.g., details that may be important to one type of driver or property owner may not be important to another driver or property owner). In an example, for a regular commuter, the libraries implemented may be configured to detect details about traffic accidents (e.g., vehicles, traffic signals, pedestrians, etc.). In another example, for a delivery vehicle, the libraries implemented may be configured to detect details about property damage (e.g., to provide proof that the delivery truck did not cause damage). In yet another example, for a municipal traffic enforcement vehicle, the libraries may be configured to detect parking locations and/or parking violations. The types of libraries implemented by the libraries module334may be varied according to the design criteria of a particular implementation.

The lens module306may be configured to focus light onto the camera sensor308. The lens module306may comprise mechanisms for zooming, panning, tilting, focusing, adjusting a DC iris, etc. The camera sensor308may be configured to convert the light from the lens module306into images. The images generated by the camera sensor308may be the video frames of the captured video data. The video frames generated by the camera sensor308may be processed by the processor302to generate the captured video.

The microphone310may be configured to capture sound in the environment near the vehicle200a. In some embodiments, the microphone310may be located on the device102a. In some embodiments, the microphone310may be part of the vehicle200a. The location of the microphone310may be varied according to the design criteria of a particular implementation.

The audio processor312may be configured to perform various operations on the sound captured by the microphone310. In an example, the audio processor312may be configured to encode and/or compress the captured sound into an audio file. The audio file generated by the audio processor312may be associated with the captured video data (e.g., to synchronize the audio data with the video data). In some embodiments, the audio processor312may be configured to perform audio analytics. For example, the audio processor312may be configured to implement speech recognition and/or speech-to-text. The audio processor312may be configured to generate a text reproduction of the sound captured (e.g., provide a transcription of words spoken by occupants of the vehicle200a, provide a transcription of pedestrians near the vehicle200a, provide a text description of sounds heard (e.g., crashes, tires squealing, engines revving, etc.). In the example shown, the audio processor312is shown as a separate module from the processor302. In some embodiments, the functionality of the audio processor312may be performed by the processor302.

The sensor cluster314may comprise various input sensors implemented by the device102a. The sensor cluster314may be configured to generate sensor data that may be added to the metadata. The sensor cluster314may be configured to perform sensor fusion to interpret data from and/or make decisions based on disparate sources of data that would not be achievable using one data source alone. The sensor cluster314may be configured to perform various measurements (e.g., temperature, humidity, wind speed, acceleration, rotation, etc.). In the example shown, the sensor cluster314may comprise a block (or circuit)340and/or a block (or circuit)342. The circuit340may implement a location device. The circuit342may implement a magnetometer. The sensor cluster314may comprise other components (not shown). The number, type and/or arrangement of the sensors implemented by the sensor cluster314may be varied according to the design criteria of a particular implementation.

The location device340may be configured to calculate the location coordinates of the vehicle200a. In an example, the location device340may be a GPS/GNSS module. The location device340may be configured to communicate with GPS/GNSS satellites, receive data from the GPS/GNSS satellites and/or perform calculations on the data from the GPS/GNSS satellites to determine the current location of the capture device100a. The current location of the capture device100adetermined by the location device340may be stored as part of the metadata.

The magnetometer342may be configured to calculate a direction of travel and/or a direction the capture device100ais facing. The direction of travel and/or direction the capture device100ais facing may be a numerical representation of a direction and/or angle. The direction of the capture device100adetermined by the magnetometer342may be stored as the location coordinates, which may be part of the metadata. In some embodiments, if the magnetometer342is not available the previous GPS coordinates may be used to infer the direction of travel.

The memory304may be configured to store data. The memory304may comprise a solid state memory (e.g., NAND flash). In some embodiments, the memory304may comprise memory onboard the circuit102aand/or external memory (e.g., a microSD card). The memory304may comprise blocks (or circuits)350a-350n, a block (or circuit)352and/or a block (or circuit)354. The blocks350a-350nmay each implement storage of a video file. The block352may implement object data. The block354may implement a metadata cache. The memory304may comprise other data (not shown). For example, the memory304may further comprise a firmware. The type of memory implemented by the data arrangement of and/or the data stored by the memory304may be varied according to the design criteria of a particular implementation.

The video files350a-350nmay comprise the captured/recorded video and/or other information associated with the video. Each of the video files350a-350nmay comprise storage locations for a video clip (e.g., a discrete video recording and/or a video segment). In some embodiments, the video files350a-350nmay represent video clips having a same length (e.g., 2 minute video clips). In some embodiments, the video files350a-350nmay represent video clips (or segments) having different lengths (e.g., based on a storage size of each video file). Each video file storage location350a-350nmay comprise a corresponding one of a block (or circuit)370and/or a block (or circuit)372. The block370may comprise video data. The block372may comprise metadata. For example, the metadata372may be associated with the corresponding video data370. The metadata372may each comprise blocks (or circuits)380a-380n. The blocks380a-380nmay comprise various data sets of the metadata372. The block380amay comprise the video ID. The block380bmay comprise a time stamp. The block380cmay comprise a flag. The block380dmay comprise audio data. The block380emay comprise object information. The block380fmay comprise location coordinates and/or a direction. The block380gmay comprise absolute locations. The block380hmay comprise sensor data. The block380nmay comprise vehicle status information. The video files350a-350nmay comprise other data (not shown). The type of data stored and/or the arrangement of data stored in the video files350a-350nmay be varied according to the design criteria of a particular implementation.

The video data370may comprise the viewable captured video. The video data370may be the data transmitted in response to a request from the subscriber user (e.g., one of the subscriber devices110a-110n). The video data370may be encoded before the video data370is created and stored. In some embodiments, the video data370may not be stored (e.g., only the metadata372may be stored and the video data370may be discarded after the video analytics is performed).

The video ID380amay comprise an identification for the video files350a-350n. The video ID380amay be implemented as a data string comprising alphanumeric characters (e.g., a hash). The video ID380amay represent a portion of the metadata372associated with the video files350a-350n. The video ID380amay be used by the database130to locate and/or match the metadata clips180a-180nto a particular one of the video files350a-350non the devices100a-100n.

The time stamp380bmay comprise an indication of when the video files350a-350nhave been recorded. The time stamp380bmay comprise a data string. For example, the time stamp380bmay indicate a year, a month, a date, a day of the week, an hour, a minute, seconds, fractions of a second, etc. The time stamp380bmay represent a portion of the metadata associated with the video files350a-350n. The time stamp380bmay be used by the processor302to determine which of the video files350a-350nis the oldest. The time stamp380bmay be used by the processor302to determine an order that the video files350a-350nwere recorded, based on time. The time stamp380bmay be used by the processor302to correlate the captured video data370and/or extracted metadata372with the location coordinates380f.

The flags380cmay be used to indicate whether the video data370may be overwritten. The flags380cmay represent one or more preservation bits for the video files350a-350n. In one example, when one of the flags380cis set (e.g., to indicate the file should be preserved) by the capture devices100a-100n, the corresponding video data370of one of the video files350a-350nmay not be overwritten. In another example, when one of the flags380cis not set (e.g., to indicate the file should not be preserved), the corresponding video data370of one of the video files350a-350nmay be made available to be overwritten (e.g., by a new video file).

In some embodiments, the flags380cmay be implemented using a single bit. In some embodiments, multiple bits may be used for each of the flags380c. In one example, when using multiple bits for the flags380c, a condition for determining how long to preserve the video data370may be set (e.g., an amount of time, a particular date, an occurrence of an event, etc.). In another example, when multiple bits are used for the flags380c, different lengths of time to preserve the video data370may be set based on a reason for preserving the video data370(e.g., preserve video files for 3 days if a video is the result of a subscriber search, preserve video files indefinitely if the video is evidence, preserve video files one week if an event was captured, etc.). In yet another example, when multiple bits are used for the flags380c, the bits may represent an amount of time to preserve the video data370based on a value of a subscriber user and/or subscription tier (e.g., a subscriber user paying a higher fee may result in a value of 5 being stored for the flags380cto indicate preservation for 5 days, a subscriber user paying a lower fee may result in a value of 2 being stored for the flags380cto indicate preservation for 2 days, etc.). Generally, a user that does not have a subscription to access the database130(e.g., a user that is not a subscription user) may not receive search results from the database130(e.g., the database130may not generate search results and/or may not allow a search query if a user is not one of the subscriber users). In some embodiments, the processor302may use the flags380cto decide when to overwrite the video data370(e.g., compare the creation date of the video files350a-350nplus the flag value to the current date). The implementation of the flags380cmay be varied according to the design criteria of a particular implementation.

The audio data380dmay comprise audio information corresponding to the video data370. In some embodiments, the audio data380dmay comprise compressed audio generated by the audio processor312(e.g., sound information compressed into an .mp3 file, a .flac file, a .aac file, a .ogg file, etc.). For example, since sound information may be relatively smaller than the video data370, the sound information captured by the microphone310may be preserved even if the video data370is not. In some embodiments, the audio processor312may perform audio analytics to transcribe sounds captured in the audio data (e.g., speech-to-text, describe sounds such as crashes and car horns, etc.) and the transcribed sounds may be stored as the audio data380d. Audio transcribed as text may be relatively smaller than storing sound information.

The object data380emay comprise information about the objects detected in response to the video analytics performed by the video analytics module330. In an example, the object data380emay be comprised of relative coordinates of each object detected (e.g., a distance and/or direction from the lens306, an X/Y coordinate position, etc.). In another example, the object data380emay comprise a classification of the objects detected (e.g., vehicles, people, signs, street lights, etc.). In yet another example, the object data380emay comprise characteristics of the objects detected (e.g., make/model/year of vehicle, color of vehicle, identifying markers on the vehicle (e.g., scratches, truck balls, bumper stickers, etc.), description of pedestrians, status of traffic lights, license plate information, text on signs, condition of roads, etc.). The type of data stored about the objects detected in the object data380emay be varied according to the design criteria of a particular implementation.

The location coordinates380fmay comprise the location coordinates determined by the sensor cluster314determined at the time the corresponding video files350a-350nwere created. The location coordinates380fgenerally correspond with the location and/or direction of travel of the capture device100a. For example, the location of the capture device100amay be used as a proxy for the location and/or direction of travel of the vehicle200a. The location coordinates380fmay comprise a sequence of values (e.g., to track the location of the capture device100aas the vehicle200amoves over the duration of the recording). The location coordinates380fmay comprise longitude values, latitude values, altitude values and/or the alternate coordinates (e.g., location information from the user communication devices112a-112n) generated by the location module340. The location coordinates380fmay further comprise the direction of travel determined by the magnetometer342in the sensor cluster314. The direction of travel may correspond to the direction that the capture device100awas facing when while the corresponding video files350a-350nwere recorded. The direction may be a sequence of values (e.g., to track the direction of the capture device100aas the associated vehicle200amoves over the duration of the recording).

The absolute location380gmay comprise the actual location of each of the objects in the object data380eidentified by the video analytics module in the video data370. The absolute coordinates380gmay be determined based on the location coordinates380f(e.g., location and direction of the capture devices100a-100n) and the distance of the detected objects from the capture devices100a-100n. The distance of the objects may be determined by the video analytics module330. The absolute locations380gmay be a sequence of values associated with a particular detected object (e.g., each object detected may have a corresponding array of values). The sequence of values for the absolute locations380gmay indicate where the detected object is over time (e.g., to track each of the detected objects over the duration of the recording). In some embodiments, the absolute locations380gmay comprise a distance with reference to the capture device100a(e.g., how far the object is from the capture device100a). In some embodiments, the absolute locations380gmay comprise real-world location values without reference to the capture device100a(e.g., latitude, longitude, altitude, etc.).

The sensor data380hmay comprise the other sensor data generated from the sensor cluster314. The sensor data380hmay comprise sensor data that may have been captured alongside the video data370. In an example, the sensor data380hmay comprise acceleration information generated by an accelerometer, positional, rotational and/or acceleration information generated by a gyroscope, temperature information generated by a thermometer, etc. In one example, the thermometer may measure a temperature of the environment near the vehicle200a. In another example, the thermometer may measure a temperature of the processor302(e.g., when the temperature of the processor302reaches a high threshold, various operations may be scaled back such as real-time video analytics in order to prevent damage to the processor302). The sensor data380hmay provide information about the environment associated with the time that the video data370has been captured. The type of information stored in the sensor data380hmay be varied according to the design criteria of a particular implementation.

The vehicle status information380nmay comprise information about the vehicle200a. The vehicle status information380nmay be data generated by various systems of the vehicle200a. In an example, the vehicle status information380nmay be communicated over a communication bus (e.g., a CAN bus) of the vehicles200a-200n. The vehicle status information380nmay be received as the input signal CAN. In an example, the vehicle status information380nmay comprise steering wheel position and movement, gas and brake pedal activation, service codes related to the vehicle engine state, tire inflation details, seatbelt engagement status, door ajar sensor warnings, etc. The type of information stored in the vehicle status information380nmay depend on the communication and/or sensor capabilities of each individual one of the vehicles200a-200n. The number and/or types of data stored in the vehicle status information380nmay be varied according to the design criteria of a particular implementation.

The object data352may comprise information about known objects. The information about known objects may be used to identify, detect, and/or compare objects in the video data370. In one example, the object data352may comprise a database of vehicle makes, models, years, colors, sizes, dimensions, etc. In another example, the object data352may comprise regional information about license plates (e.g., shape, size, font, character size, design for each state/country, etc.). In yet another example, the object data352may comprise information about signs, billboards, buildings, cell towers, and/or other objects (trees, lane dividers, pylons, pedestrians, animals, etc.). The object data352may be used in order to provide various information for the object data380e.

The video analytics module330may be configured to compare the objects detected in the video data370with the known object data352. The comparison may be used to determine the distance of the detected object from the capture device100a. For example, if the detected object is a 2018 Ford F-150, the known dimensions of a 2018 Ford F-150 in the stored in the object data352may be compared with the dimensions of the detected 2018 Ford F-150. Based on a size comparison of the known and detected dimensions, the distance of the 2018 Ford F-150 from the capture device100amay be determined. In another example, if a license plate is detected as a Michigan license plate, the size of the alphanumeric plate characters may be compared to the known size of characters for Michigan license plates to determine the distance of the license plate from the capture device100a. The types of objects and/or information about objects stored in the object data352may be varied according to the design criteria of a particular implementation.

In some embodiments, the object detection may be performed in real-time and the distance calculations for the detected objects may be performed later. In an example, the object data352may not have the capacity to store known characteristics of all objects. For example, if the dimensions of the 2018 Ford F-150 are not currently stored in the known object data352, the size comparison may be delayed. In an example, the capture device100amay send a request to the user communication device112ato retrieve dimensions for the 2018 Ford F-150. When the information becomes available to the capture device100a, the size comparison may be performed.

The metadata cache354may be configured to store the metadata extracted by the video analytics module330and/or any other metadata corresponding to the captured video data370. The metadata cache354may provide temporary storage. In some embodiments, the metadata372may not be stored long-term by the memory304. For example, the metadata372may be deleted after being successfully stored by the database130as the metadata158. Temporarily storing the metadata may increase an amount of the memory304available for storing the video data370. Some of the metadata372may be stored long-term by the memory304(e.g., the video ID380aand/or the time stamp380b). The metadata cache354may provide storage for the metadata372until the metadata372is uploaded to the database130. In some embodiments, when the database130communicates that the metadata372for a particular one of the video files350a-350nhas been successfully stored, the metadata cache354may purge the corresponding metadata372. Generally, the metadata files may be created in the metadata cache354, filled with metadata, compressed, encrypted, transferred to the user communication devices112a-112n, and then deleted from the metadata cache354(e.g., after a confirmation of successful upload to the database130has been received). In some embodiments, the metadata cache354may provide an output buffer for the metadata372that may be uploaded to the database130and the metadata372stored with the video files350a-350nmay be stored long-term.

The signal META may comprise the data that may be stored by the database130as the metadata158extracted from the captured video by the processor302. The signal META may comprise the metadata stored by the metadata cache354. The signal META may be generated by the metadata cache354and transmitted to the database130for storage and/or indexing. The wireless communication module300may implement a short-range, inexpensive and/or low power consumption communications protocol (e.g., Bluetooth) to communicate with the user communication device112a. The user communication device112amay forward the signal META from the circuit102ato the network60. The network60may transmit the signal META to the server120for storage in the database130.

The signal VIDEO may comprise the video data370recorded by the capture device100aand/or the circuit102a. The signal VIDEO may be generated by the wireless communication module300and transmitted to the server120and/or one of the subscriber devices110a-110n. The wireless communication module300may implement a short-range, inexpensive and/or low power consumption communications protocol (e.g., Bluetooth) to communicate with the user communication device112a. The user communication device112amay forward the recorded video to the network60.

Transmitting the video data370to the subscriber users may be a low-priority data transfer. One of the flags380ccorresponding to the video file350a-350nbeing transferred may be set to ensure the video file350a-350nis preserved at least until the data transfer is completed. Generally, the transfer of the signal VIDEO may not be initiated by the circuit102auntil the subscriber user requests the video. For example, if a user does not request a recorded video, the video file may eventually be overwritten by the capture device100a. Since the recorded video is not transmitted until requested, unnecessary transfer and/or storage of data may be reduced.

The signal CAN may comprise the vehicle status information generated by the vehicle200a. In an example, the signal CAN may comprise an input presented by the CAN bus of the vehicle200a. The information from the signal CAN may be used to generate the vehicle status information380nfor the metadata372.

The metadata158stored in the database130may comprise the video ID380aof the corresponding video350a-350nand/or store data used to derive the video ID380aand/or the capture device100a-100nthat stores the corresponding video file350a-350n. For example, the metadata158may store the time stamp380bas the time182a(e.g., when the first entry is inserted to the video files350a-350n), a user ID as the camera ID182n(e.g., to identify the user communication device112a-112ncorresponding to the capture device100a-100nthat captured the corresponding video files350a-350n), a sequence number (e.g., a sequential number incrementing from 0 to 1029 and then looping back to 0). In one example, the metadata158may be a filed named “DATA-2017-03-07-14-30-00-00-01-0099.txt” (e.g., corresponding to the first timestamp and/or GPS entry deposited Mar. 7, 2017 at 2:30:00 PM by Driver ID #0001 and a sequence number of video file 0099). The contents of the file for the video clip metadata180a-180nmay comprise a sequence of lines of data. Each line of data may comprise at least a time stamp entry, a latitude and/or a longitude (e.g., 2017 Mar. 7 14:30:00, 37.804440, −122.422874, 2017 Mar. 7 14:30:20, 37.804440, −122.422874, etc.). Some lines of data may comprise a video file name and/or status of the video file (e.g., FILE DELETED: VIDEO-2017-03-06-14-30-00-0001-0022. MP4, FILE CREATED: VIDEO-2017-03-07-14-31-40-0001-0052.MP4, etc.). The video file names of the metadata180a-180nmay correspond with the names of the video files350a-350n. The format of the metadata clips180a-180nstored in the database130may be varied according to the design criteria of a particular implementation.

The processor302may be configured to receive captured images from the sensor308and/or perform video analytics using the video analytics module330to extract the metadata. In some embodiments, the processor302may be further configured to encode the captured video and/or store the encoded video data370as the video files350a-350nin the memory304. The video files350a-350nmay be created in the memory304to implement a loop recording. The loop recording implementation may create new video files350a-350nin available (e.g., free) space in the memory304. For example, if the memory304has free space available, the newest encoded video data may be stored in the free space of the memory304(e.g., previously stored data may not be overwritten). When the memory304is filled (e.g., stored to capacity), the memory304may overwrite previously stored data with the new data.

Generally, a loop recording default storage implementation may overwrite the oldest data (e.g., the video data370having the oldest/lowest time stamp380b) first. For example, if the video data370of the video file350ais written into the memory304first and the video data370of the video file350iis written into the last of the available free space (e.g., video files350a-350itake up all the available free space), then the video data370for the next video file350jmay overwrite the video data370of the video file350a(e.g., and subsequently the video data370of the video files350b,350c, etc. may be overwritten by new video data) but the metadata372may be preserved. The loop recording default storage implementation may enable the video data370of the video files350a-350nto be stored in the memory304as portions of a continuous recording (e.g., the video data370may be continuously captured as the vehicle200is being driven and the video files350a-350nmay be the continuous video segmented into shorter clips).

The flags380cmay be configured to override the loop recording default storage implementation. The flags380cmay indicate that the video data370of one or more of the video files350a-350nshould be preserved even if the video data370of the video file would be the next (e.g., oldest) file to be overwritten in the loop recording default storage implementation. For example, if the video data370of the video file350ais written into the memory304first and the flag380cis set, the video data370in the video file350bis written into the memory304second and the flag380cis not set and the video data370of the file350iis written into the last of the available free space (e.g., video data370of the video files350a-350itake up all the available free space), then the next video file350jmay overwrite the video data370of the video file350binstead of the video data370video file350a(e.g., and subsequently the video files350c,350d, etc. may be overwritten unless a respective one of the flags380chas been set) but not the corresponding metadata372. Generally, the processor302may store the next portion of the captured video (e.g., the newest created video file350a-350n) in an empty storage location of the memory304when free space is available (e.g., one or more storage locations of the memory304are empty), and in one of the storage locations of the memory304that stores the oldest portion of the captured video (e.g., overwrite the video data370of one of the video files350a-350nhaving the oldest time stamp380b) that does not have the flag380cset if there are no empty storage locations in the memory304.

The flags380cmay be modified to enable the video data370of the video files350a-350nto be preserved in case that one of the subscriber users decides to request the video data370of one of the video files350a-350n. The flags380cmay be implemented to strike a balance between preserving the video data370that may be potentially requested, and ensuring that the memory304may have space available for recording new video data370as well as the associated metadata372.

In some embodiments, the flags380cmay be set to preserve the corresponding video data370of one of the video files350a-350nwhen the database130selects the corresponding one of the video clips180a-180nas the search result(s) in response to a search query from the subscriber users (e.g., sent using the app114aon the subscriber devices110a-110n). For example, from the search results of the query, the database130may identify the video ID(s) and/or the ID of the capture device(s)100a-100nfor the video clips180a-180nbased on the metadata158. The database130may transmit a signal to the capture device(s)100a-100ncorresponding to the ID of the capture device stored in the metadata158. The signal may identify the video IDs380acorresponding to the search result. When the signal is received by the corresponding one of the circuits102a-102n, the processor302may match the video IDs380awith the video data370of the video files350a-350nstored in the memory304and set (e.g., modify) the corresponding one of the flags380cto preserve the video. The flags380cmay be set to preserve the video data370of the video files350a-350nwhen the metadata158corresponds to the search result determined by the database130(e.g., a request for the video files350a-350nmay not be needed for preservation). By preserving the video data370of the video files350a-350nin response to a match to a search query by the subscriber user, the system50may preserve the video data370of the video files350a-350nin the event that the subscriber user later decides to request the video file. In some embodiments, the preserved video data370of the video files350a-350nmay not be requested and to prevent any unnecessary use of data (e.g., data usage limits imposed by internet and/or cellular communication providers) the signal VIDEO may not be uploaded until formally requested by the subscriber user. The video data370of the video files350a-350nmay be requested by the subscriber users using the app114a.

In some embodiments, the flags380cmay be set to preserve corresponding video data370when an event has been determined to have occurred within a range of the capture devices100a-100n. When an event has been determined to occur (e.g., a pre-scheduled notification to the system50, a notification from police, a notification from news reporters, notifications from insurance investigators, etc.), the location and/or a time may be presented to the database130(e.g., similar to a search query from the subscriber users). In an example, when the event is pre-scheduled, the processor302may store the recorded video data370of the video files350a-350nthat correspond to the event with the flags380cset to preserve. In another example, when the event is determined after the event has occurred, the database130may search the metadata158to determine the clips180a-180nthat match the event (e.g., a location, a time, a person of interest, a vehicle of interest, etc.). When the clips180a-180nthat match the event are found, the database130may find the video ID and/or the ID of the capture devices100a-100nthat potentially captured the event. A signal may be sent to the capture devices100a-100nthat potentially captured the event, and the flags380cmay be modified (e.g., set) for the video files350a-350nthat match the video IDs380aof the request signal. The flags380cmay be set to preserve the video data370of the video files350a-350nwhen the metadata158corresponds to the event, even if the video data370has not been requested. By preserving the video data370of the video files350a-350nin response to a match of the event, the system50may preserve the video data370in case that the video files350a-350nare requested later.

In some embodiments, the event may be conditions used for machine learning for autonomous driving (e.g., deep learning). For example, to provide training data for machine learning, particular situations, environments and/or circumstances may need to be analyzed. The processor302may be configured to identify particular situations, environments and/or circumstances. For example, if more training data is requested for blizzard (or flooding, or hail) conditions, the event may be when there is a blizzard (or flood, or hail) outside. In another example, if more training data is requested for identifying a type of animal, the event may be when an animal is detected. In yet another example, if more training data is requested for particular types of roadways, the event may be when pre-determined road conditions are detected. The conditions for an event may be varied according to the design criteria of a particular implementation.

When the flags380care modified to preserve the video data370, the video data370of one or more of the video files350a-350nmay be preserved for a pre-determined amount of time. For example, when the flags380care implemented as a single bit, the single bit may represent the pre-determined amount of time (e.g., one week). In another example, when the flags380care implemented as multiple bits, the multiple bits may represent an amount of time to preserve the video (e.g., the bits may be encoded to represent time). Limiting the amount of time that the video data370may be preserved may ensure that memory space is available for recording new the video data370and the associated metadata372(e.g., if too many videos are preserved, new videos may not be recorded). Limiting the amount of time that the video data370may be preserved may prevent against malicious attacks and/or spam bots (e.g., prevent attacks that request all videos to prevent new data from being recorded).

In some embodiments, the pre-determined amount of time for preserving the video data370may be configured to comply with local legislation. For example, privacy laws may limit data retention. In some embodiments, the processor302may determine a location of the capture device100and adjust the length of time for preservation based on the legislation of the region. The pre-determined amount of time for preserving the video data370may be configured to be updated. For example, as legislation is adjusted, introduced and/or repealed and/or new court decisions are provided, the pre-determined amount of time may be updated to comply with the latest legislation. The pre-determined amount of time may be varied according to the design criteria of a particular implementation. In some embodiments, the video files350a-350n(e.g., the video data370and/or the metadata372) may be purged to comply with the law. In some embodiments, the video data370may have a default amount of time set to purge the data from the memory304. The default amount of time to purge the data may be selected arbitrarily and/or based on the design criteria of a particular implementation.

In some embodiments, the system50may be implemented to aid in one or more of asset recovery (e.g., locating an asset with a large number of GPS data points from the metadata158), roadway investigation (e.g., video evidence for post mortem analysis of roadway events), motor vehicle theft (e.g., real time location of stolen vehicles), locating a person of interest and/or providing roadway video for deep learning and autonomous driving training. In an example, when locating a person of interest, a person may be located based on a large number of GPS data points (e.g., the metadata158) corresponding to the vehicle of the person of interest (e.g., often individuals are within one thousand feet of the vehicle the individual drives). In an example of providing roadway video, self-driving car developers train machine learning techniques (e.g., for deep learning) by collecting video and/or sensor streams of various scenarios and then annotate the streams (e.g., manually and/or automatically outlining and labeling various objects in each frame). In another example of providing roadway video data, an event recorded in the video files350a-350nmay comprise a particular road and/or weather type (e.g., the event may be defined as a hail storm that occurs within city boundaries, approaching a traffic signal during night, etc.). In some embodiments, the video analytics module330may have the ability to distinguish objects and the database130may be searched for video streams with particular characteristics (e.g., the hail storm). The hail storm may only last a few minutes, but assuming a large number of drivers are collecting data during the hail storm event, there may be a significant number of video streams available for download that match the criteria of a hail storm. The content of the video data370may be determined based on the information stored in the metadata372, even if the video data370is no longer available.

In some embodiments, the video data370of the video files350a-350nmay not be stored long term. For example, the metadata372may provide a sufficient amount of detail for the metadata server120to re-create the scene that was captured in the video data370. The metadata server120may use all the data sets380a-380nin the metadata372in order to generate an animation that represents the content of the video data370. Relying on the metadata372for providing an animation that represents the content of the video data370may reduce an amount of storage in the memory304since the metadata372may occupy less space than the video data370. For example, the amount of data stored for the metadata372in order to re-create a scene similar to the content of the video data370may be less than the amount of data stored for the scene in the video data370.

Referring toFIG.6, a diagram illustrating video analysis performed on an example video frame captured by the camera is shown. An example video frame400is shown. The example video frame400may be an example of a video frame captured by one of the capture devices100a-100nand/or stored as part of the video data370. In the example shown, the video frame400may comprise a captured view of an urban area.

The vehicle200is shown in the example video frame400. For example, one of the capture devices100a-100n(e.g.,100a) may be a dash-mounted camera located within the vehicle200directed towards the windshield of the vehicle200. The portion of the vehicle200shown may be a hood402. Generally, the dashcam100amay be mounted in a static position with respect to the vehicle200. Since the location of the dashcam100amay not move (or may have minimal movement) with respect to the vehicle200, the hood402may be located in the same space of each captured video frame.

The example video frame400may comprise the hood402, a middle section404and an upper section406. The hood402may be located at a bottom edge of the video frame400. Generally, the middle section404may comprise roadway. Generally, the upper section406may comprise the sky and buildings.

Various objects410a-410fare shown in the example video frame400. The various objects410a-410fmay be detected by the video analysis module330. As an illustrative example, boxes are shown around the detected/recognized objects410a-410f. In the example shown, the object410amay be a vehicle, the object410bmay be a vehicle (e.g., a utility van), the object410cmay be a group of pedestrians, the objects410d-410emay be stop lights, and the object410fmay be a road sign (e.g., a one-way sign). Other objects may be detected such as curbs, street markings, buildings, billboards, lampposts, sidewalks, lane markers, etc. The number and/or type of objects detected by the video analysis module330may be varied according to the design criteria of a particular implementation.

Status captions412a-412fare shown. The status captions412a-412fmay be associated with the detected objects410a-410f. The status captions412a-412fmay represent some of the information that the video analysis module330may tag for the detected objects410a-410f. In the example shown, the status caption412amay indicate that the object410ahas been identified as a vehicle, is located at a particular coordinate in the video frame400(e.g., an X coordinate, a Y coordinate location and depth/distance from the camera system100) and/or status information about the detected object410amay be stored (e.g., make/model of vehicle, whether the object is moving or stationary, color, size, license plate number, presence of decorations such as bumper stickers and/or truck nuts, etc.). In another example, for the detected object410e, the object may be detected as a traffic light and the status may indicate that the light is green. In yet another example, for the detected object410f, the object may be detected as a street sign and the status may indicate that the sign is a one-way street sign. Generally, the information in the status captions412a-412fmay correspond to the information stored in the metadata158(in the database130) and the object data380e(in the metadata372stored on the camera100a).

In some embodiments, the location in the status captions412a-412fmay be a GPS location. In one example, the GPS location may be acquired by the location module340. In another example, the GPS location may be acquired from the user device112a(e.g., the smartphone112amay comprise a GPS module and provide the location to the camera100aand/or the smartphone112amay add to the metadata before uploading). In yet another example, the GPS location may be received from an OBD device of the vehicle200capable of determining GPS coordinates (e.g., stored as the vehicle status information380nand received by the signal CAN). In some embodiments, the camera100amay be configured to determine more granular location coordinates based on the location of the camera100athat captured the video frame400and using video analysis to determine distances to the detected objects410a-410fas described in U.S. patent application Ser. No. 16/106,777, filed Aug. 21, 2018, appropriate portions of which are hereby incorporated by reference.

Generally, the status captions412a-412fand/or the boxes shown around the detected objects410a-410fmay not appear in the example video frame400when stored as a recording (e.g., when the video data370is played back using a playback device). The boxes around the detected objects410a-410fand/or the status captions412a-412fmay be illustrative examples indicating the types of objects detected by the video analysis module330and/or the type of information stored as the object data380ein the metadata372that may be uploaded to the database130as the metadata158. In some embodiments, for testing purposes (e.g., in a debug mode), the boxes around the objects410a-410fand/or the status captions412a-412fmay be shown overlaid on the video frame400.

An unimportant object414and a potentially important object416are shown. In the example shown, the unimportant object414may be a cloud and the potentially important object416may be a shrub. In the example shown, there may be one unimportant object414and one potentially important object416. However, the number of unimportant objects414and/or potentially important objects416may be varied according to particular circumstances and/or the settings of the computer vision model implemented by the library module334.

Generally, the unimportant object414may be a classification of one or more objects by the video analytics module330. The unimportant object414may be an object determined by the processor302to be irrelevant to re-creating the scene in the video frame400(e.g., unimportant visual information). For example, if the video frame400corresponds to an event that may be later re-created using the system50, the presence or absence of the unimportant object414may be irrelevant. In the example shown, where the unimportant object414is a cloud, an animation that illustrates the video data370in the video frame400may not be affected by the presence of the cloud. Generally, locations of vehicles, buildings, pedestrians, animals, signs, traffic lights, trees, road markings, etc. may be important in re-creating a scene. The unimportant object414may be a trivial detail. For example, the presence or absence of the unimportant object may not affect a driving behavior of the vehicle200, the detected vehicles410a-410b, the pedestrians410c, etc. The types of objects determined to be the unimportant objects414may be determined according to the object detection libraries implemented by the detection libraries module334. Which types of the objects are the unimportant objects414may be determined through machine learning and/or from input from experts (e.g., insurance investigators may provide input about which types of objects and/or obstacles are likely to contribute to a vehicle accident). In some embodiments, information about the unimportant objects414may not be included in the metadata372. In some embodiments, information about the unimportant objects414may be used for general information (e.g., the metadata372may include information that the video data370corresponds to a cloudy day based on the unimportant cloud detected), but not necessarily record the location information about the unimportant objects414. The types of the unimportant objects414may be varied according to the design criteria of a particular implementation.

The potentially important object416may be a classification of one or more objects by the video analytics module330. The potentially important object416may be an object determined by the processor302to be irrelevant to re-creating the scene in the video frame400in some circumstances (e.g., the unimportant visual information) and relevant to re-creating the scene in the video frame400in other circumstances (e.g., desired visual information). Often times the potentially important object416may be a trivial object. In the example shown, the potentially important object416may be a shrub on the sidewalk. In some scenarios, the presence or absence of an object like a shrub may not affect an event (e.g., the shrub may not have significant value, the shrub may not cause damage to vehicles, the shrub may not provide insight as to why an event occurred, etc.). However, in some scenarios, the potentially important object416may be important and/or relevant. In an example, the shrub may be expensive flowers and may be part of the damages caused by a vehicle accident.

Whether the potentially important object416is used for re-creating the scene in the video data370(e.g., whether the information about the potentially important object416is included in the metadata372) may be determined according to the computer vision model implemented by the detection libraries module334. Each of the cameras100a-100nmay implement different video analytics libraries. Some of the cameras100a-100nmay be configured to ignore the potentially important object416, while other of the cameras100a-100nmay be configured to extract the object data380eabout the potentially important object416. In one example, a police vehicle implementing one of the cameras100a-100nmay perform the video analytics using a detection library that does not bother to detect landscaping details, but a homeowner may install one of the cameras100a-100n(e.g., as a home security camera) that may perform the video analytics using a detection library that does detect the landscaping details. Since the metadata server120may store the metadata158received from multiple sources, the re-created scene may comprise the details about the potentially important object416if at least one of the cameras100a-100nrecorded the metadata372about the potentially important object416. The potentially important objects416that are determined to be unimportant may be treated as the unimportant objects414. Which of the potentially important objects416are determined to be relevant to scene re-creation and which of the potentially important objects416are determined to be irrelevant to scene re-creation may be varied according to the design criteria of a particular implementation.

In some embodiments, the cameras100a-100nmay be configured to generate automated incident reports. In one example, the video analytics module330may be configured to flag an event (e.g., a collision such as the collision between the vehicle250a-250bas shown in association withFIG.4). In another example, the driver may manually flag an event by providing an input to the camera100a(e.g., by pressing a button on the camera100aand/or using the smartphone112aconnected to the camera100a). In yet another example, the sensor cluster314may comprise an accelerometer and/or gyroscope that may detect an abrupt speed and/or direction change that may be flagged. In response to flagging the event, the cameras100amay preserve the video data370corresponding one of the video files350a-350n(e.g., including a pre-determined amount of time before and after the event, such as 30 seconds before and 30 seconds after). The camera100a(e.g., via the smartphone112a) may communicate the video data370associated with the event and the metadata372(e.g., including the sensor data380hfrom the vehicle200and/or the camera100a) to the metadata server120. In some embodiments, the metadata372alone may provide sufficient information to enable the metadata server120to generate an animation that may be used for the incident report.

In some embodiments, the metadata server120may overlay the metadata158on the video corresponding to the event. In some embodiments, the processor302may be configured to overlay the metadata on the video corresponding to the event before communicating the video file to the metadata server120. The data overlay may be similar to the status captions412a-412f. For example, the automated incident report may comprise the location and/or speed data as well as showing other vehicles and/or pedestrians that may have been involved (e.g., to indicate which party may have crossed illegally, signaled incorrectly and/or behaved improperly). The annotated data of the status captions412a-412fmay provide additional information to insurance investigators and/or police to see the status of the traffic light410e, license plates and/or make/model of the vehicle410a, determine an identity of the pedestrians410c(e.g., potential witnesses), etc. In an example, the camera100amay be configured to generate a file (e.g., a PDF file) as the incident report that may reside in the memory304(e.g., an expandable SD-type card slot) and/or be downloaded to the user device112a) to enable the incident report to be communicated later when internet connectivity is available.

In the example shown, most of the objects410a-410fmay be located in the middle section404of the video frame400. For example, the vehicle410a, the vehicle410b, the pedestrians410c, the street light410dand/or the sign410fmay be on, or slightly above the roadway in the middle section404. Two objects (e.g., the traffic light412eand the cloud414) may be located in the upper section406. No objects may be detected in the portion of the video frame400corresponding to the hood402. In the example shown, the hood402may comprise reflections (e.g., of the objects410a-410c). If the video analysis module330interprets the reflections on the hood402as objects, the results of the object detection and/or the associated metadata may be incorrect and/or misleading.

The video analysis module330may be configured to perform hood detection. For example, one of the library modules for object detection may be configured to detect where the hood402is located in the video frame400. The video analysis module330may dynamically determine the location of the hood402in video frames in the input buffer332based on hood detection. For example, the cameras100a-100nmay be installed by the consumer and the field of view captured by each camera100a-100nmay be slightly different based on the installation. A varying amount of hood and sky may be present in the field of view of the cameras100a-100nfor each installation.

In some embodiments, the hood detection may be performed by the video analytics module330(e.g., on-camera). In some embodiments, the hood detection may be performed through cloud (or distributed) processing. For example, after the camera100ais installed, a series of photos (e.g., video frames) may be captured and transferred to the cloud processing service. The video frames may be analyzed on the cloud server side to detect where the view of the hood starts in the video frame. Generally, the field of view does not change after installation. In one example, a still image may be captured and sent periodically to audit if the field of view has changed (e.g., and the location of the hood402).

In some embodiments, general statistics may be collected from each of the cameras100a-100nin the system50and/or by the processor302. The statistics may be analyzed to learn and/or understand how many license plates and/or other objects are typically collected for each time period and general location. For example, if the number of objects detected dramatically falls, then the cameras100a-100nmay generate an alert along with a still image or video file information for an automated (e.g., hood detection) and/or manual analysis. For example, the alert may be used to determine if one or more of the cameras100a-100nhave been damaged or has become dislodged so that the field of view is no longer aligned to position the location of the hood402to a particular area of the frame buffer332.

Metadata extraction from video generated from one of the dash cameras100a-100nmay take advantage of inherent characteristics of the scene to limit the amount of spatial data used from the input video frame400as input to the input frame buffer332used for video analytics. By limiting the amount of spatial data input to the working buffer332the inherently limited system resources of the real time system may be allocated more efficiently. A typical scene captured from the dash cameras100a-100nmay have about 15% to 25% hood surface captured as well as 15% to 25% sky. The hood402is typically unimportant for the collection of roadway data. The reflections off the hood surface402may further complicate any analysis that is using the bottom of the video frame400to look for relevant objects. The sky region may have decreasing importance for roadway data as the top of the video frame400is reached. Details of the detection libraries for the video analytics module and/or hood detection may be described in association with U.S. patent application Ser. No. 16/156,254, filed on Oct. 10, 2018 and U.S. patent application Ser. No. 17/568,746, filed on Jan. 5, 2022, appropriate portions of which are incorporated by reference.

Referring toFIG.7, a diagram illustrating an example animation generated in response to a scene re-creation based on metadata is shown. A re-created scene450is shown. The re-created scene450may be a still image of an animation created by the metadata server120. The re-created scene450may be generated in response to the metadata158stored in the database130. The metadata158may be stored in the database130in response to the cameras100a-100nuploading the metadata372associated with the video files350a-350nvia the signal META.

In the example shown, the re-created scene450may comprise an overhead view. The overhead view in the re-created scene450may be re-created from the metadata associated with the video frame400shown in association withFIG.6. Using only the metadata158(e.g., without any of the video data370), the metadata server120may generate an animation of the re-created scene450that may provide information similar to the visual data in the video frame400. Since the metadata158stored in the database130is not video data, the re-created scene450may provide an alternate perspective than the source video frame400or the same perspective as the source video frame400. In the example shown, metadata server120may have created the overhead view in the re-created scene450from the first-person perspective of the video frame400. In some embodiments, the re-created scene450may be generated in response to the metadata372generated in response to the video data370captured by the multiple camera systems100a-100d(e.g., the trawl mode that captures the video data370all around the vehicle200, as shown in association withFIG.3). Generally, the re-created scene450may be generated based on the information in the metadata158, regardless of the perspective of the cameras that captured the video data370that was used to generate the metadata372.

The cameras100a-100nmay be configured to recognize the various objects of interest and convert the information about the objects detected into the metadata372. For example, the metadata372may comprise the absolute location data380gthat may be calculated based on the known sizes of license plates, characteristics of the vehicles detected (e.g., a 2010 Ford F150 tailgate may have a known size), etc. The metadata372may be communicated to the metadata server120as the signal META and stored as the metadata158. The metadata server120may generate the re-created scene450and/or a sequence of scenes (e.g., an animation) in response to the metadata158received over time.

The re-created scene450may comprise a road452and a road454that may form an intersection and buildings456a-456d. The roads452-454and/or the buildings456a-456dmay be generated based on the roadway in the middle section404of the video frame400. Information about the roads452-454and/or the buildings456a-456dmay be part of the metadata372generated by the cameras100a-100n. In some embodiments, the metadata server120may retrieve roadway and/or structure information based on a third-party mapping service (e.g., Google Maps, a GPS/GNSS service, etc. may provide location information about the locations of roads and buildings).

The re-created scene450may comprise re-created vehicles460a-460c, re-created road markings462, re-created traffic signals464a-464cand/or re-created pedestrians466a-466b. The re-created vehicles460a-460cmay be generated from the vehicle200and/or the objects410a-410bin the video frame400. The re-created road markings462may be detected on the roadway in the video frame400. The re-created traffic signal464amay correspond to the detected traffic light410e, the re-created traffic signal464bmay correspond to the detected street sign410fand the re-created traffic signal464bmay correspond to the detected traffic light410din the video frame400. The re-created pedestrians466a-466bmay be re-created from the pedestrian group410cdetected in the video frame400.

The various re-created objects (e.g., the vehicles460a-460c, the road markings462, the traffic signals464band/or the pedestrians466a-466b, etc.) may be generated in response to the metadata372captured. For example, the re-created objects may be placed in the re-created scene450based on the information in the status captions412a-412f(which corresponds to the metadata information380a-380n), which may define the various characteristics of the re-created objects. The re-created scene450may exclude the unimportant objects414and/or any of the potentially important objects416that were determined not to be important. In the example re-created scene450, the cloud (e.g., the unimportant object414) and the shrub (e.g., the potentially important object416) have not been re-created.

Captions470a-470eare shown. The captions470a-470emay provide an illustrative example of details that may be shown for the re-created objects in the re-created scene450. In the example shown, the caption470amay indicate that the re-created vehicle460a(e.g., the vehicle200) may be a 2019 Dodge Ram 1500 with license plate JKL 567, the caption470bmay indicate that the re-created vehicle460b(e.g., the vehicle410a) may be a 2018 Ford Taurus with license plate ABC 123, the caption470cmay indicate that the re-created vehicle460c(e.g., the vehicle410b) may be a 2020 Ford Transit with license plate XYZ789, the caption470cmay indicate that the re-created traffic signal464c(e.g., the traffic light410d) may have been a green light at the time of capture of the video frame400, and the caption470emay indicate that the re-created pedestrians466a-466b(e.g., the pedestrian group410c) may be two pedestrians crossing the road452at a crosswalk. The captions470a-470emay be configured to provide the details about the objects detected and stored in the metadata372at the time that the video data370was captured.

In some embodiments, the re-created scene450may display the re-created objects with the details in the captions470a-470e. For example, the re-created vehicle460amay appear in the animation as the 2019 Dodge Ram 1500 (e.g., the re-created vehicle460amay look similar to the 2019 Dodge Ram 1500). In some embodiments, the re-created scene450may comprise generic blocks representing the re-created objects (e.g., a wireframe view, generic polygon models, etc.). For example, the information in the captions470a-470emay be displayed as a pop up caption when a user hovers over the various objects with a mouse cursor. In some embodiments, the captions470a-470emay be displayed as a separate window with only a text presentation of the data (e.g., so the captions470a-470edo not cover the re-created scene450).

The placement of the re-created objects (e.g., the re-created vehicles460a-460c, the re-created road markings462, the re-created traffic signals464a-464c, the re-created pedestrians466a-466b) may be determined based on the absolute location data380gin the metadata372. The captions470a-470emay comprise all of the metadata372(e.g., the object data380e, the sensor data380h, the vehicle status information380n, etc.). The re-created scene450may further comprise other information about the environment (e.g., weather information such as ‘party cloud’). In some embodiments, the unimportant objects414and/or the potentially important objects416may be analyzed to provide general details (e.g., the clouds and shrubs may not be important to show in the re-created scene450but may be used for a general description such as, “cloudy, heavily landscaped region”). Various details of the re-created object may be enhanced based on known data about the objects (e.g., details of the roadway may be identified based on third-party map information, a size of street signs may be based on known attributes of street signs, road markings may be determined based on regional practices for marking roads, etc.).

A combination of the timestamp data380band the absolute location data380gin the metadata372extracted by the video analytics module330may enable the absolute locations of each of the re-created objects to be determined from moment to moment. In the example shown, the re-created scene450may be one still image. However, using the absolute locations of the detected objects over time, the re-created scene450may comprise an animation that follows the absolute locations at each timestamp. In an example, a user viewing the re-created scene450may play, pause, fast forward and/or rewind the re-created scene450similar to viewing video data. The playback of the animation of the re-created scene450may update the absolute locations of the re-created objects based on the timestamp data380b(e.g., the absolute locations data380gand/or the other metadata372may be updated at each timestamp, and each timestamp may be similar to a video frame when playing back video data). For example, playing back the animation of the re-created scene450“frame-by-frame” may update the re-created scene at each available timestamp from the metadata372. Playing back the re-created scene450timestamp by timestamp (e.g., using the metadata received over time) may provide the animation.

The re-created objects in the re-created scene450may be implemented as 3D rendered geometry. The object dimensions and/or characteristics of the detected objects may be well defined in order to accurately represent the detected objects as 3D rendered geometry to provide a visualization similar to the captured video370. Many objects viewed on and/or from the roadway have a standardized shape, form, geometry, etc. In an example, a California license plate may have a specific size (e.g., with letters of a given font and size). The license plate may have an approximate location on a vehicle that has a finite subset of makes, models, years, and colors. In an example, the video analytics module330may identify, with a high amount of accuracy, a red 2020 Ford Explorer LX with California license plate 7FVV019. Since the geometry particular to the detected vehicle may be known, along with the license plate size and/or location, a precise relative location from the camera system100may be derived. Using the onboard GPS system340, the absolute location380gof the detected vehicle in the video data370may also be derived. The metadata server120may generate the re-created object as a 3D rendered object in the re-created scene based on the known data about the particular vehicle (e.g., the dimensions of the object based on the specifications of the make/model/year of the vehicle).

Since the re-created scene450may comprise 3D rendered geometry, the re-created scene450may be viewed from various perspectives. In an example, a user may view the re-created scene450and zoom and pan around the re-created scene (e.g., similar to a free-floating camera that may be moved around to provide a desired perspective of the re-created scene450). In one example, if the vehicle200athat implements the camera system100athat provided the metadata372for the re-created scene450is following directly behind the Ford Taurus vehicle410a(e.g., the re-created vehicle460b), then the video data370may capture a perspective from the camera system100awith only the back side of the vehicle410avisible. However, in the re-created scene450, the user may be able to zoom and pan around to view the front of the re-created vehicle460band see an alternate perspective. For example, the re-created vehicle460bmay comprise more details in the re-created scene than would be available in the video data370alone.

The re-created scene450may comprise the 3D rendered geometry corresponding to the detected objects. However, since one of the camera systems100a-100nmay not capture all the details of a vehicle (e.g., if the 2018 Ford Taurus is only captured from behind), the metadata server120may be configured to interpolate one or more details of the object when creating the 3D rendered geometry. For example, the color and/or the factory details of the front of the vehicle may be interpolated (e.g., estimated based on known data about an object). However, the captured vehicle may have specific and/or unique details (e.g., scratches, dents, bumper stickers, etc.) on the front that may not be part of the known object data. In some embodiments, the interpolated details may be displayed with placeholder information (e.g., the re-created scene450may draw interpolated portions of the detected objects with hatched and/or diagonal lines instead of solid colors of the unknown details). The placeholder information may provide a visual communication to a user that particular details may be interpolated. In some embodiments, the metadata372uploaded by other of the camera systems100a-100nmay be used to fill in the information. For example, multiple of the camera systems100a-100nmay each provide different perspectives of the same scene at the same timestamp and the metadata server120may fill in details for the re-created objects by aggregating the object data380efrom multiple sources (e.g., the vehicle200aimplementing the camera system100amay capture details about the re-created object460bfrom the rear while the vehicle200bimplementing the camera system100bmay capture details about the re-created object460bfrom the front at the same time).

In some embodiments, the camera system100amay track the same objects over time to continually fill in details about the detected objects. In an example, a license plate number may be partially obscured by a trailer hitch at one timestamp, but at a later timestamp, the vehicle200may have moved to enable the camera system100ato capture a perspective that shows the previously obscured license plate number. By tracking the objects over time, the video analytics module330may fill in details that may be missing at one timestamp with the details captured at another timestamp (e.g., the details may be accumulated over time). In another example, the camera system100amay provide the video data370of the vehicle200bfrom behind (e.g., would not provide details about a bumper sticker on the front of the vehicle and a cracked windshield). Eventually the camera system100amay pass the vehicle200band capture the bumper sticker and cracked windshield. The camera system100amay extract the metadata372from the view of the front of the vehicle200bto fill in the details about the bumper sticker and cracked windshield in the object data380e. A level of confidence may be associated with details tracked over time (e.g., an indication of certainty about whether the details belong to the same object or a different object). The visual placeholder for interpolation may be adjusted based on the level of confidence (e.g., yellow hatched lines for a medium level of confidence and red hatched lines for a low level of confidence). The level of confidence may decrease the greater the time difference between the captured data (e.g., a higher confidence level for the captured data that is a few seconds or minutes apart and a lower level of confidence for the captured data that is a day or a week apart).

Referring toFIG.8, a diagram illustrating communicating a scene re-creation to a driving advisor is shown. A communications dataflow500is shown. The communications dataflow500may comprise the network60, the user device112i, the metadata server120, the vehicle200iimplementing the camera system100iand an operator location502. The communications dataflow500may comprise other devices and/or nodes (not shown). In an example, more than one of the camera systems100a-100nand/or more than one of the users devices112a-112nmay be implemented. The number, type and/or arrangement of the devices and/or nodes in the communications dataflow500may be varied according to the design criteria of a particular implementation.

The camera system100iis shown capturing the video data370of the environment near the vehicle200i. In the example shown, the camera system100imay capture multiple views around the vehicle200i. In one example, the camera system100imay comprise a 360 degree camera configured to generate an all-around view (e.g., capture video data in all directions around the vehicle200i). In another example, the camera system100imay comprise multiple cameras that may each capture a different direction around the vehicle200i. The camera system100iis shown receiving a signal (e.g., SEN) and the signal CAM. The signal SEN may be the sensor data generated by the sensor cluster314. The signal SEN and the signal CAN are shown for illustrative purposes. Generally, the signal CAN and the signal SEN may be internal to the vehicle200iand/or the circuit102i.

The camera system100imay generate the metadata372and communicate the metadata372via the signal META_I. Since the amount of data in the signal META_I may be small (e.g., small compared to communicating the video data370), the signal META_I may be communicated to the user device112i. In an example, the signal META_I may be transmitted via Bluetooth (or another short-range, low-power communications protocol). In the example shown, the user device112imay be a smartphone. The smartphone112imay forward the signal META_I to the network60. The smartphone112imay communicate with the network60via a higher power communications protocol than the communications protocol between the camera system100iand the user device112i. In an example, the camera system100imay be implemented without Wi-Fi to keep costs low for the camera system100i, and the user device112imay use Wi-Fi to communicate with the network60.

The metadata server120may receive the metadata signal META_I generated by the camera system100ivia the network60. The metadata server120may receive the metadata372from each of the metadata signals META_A-META_N. For example, each of the camera systems100a-100nimplemented in the vehicles200a-200n(not shown) may independently generate the metadata372. The metadata372from the camera systems100a-100nmay provide different perspectives of a same location and time and/or metadata corresponding to the video data370captured at different times and/or different locations. The signals META_A-META_N may be generated by the camera systems100a-100n, respectively. The metadata server120may store the metadata372generated by each of the camera systems100a-100nin the database130as the metadata158(as shown in association withFIG.2).

The metadata server120may comprise the database130, a block (or circuit)510and/or a block (or circuit)512. The circuit510may implement a re-creation processor. The circuit512may implement known object data. The metadata server120may comprise other components (not shown). In an example, the metadata server120may comprise memory, hardware acceleration processors (GPUs), etc. that may be used in combination with the re-creation processor510. The number, type and/or arrangement of the components of the metadata server120may be varied according to the design criteria of a particular implementation.

The re-creation processor510may be configured to re-create various scenes from the metadata158without using the video data370. In an example, the database130may only store the metadata158and the re-creation processor510may only have access to the metadata158. In an example, the re-creation processor510may generate the re-created scene450shown in association withFIG.7.

The re-creation processor510may be configured to generate the 3D geometric rendering of a re-created scene and/or an animation. The re-creation processor510may interpret the information in the metadata158in order to re-create a scene for particular timestamps at a particular location. In some embodiments, the re-creation processor510may re-create a scene in response to the metadata372from one of the camera systems100a-100n. For example, the re-creation processor510may generate one re-created scene from the metadata372communicated by the camera system100ain the signal META_A, another re-created scene from the metadata372communicated by the camera system100bin the signal META_B, another re-created scene from the metadata372communicated by the camera system100cin the signal META_C, etc.

In some embodiments, the re-creation processor510may be configured to aggregate all the metadata158to generate the re-created scenes based on multiple sources of metadata. For example, each re-created scene may correspond to a particular location at a particular time. The timestamp data380band the absolute location data380gmay be used to determine which of the metadata clips180a-180nare applicable to a particular time and location. By aggregating the metadata clips180a-180nfrom multiple sources, the re-creation processor510may fill out all the details in the re-created scene. For example, one of the camera systems100amay only capture a partial view of a scene, but by using the metadata from the camera system100b, the re-creation processor510may fill in any blanks (or gaps) in information using a different partial view of the same scene. Generally, the more metadata from different of the camera systems100a-100navailable to the re-creation processor510, the more detailed the re-created scene generated by the re-creation processor510may be.

The re-creation processor510may generate a signal (e.g., RSCENE). The signal RSCENE may comprise the re-created scene generated in response to the metadata signals META_A-META_N. The metadata server120may communicate the signal RSCENE to an end user. The end user may be at the operator location502.

The known object data512may be configured to store known information about various types of objects. The known object data512may provide data for the re-creation processor510to create the 3D rendered geometry of the objects in the metadata158. The known object data512may comprise information about sizes, shapes and/or other features of various objects that may be detected. In an example, the known object data512may store information about a size and shape of various makes, models and years of vehicles. For example, when the metadata158indicates that a 2018 Ford Taurus is located at a particular absolute location, the known object data512may provide information about a width and length of the 2018 Ford Taurus to enable the re-creation processor510to generate the 3D rendered geometry that provides a model of the 2018 Ford Taurus in the re-created scene450. For example, the video data370may capture a rear end of the 2018 Ford Taurus and the video analytics module330may determine that the 2018 Ford Taurus is located at a particular absolute location without having a visual captured of the front end of the 2018 Ford Taurus (or being able to provide details about the length and width of the detected vehicle). The known object data512may be used to fill in the details (e.g., missing measurement data) that is not present in the metadata158based on the stored parameters about the known objects.

The known object data512may comprise generic information about the known objects. For example, the known object data512may enable the 3D rendered geometry generated by the re-creation processor510to appear the same for every 2018 Ford Taurus (or possibly with different colors) in the re-created scene. Details in the metadata158may override the generic information in the known objects data512. For example, the metadata158may indicate the red, 2018 Ford Taurus is at a particular location, and the re-creation processor510may use the generic information about the 2018 Ford Taurus to render a 3D model version of a generic red, Ford Taurus in the re-created scene. The metadata158may further indicate that the detected vehicle has a scratch on the passenger side and a bumper sticker. The details about the scratch and the bumper sticker may be unique to the particular vehicle detected by the video analytics module330. The re-creation processor510may add the unique scratch and bumper sticker to the generic model of the 2018 Ford Taurus model from the known object data512based on the information in the metadata158for the one particular vehicle (e.g., another detected Ford Taurus may use the generic information from the known object data512).

In some embodiments, the known object data512may be provided by a third-party service. In an example, each vehicle manufacturer may provide specifications about the models of vehicles that are produced. In another example, the known object data512may be crowd-sourced (e.g., volunteers may provide information about various models of vehicles). The known object data512may comprise information about traffic signs, traffic signals, map data (e.g., the locations of roads and intersections), etc. The data stored in the known object data512may be similar to information stored in the object data352. The type of data and/or amount of data stored in the known object data512may be varied according to the design criteria of a particular implementation.

The operator location502may comprise an end user520, a display device522and a communication device524. In an example, the display device522may be a computer monitor for one of the subscriber devices110a-110n. The re-created scene450is shown displayed on the display device522. The re-created scene450may be generated on the display device522in response to the signal RSCENE.

In an example, the end user520may be a coach (or a supervisor or a spotter). The re-created scene450may provide a real-time animation of the environment near the vehicle200i. For example, the metadata372generated by the camera system100imay be communicated to the metadata server120in real-time. In an example, the operator location502may be a central fleet hub for a business or service that uses a fleet of vehicles (e.g., delivery trucks, taxis, a police fleet, etc.). For example, the end user520may view the real-time feed of the re-created scene450in order to track a real-time location of the vehicle200iand/or to view the driving behavior of the vehicle200i.

The end user520may use the communication device524(e.g., a headset) to communicate real-time instructions to the driver of the vehicle200i. For example, the end user520may be a driving coach and/or a supervisor viewing the progress of the vehicle200ivia the re-created scene450. The end user520may provide user to the driver using the communication device524, while watching the re-created scene450in real-time. In an example, if the end user520is a police dispatcher, the end user520may view a crime in progress and inform the vehicle200iby communicating over the communication device524(e.g., “We are watching a stolen truck on highway101!”). In another example, the end user520may cross reference vehicles in the field of view of the camera system100iwith a bank list of vehicles that need to be repossessed (e.g., the captions470a-470ein the re-created scene450may be compared to a make, model, year and license plate data of a repossession list). In yet another example, the end user520may be a coach for a delivery truck driver and may inform the vehicle200iof potential issues (e.g., radio or phone back to the driver saying, “Watch out for that truck ahead of you, it is known to drive erratically!”). The scenario for communication between the end user520and the vehicle200ibased on the re-created scene450may be varied according to the design criteria of a particular implementation.

Referring toFIG.9, a block diagram illustrating an object detection pipeline is shown. A block diagram of the processor302is shown. The processor302may comprise the input buffer332, a block (or circuit)550, a block (or circuit)552and/or a block (or circuit)554. The circuit550may implement video processing. The circuit552may implement a video storage pipeline. The circuit554may implement an object detection pipeline. The processor302may comprise other components (not shown). The processor may receive the signal VIDEO and/or generate the signal META. The components implemented by the processor302and/or the signals sent/received by the processor302may be varied according to the design criteria of a particular implementation.

The input buffer332may receive input video frames from the signal VIDEO. The input buffer332may present the video data370to the video processing module550. The video processing module550may be configured to perform various pre-processing operations to prepare the video data for encoding and/or video analytics.

The video storage pipeline552may be an optional component. Since the re-creation processor510of the metadata server120may be capable of re-creating scenes that provide a visualization similar to the captured content of the video data370, the camera systems100a-100nmay not necessarily need to store the video data370long-term. For example, the video data370may be flushed as soon as the corresponding metadata372is stored by the memory304. Implementing the camera systems100a-100nwithout the video storage pipeline552may enable the camera systems100a-100nto be produced at a lower per-unit cost than if the video storage pipeline552is implemented. In an example, the camera systems100a-100nmay be produced at two different pricepoints (e.g., a higher pricepoint with the video storage pipeline552implemented and a lower pricepoint with the video storage pipeline552not implemented). Implementing the camera systems100a-100nwithout the video storage pipeline552may further reduce power consumption and/or heat generation. Both models of the camera systems100a-100nmay communicate the signal META comprising the processed metadata372, which may consume less bandwidth (e.g., and use less expensive radio transmission equipment).

The video storage pipeline552may comprise a block (or circuit)560and/or a block (or circuit)562. The circuit560may implement a video encoding module. The circuit562may implement a video storage module. The video storage pipeline552may comprise other components (not shown). The number, type and/or arrangement of the components of the video storage pipeline552may be varied according to the design criteria of a particular implementation.

The video encoding module560may be configured to encode and/or decode the captured video data. Encoding and/or decoding the captured video may compress the video data to enable more video data to be stored in the memory304. The encoding and/or decoding performed by the video encoding module560may be lossless and/or lossy compression. The video encoding module560may perform the encoding on the captured video data in real-time. For example, the video encoding module560may implement encoding such as H.264, H.265, VP8, VP9, Daala, etc. The video encoding module560may be further configured to encode and/or decode the captured video frames as still images. For example, the image encoding module496may implement image formats such as JPEG, JPEG-XL, BMP, PNG, WebP, SVG, etc. The type of encoding implemented by the video encoding module560may be varied according to the design criteria of a particular implementation.

The encoded video data from the video encoding module560may be presented to the video storage module562. The video storage module562may comprise an interface to the memory304. The video storage module562may organize the storage of the video data370in the memory304. In an example, the video storage module562may manage a circular buffer for storing the most recent video data and/or overwriting the oldest video data. The video storage module562may manage the flag data380cin the metadata372to determine which of the video data370to preserve.

The object detection pipeline554may be configured to generate the metadata372in response to the video data370. The object detection pipeline554may comprise the video analytics module330, a block (or circuit)570and/or a block (or circuit)572. The circuit570may implement a location analysis module. The circuit572may implement a metadata storage module. The object detection pipeline554may comprise other components (not shown). The number, type and/or arrangement of the components of the object detection pipeline554may be varied according to the design criteria of a particular implementation.

The location analysis module570may be configured to determine the absolute location of the objects detected by the video analytics module330. In an example, the video analytics module330may provide the relative locations of the objects detected and the absolute location module570may generate the absolute location data380gin response to the relative locations and the camera location data380f. The location analysis module570may be configured to determine a direction of travel based on a skew of various objects. The location analysis module570may be configured to determine the absolute location of the objects based on a comparison to the known size of the objects stored in the object data352. Details of the determination of the absolute location of the objects may be described in association with U.S. application Ser. No. 16/208,861, filed on Dec. 4, 2018 (now U.S. Pat. No. 10,984,275, issued on Apr. 20, 2021), appropriate portions of which are incorporated by reference.

The metadata storage module572may comprise an interface to the memory304. The metadata storage module572may organize the storage of the metadata372in the memory304. The metadata storage module372may be configured to manage which of the metadata372is transferred to the metadata cache354. In an example, the metadata storage module372may packetize the metadata372for communication via the signal META.

Referring toFIG.10, a diagram illustrating data management in a removable storage media for a video capture device is shown. Data management600is shown. The data management600may comprise the memory304. In the example shown, the memory304may comprise a removable storage medium. For example, the memory304may comprise a SD card (or a microSD card).

The memory304may be configured to receive the signal VIDEO (e.g., the video data370), the signal SEN (e.g., the sensor cluster data380h), the signal CAN (e.g., the vehicle status data380n) and/or a signal (e.g., OBJ). The signal OBJ may comprise the object data380egenerated by the video analytics module330. The data management600implemented by the camera systems100a-100nmay arrange the storage of the various data in the memory304.

The memory304may comprise memory portions602-606. The memory portion602may implement a circular buffer. The memory portion604may implement a video events portion. The memory portion606may implement a metadata portion. The memory304may comprise other memory portions (not shown). The number, type and/or arrangement of the memory portions of the memory304may be varied according to the design criteria of a particular implementation.

The circular buffer602may be a designated portion of the memory304for storing the video data370. The circular buffer602may provide a large capacity in order to store the new incoming video data370. Eventually, the circular buffer602may store a full capacity of the video data370. In order to continue to record new video, the circular buffer602may be configured to overwrite the oldest video data with the newest video data. Generally, the oldest video data may already have the metadata372stored and the oldest video data may no longer be needed for purposes other than viewing by an end user. The circular buffer602may comprise a large portion of the capacity of the memory304(e.g., HD or Ultra-HD video uses a lot of storage capacity). The circular buffer602may provide temporary storage for the video data370. The video data370that does not correspond to an event may be flagged for deletion (e.g., remain in the circular buffer602until overwritten by new incoming video data).

The video events portion604may comprise flagged events610a-610n. The flagged events610a-610nmay comprise video clips (e.g., portions of the video data370) that may be stored in the memory304outside of the circular buffer602. For example, the flagged events610a-610nmay be preserved instead of being overwritten in the normal course of data input for the circular buffer602. For example, the flagged events610a-610nmay correspond to the video files350a-350nthat have the flag380cset. The video events portion604may comprise a relatively large portion of the capacity of the memory304(e.g., the video clips may be HD or Ultra-HD video). The video events portion604may use less capacity than the circular buffer602. In some embodiments, the flagged events610a-610nmay be flagged automatically (e.g., by the video analytics module330). In some embodiments, the flagged events610a-610nmay be flagged manually (e.g., the driver may press a button to indicate that an important event has happened).

The metadata portion606may comprise video metadata612, vehicle data614and/or sensor data616. The video metadata612may correspond to the object data380eand/or other information about the objects detected (e.g., the absolute location data380g, the audio data380d, etc.). The vehicle data614may correspond to the vehicle status data380n. The sensor data616may correspond to the sensor data380h. The metadata portion606may be a reserved portion of the memory304configured to ensure storage space for the metadata372. Since the metadata372is significantly smaller than the video data370, the metadata portion606may be relatively smaller than both the circular buffer602and the video events portion604.

The data management600may be implemented by the video storage module562and/or the metadata storage module572. The data management600may be configured to determine how data is liberated and/or stored in the memory304. The data management600may operate differently depending on whether the camera systems100a-100n(or the respective user devices112a-112n) have internet access. In an example, the vehicles200a-200nmay be in locations that do not have internet access, and may eventually return to a home base (e.g., a fleet of taxis that has limited internet access on the road, but has internet access when returning to the garage). For example, after an eight hour shift, the data in the memory304may be transmitted to other locations (e.g., backed up). In an example, the data may be backed up either over a Wi-Fi connection or manually (e.g., by removing the storage medium and connecting to a computing device or connecting the camera systems100ito the computing device using a wired connection).

Generally, the data management600for the circular buffer602may operate the same whether or not there is internet access. The circular buffer602may provide a large and temporary storage for the video data370and overwrite the oldest video data with the newest incoming video data.

The data management600for the video events portion604may be reserved while there is no internet access. For example, the flagged events610a-610nmay not be overwritten while there is no internet access (e.g., until the flagged events610a-610nare backed up). When there is internet access, the flagged events610a-610nmay be moved to an archive for backup. The flagged events610a-610nmay be deleted from the video events portion604after the data is backed up. In an example, a person may copy the flagged events610a-610nto a local computer (or a home NAS) for archival purposes.

The data management600for the metadata portion606may be reserved while there is no internet access. For example, the video metadata612, the vehicle data614and the sensor data616may not be overwritten while there is no internet access (e.g., until the metadata372is uploaded to the database130). When there is internet access, the metadata372may be moved to the metadata server120for long term storage. The metadata server120may use the metadata uploaded to generate the re-created scene450. The video metadata612, the vehicle data614and the sensor data616may be deleted from the metadata portion606after the data is uploaded to the database130. Since the metadata372uses such a small amount of storage space, the metadata372may be preserved on the memory304for a long time (e.g., an entire eight hour shift).

The metadata server120may reproduce the captured scene using the metadata372. The data management600may further enable the flagged events610a-610nto be preserved so that some of the original video data is still available. Generally, the flagged events610a-610nmay not be uploaded to the database130unless specifically requested. In some embodiments, the metadata372may be uploaded in real time (e.g., no SD card may be implemented). In some embodiments, the camera systems100a-100nmay be implemented without the video encoding module560or the wireless communication module300(e.g., to reduce costs). Instead, the driver may manually remove the SD card periodically (e.g., once a week, or when notified by the camera systems100a-100nvia an LED, alarm, prompt, etc.) and the data could be moved from the SD card to a fleet hub archive via an SD card reader.

Referring toFIG.11, a method (or process)650is shown. The method650may perform roadway scene re-creation using video analytics and metadata. The method650generally comprises a step (or state)652, a step (or state)654, a step (or state)656, a decision step (or state)658, a step (or state)660, a step (or state)662, a step (or state)664, a step (or state)666, a step (or state)668, a step (or state)670, and a step (or state)672.

The step652may start the method650. In the step654, the camera system100may capture the video data370and the location data of the camera system100. The location data may be stored as the camera location data380f. Next, in the step656, the video analytics module330may perform the video analysis using the feature sets in the library module334in order to detect one or more objects. Next, the method650may move to the decision step658.

In the decision step658, the video analytics module330may determine whether there are any important objects. Whether an object is considered to be important or not may be determined based on the feature set(s) provided in the library module334. For example, different camera systems100a-100nthat implement different feature sets may be configured to detect different types of objects. If there are no important objects, then the method650may return to the step654. If there are important objects, then the method650may move to the step660. In the step660, the video analytics module330may identify characteristics about the objects410a-410ndetected in the video data370. Next, the method650may move to the step662.

In the step662, the video analytics module330may determine an absolute location of the detected objects410a-410n. In an example, the current location380fof the camera system100may be used as a reference point and the distance measured in the video frames to the objects may be stored as the absolute location data380g. Next, in the step664, the processor302may store the metadata372for the video files350a-350nseparate from the video data370. In an example, the processor302may store the video ID380a, the timestamp380b, audio data380d, the object data380e, the camera location380f, the absolute location data380g, the sensor data380h, the vehicle status data380n, etc. as the metadata372. In the step666, the camera system100may communicate the metadata372to the remote database130without communicating the video data370. In some embodiments, the wireless communication module300may communicate with the user device112and the user device112may send the metadata (e.g., the signal META) to the metadata server120. The metadata372communicated in the signal META may be stored as the metadata158in the database130. The video data370may be deleted after uploading the metadata372to the metadata server120. Next, the method650may return to the step654(e.g., the camera system100may continually capture the video data370and generate the metadata372) and move to the step668.

In the step668, the metadata server120may re-create a scene in response to the characteristics of the objects and the absolute location of the objects stored in the metadata158. In an example, the absolute location data380gand the object data380emay be used to re-create a scene that occurred at a particular timestamp380b. Next, in the step670, the re-creation processor510may generate the animation450based on the scene described in the metadata158received over a period of time. In an example, the re-creation processor510may generate a single (e.g., static) scene corresponding to one timestamp. In another example, the re-creation processor510may generate a sequence of scenes (e.g., an animation) corresponding to a span of timestamps received over time. In some embodiments, the animation may be generated after the video data370has been deleted. Next, the method650may move to the step672. The step672may end the method650.

Referring toFIG.12, a method (or process)700is shown. The method700may re-create a scene using metadata from multiple camera systems to fill in details. The method700generally comprises a step (or state)702, a step (or state)704, a decision step (or state)706, a step (or state)708, a decision step (or state)710, a step (or state)712, a decision step (or state)714, a step (or state)716, a step (or state)718, a step (or state)720, and a step (or state)722.

The step702may start the method700. In the step704, the metadata server120may store the metadata372received from multiple of the camera systems100a-100nin the database130. For example, the signals META_A-META_N may each be communicated by a different one of the camera systems100a-100nand the metadata server120may store the information in the signals META_A-META_N as the metadata158(e.g., as the clips180a-180n). Next, the method700may move to the decision step706. In the decision step706, the re-creation processor510may determine whether a request to re-create a scene has been received. In an example, one of the subscriber devices110a-110nmay request a scene to be re-created that occurred at a particular time and location (e.g., for an insurance claim, for a criminal investigation, etc.). If there has been no request, then the method700may return to the step704. If there has been a request, then the method700may move to the step708. In the step708, the re-creation processor510may search the database130for the metadata158that corresponds to the requested time and location. In an example, the clips180a-180nmay comprise information about the time182aand the location186bfor each of the objects184a-184n. Next, the method700may move to the decision step710.

In the decision step710, the re-creation processor510may determine whether there is any metadata stored in the database130for the requested time and location. For example, the re-creation processor510may compare the time182aand the location186bstored for the clips180a-180nwith the time and location of the request. If there is no metadata that matches, then the method700may move to the step722. If there is metadata that matches, then the method700may move to the step712. In the step712, the re-creation processor510may interpret the metadata158to re-create the scene. In an example, one of the camera systems100a-100nmay have provided the metadata372that corresponds to the requested time and location. Next, the method700may move to the decision step714.

In the decision step714, the re-creation processor510may determine whether the metadata158comprises metadata of the scene from more than one of the multiple camera systems100a-100n. If there is only one of the camera systems100a-100nthat provided the metadata372that corresponds to the requested time and location, then the re-creation processor510may move to the step718using the metadata from the single source. If there is metadata from more than one of the camera systems100a-100n, then the method700may move to the step716. In the step716, the re-creation processor510may fill in details using the multiple sources of metadata. In an example, the re-creation processor510may generate the scene using the metadata158stored from one of the camera systems100a-100nand then fill in extra (or missing) details that may be provided by another of the camera systems100a-100n(e.g., details about a front end of a car provided by a second camera may be used to fill in the details that another camera system that captured the same vehicle from behind did not provide). The re-creation processor510may combine the metadata provided by one camera system100awith the metadata provided by another camera system100b. Next, the method700may move to the step718.

In the step718, the re-creation processor510may re-create the scene450using the stored metadata158, without using the video data370. For example, the camera systems100a-100nmay have never uploaded the video data370that corresponds to the re-created scene to the metadata server120. Next, in the step720, the metadata server120may communicate the animation (e.g., via the signal RSCENE) to the subscriber that provided the request. Next, the method700may move to the step722. The step722may end the method700.

Referring toFIG.13, a method (or process)750is shown. The method750may use known object data to generate 3D rendered geometry to represent objects in the metadata. The method750generally comprises a step (or state)752, a step (or state)754, a step (or state)756, a step (or state)758, a decision step (or state)760, a step (or state)762, a step (or state)764, a decision step (or state)766, a step (or state)768, a step (or state)770, and a step (or state)772.

The step752may start the method750. In the step754, the re-creation processor510may interpret the metadata158. Next, in the step756, the re-creation processor510may determine the placement of the re-created objects (e.g., the re-created vehicles460a-460c, the re-created traffic signals464a-464c, the re-created pedestrians466a-466b, etc.) based on the absolute location data380g. In the step758, the re-creation processor510may search the known objects data512. Next, the method750may move to the decision step760.

In the decision step760, the re-creation processor510may determine whether the known objects data512has any details about the detected objects. In an example, the known objects data512may store dimensions and/or other characteristics about a particular make/model/year of a vehicle. If there are details in the known objects data512, then the method750may move to the step762. In the step762, the re-creation processor510may fill in the characteristics of the objects in the metadata158using the known objects data512. In an example, if the metadata provides information about a front end of a vehicle and does not provide the size of the vehicle, then the information from the known objects data512may be used to fill in the size and/or details about what the rear end of the vehicle looks like. The re-creation processor510may combine the details in the metadata158, with the general information about the objects provided by the known objects data512. Next, the method750may move to the step764. In the decision step760, if there are no details about the detected objects in the known objects data512, then the method750may move to the step764using only the information in the metadata158. In the step764, the re-creation processor510may generate the 3D rendered geometry of the characteristics of the objects provided in the in the metadata158. Unknown details (e.g., information not available in the metadata158or in the known object data512) may be rendered in wireframe, using a plain texture, using a hatched texture and/or another generic indicator. Next, the method750may move to the decision step766.

In the decision step766, the re-creation processor510may determine whether the user520viewing the re-created scene450has provided input to change the view. In some embodiments, the signal RSCENE may provide the full 3D, freeform view of the animation to one of the subscriber devices110a-110nand the input may be received by the subscriber devices110a-110n. If the user520has provided input, then the method750may move to the step768. In the step768, the re-creation processor510(or one of the subscriber devices110a-110nthat received the signal RSCENE) may provide an alternate perspective of the re-created scene450by rotating the 3D rendered geometry of the various re-created objects. For example, the signal RSCENE may provide the re-created scene450in a format that enables a free floating camera view to be manually moved and/or rotated by the end user520. Next, the method750may move to the step770. In the decision step766, if the user520did not provide input, then the method750may move to the step770. In the step770, the re-creation processor510and/or the subscriber devices110a-110nthat received the signal RSCENE may playback the animation of the re-created scene (e.g., the 3D rendered geometry of the location representing a particular span of time). Next, the method750may move to the step772. The step772may end the method750.

Referring toFIG.14, a method (or process)800is shown. The method800may manage data in a storage medium to preserve metadata. The method800generally comprises a step (or state)802, a step (or state)804, a decision step (or state)806, a step (or state)808, a step (or state)810, a decision step (or state)812, a step (or state)814, a step (or state)816, a decision step (or state)818, a step (or state)820, a step (or state)822, a step (or state)824, and a step (or state)826.

The step802may start the method800. In the step804, the processor302may overwrite the oldest data in the circular buffer602with the incoming video data370. Next, the method800may move to the decision step806. In the decision step806, the processor302may determine whether an event has been flagged. If an event has been flagged, then the method800may move to the step808. In the step808, the processor302may store the video data for the flagged events610a-610noutside of the circular buffer602(e.g., in the reserved video events portion604). Next, the method800may move to the step810. In the decision step806, if there are no flagged events, then the method800may move to the step810. In the step810, the video analytics module330may generate the metadata372for the incoming video data370. Next, the method800may move to the decision step812.

In the decision step812, the video analytics module330may determine whether there are any objects from the detection library334, in the video data370. For example, the video analytics module330may compare features detected in the video data370to the features in the feature library334. If the video data370does not comprise the objects from the detection library334, then the method800may move to the step814. In the step814, the video analytics module330may ignore the objects based on the detection rules in the detection library334. Next, the method800may move to the step816. If the detected object is in the detection library334, then the method800may move to the step816. In the step816, the processor302may store the object data380eabout the important objects in the reserved metadata portion606of the memory304. Next, the method800may move to the decision step818.

In the decision step818, the processor302may determine whether there is internet access available (or whether the memory304has been connected to a backup device). In an example, the video storage module562and/or the metadata storage module572may determine a connectivity status of the camera systems100to the metadata server120(or a backup device). If there is no internet access (or backup device) available (e.g., the database130is unavailable), then the method800may move to the step820. In the step820, the data management600implemented by the processor302may preserve the metadata372and the flagged events610a-610nin the memory304. Next, the method800may return to the decision step818. In the decision step818, if there is internet access (or a backup device) available, then the method800may move to the step822.

In the step822, the communication device300may upload the metadata372to the metadata server120and upload the flagged video events610a-610nto a backup device (e.g., a device separate from the metadata server120that provides a video archive). Next, in the step824, the processor302may delete the metadata612-616from the reserved metadata portion606and the delete the flagged video events610a-610nfrom the reserved video events portion604after the data has been successfully uploaded. Next, the method800may move to the step826. The step826may end the method800.

The designations of various components, modules and/or circuits as “a”-“n”, when used herein, disclose either a singular component, module and/or circuit or a plurality of such components, modules and/or circuits, with the “n” designation applied to mean any particular integer number. Different components, modules and/or circuits that each have instances (or occurrences) with designations of “a”-“n” may indicate that the different components, modules and/or circuits may have a matching number of instances or a different number of instances. The instance designated “a” may represent a first of a plurality of instances and the instance “n” may refer to a last of a plurality of instances, while not implying a particular number of instances.