Digitizing and mapping the public space using collaborative networks of mobile agents and cloud nodes

A networked system for providing public space data on demand, including a plurality of vehicles driving on city and state roads, each vehicle including an edge device with processing capability that captures frames of its vicinity, a vehicle-to-vehicle network to which the plurality of vehicle are connected, receiving queries for specific types of frame data, propagating the queries to the plurality of vehicles, receiving replies to the queries from a portion of the plurality of vehicles, and delivering matched data by storing the matched data into a centralized storage server, and a learner digitizing the public space in accordance with the received replies to the queries.

FIELD OF THE INVENTION

The present invention relates to a vehicle data on demand platform.

BACKGROUND OF THE INVENTION

Visibility as to what is happening on the road and its environmental surrounding helps improve the safety and efficiency of transportation infrastructures and systems. Conventional systems to gain visibility of city or state roads require expensive stationary hardware with limited reach, that collect visual and sensor-based road data. Conventional systems are expensive, and are limited in geographical coverage and data update frequency. At the same time, systems with mobile hardware have more extensive research, but are limited in their ability to capture large amounts of data and data update frequency. As such, they are unable to provide real-time insights.

It would thus be of advantage to have improved systems that are inexpensive, and that provide high quality road insight and mapping in real time.

SUMMARY

Embodiments of the present invention provide a collaborative network system, based on edge devices, such as smartphones, and cloud nodes, for digitizing and mapping the public space. Systems of the present invention leverage collaborative networks to make intelligent tradeoffs between computation and communication for high quality road insights and mapping. Systems of the present invention generate road maps and capture high-frequency localized road data in real time, by using mobile agents that capture the public space on-demand, visually and via sensors, and by using cloud-based machine learning for a thorough scene understanding. Systems of the present invention provides cities, transportation planners, third parties, drivers and other users, with insights including inter alia traffic patterns, real time vehicle routing, city dynamics and infrastructure management.

There is thus provided in accordance with an embodiment of the present invention a networked system for providing public space data on demand, including a plurality of vehicles driving on city and state roads, each vehicle including an edge device with processing capability that captures frames of its vicinity, a vehicle-to-vehicle network to which the plurality of vehicle are connected, receiving queries for specific types of frame data, propagating the queries to the plurality of vehicles, receiving replies to the queries from a portion of the plurality of vehicles, and delivering matched data by storing the matched data into a centralized storage server, and a learner digitizing the public space in accordance with the received replies to the queries.

There is additionally provided in accordance with an embodiment of the present invention a networked system for digitizing public space, including a plurality of mobile agents within vehicles, the mobile agents equipped with cameras and sensors and communicatively coupled via a vehicle network, the mobile agents continuously recording video, sensor data and metadata, and sending a portion of the recorded video, sensor data and metadata to a centralized cloud storage server, in response to receiving a query from a vehicle network server, the mobile agents including a learning machine (i) analyzing the video, sensor data and metadata to recognize objects in the video, sensor data and metadata, and (ii) determining which video, sensor data and metadata to send to the cloud, based on the received query, so as to maximize overall mutual information, and a centralized cloud storage server that receives the video, sensor data and metadata transmitted by the mobile agents, including an event classifier for analyzing event candidates and classifying events, and a query generator for directing the mobile agents to gather more information on a suspected event, via the vehicle network, and a map generator generating a dynamic city heatmap, and updating the heatmap based on subsequent videos, sensor data and metadata received by the mobile agent.

There is further provided in accordance with an embodiment of the present invention a computer-based method for providing public space data on demand, including propagating, by a vehicle network server, queries to a plurality of vehicles in communication with one another via a vehicle network, each vehicle including one or more edge devices that include cameras and other sensors, and that continuously generate videos, sensory data and metadata, transmitting a portion of the videos, sensory data and metadata to a centralized storage server, the portion being appropriate to one or more of the propagated queries, indexing and annotating the received videos, sensory data and metadata, by the centralized storage server, sensory data and metadata, and digitizing and mapping the public space, based on the indexed and annotated videos, sensory data and metadata.

For reference to the figures, TABLE I provides an index of elements and their numerals. Similarly numbered elements represent elements of the same type, but they need not be identical elements.

Elements numbered in the 1000's are operations of flow charts.

DETAILED DESCRIPTION

Proliferation of smartphones and Internet of Things (IoT) devices results in large volumes of data generated at edge devices. Access to actual field data to capture the variety and diversity of real-world situations, improves the software running on the edge devices. However, edge devices are limited in terms of their computational capabilities, to process all of their collected data in depth. In addition, edge device connectivity to the centralized servers with significantly larger computational resource availability is limited. These limitations are more acute when edge devices, such as LiDAR devices and cameras, rely on sensors that generate large volumes of data that communication networks are unable to transfer. Embodiments of the present invention implement a platform to select on-demand, the data to collect and transfer to the cloud.

Overview

Reference is made toFIG. 1, which is a simplified diagram of a data-on-demand (DoD) system, in accordance with an embodiment of the present invention.FIG. 1shows a network of vehicles100that communicate with the cloud, each vehicle including an edge device such as a smartphone.

Reference is made toFIG. 2, which is a simplified overview block diagram of a DoD system, in accordance with an embodiment of the present invention.FIG. 2shows a DoD client105, which generates a continuous stream of data such as video and sensor data. DoD client105may reside in an edge device that is located in a moving vehicle. DoD client105receives create, update and delete instructions from a query definition system110. DoD client105uploads object data into an object store115, and inserts data that matches the query instructions into a matched events system120. Object store115notifies an annotation system of objects that it stores, and annotation system125analyzes and tags the objects. A vehicle-to-vehicle (V2V) system130which communicates with DoD client105, sends fetch commands to DoD client105, and receives events from DoD client105. V2V system130inserts events that match queries into matched events system120.

Reference is made toFIG. 3, which is a simplified block diagram of client110, in accordance with an embodiment of the present invention.FIG. 3shows edge devices; namely, an inertial measurement unit (IMU)405/geographic positioning system (GPS)410, and a camera415. IMU405/GPS410and camera415feed into a neural network135. Neural network135generates data for event stream140. Event stream140passes events to DoD query engine145. DoD query engine receives queries from query definition system110, matches queries with events, and passes matched events to matched event stream150. Matched event stream150passes the matched events to matched events system120. Matched event stream150also generates references, in the form of uniform resource names (URNs), to matched assets generated by IMU405/GPS410and camera415. The matched assets are then stored in object store115.

Reference is made toFIG. 4, which is a simplified block diagram of queries definitions system110, in accordance with an embodiment of the present invention.FIG. 4shows a query definitions web user interface (UI)155, for use by a human in creating, updating and deleting queries. Query definitions are stored in a query definitions database160, which transmits the queries to client105.

Reference is made toFIG. 5, which is a simplified block diagram of matched events system120, in accordance with an embodiment of the present invention.FIG. 5shows a matched events web UI165, for enabling a human to identify matched events. The matched events are stored in a matched events database170. Matched events are also obtained from client105. Match events web UI165resolves references to the matched assets in the form of URNs for match events, and the matched assets are stored in object store115. Object store115also obtains data from client105. Annotation system125analyzes and tags the objects in object store115, and transmits the annotated objects to matched events database170.

Reference is made toFIG. 6, which is a simplified block diagram of annotation system125, in accordance with an embodiment of the present invention.FIG. 6shows objects from client105uploaded to object store115. Uploads from client105occur when (i) client-side DoD query engine145matches, as shown inFIG. 3, (ii) a V2V query engine matches, or (iii) an end-ride/post-ride event occurs. The uploaded object is passed to an object insertion notification queue175. Object insertion notification queue passes objects to annotation service180. Annotation service180tags objects and inserts them into an annotations database185. Annotation service180also provides an annotations web UI, to enable a human to provide annotations of objects. After the annotation is complete, annotation service180passes the annotated objects to matched events system120.

Reference is made toFIG. 7, which is a simplified block diagram of annotation service180, in accordance with an embodiment of the present invention.FIG. 7shows neural network135processing assets for tagging, including video and sensor data. Execution of neural network135is triggered by a message in object insertion notification queue175. Neural network generates events for DoD query engine145. The events are stored in annotations database185. DoD query engine145passes matched events to matched events database170.

Reference is made toFIG. 8, which is a simplified block diagram of V2V system130, in accordance with an embodiment of the present invention.FIG. 8shows events from client105stored in a V2V queue131. The events in V2V queue131are passed to DoD query engine145. DoD query engine passes matched events to matched events database170, and to a fetch commander195, which instructs client105to upload assets. In response to the instruction from fetch commander195, client105uploads assets to object store115.

TABLE II below shows several components of a system according to an embodiment of the present invention. Features of the system support inter alia the following applications.traffic blockers, e.g., school buses, double parking, garbage trucks;traffic analytics, e.g., sidewalk pedestrian occupancy, car count and type statistics;infrastructure mapping, e.g., traffic sign detection, traffic light detection, traffic light phase and timing estimation, missing lane marking, speed sign recognition, guardrails, out of order traffic light;parking space detection;pedestrian counting and movement detection; andpattern detection across time and changes in the patterns, e.g., density of traffic divided by hours and seasons, and changes in density due to obstacles such as construction sites.

TABLE IISystem componentsMobile agentsdata collection of visual and sensory data with policiesto maximize the overall collected mutual informationreal-time road understanding of the collected datausing efficient deep networks embedded in mobileagentmodel adaptation to mobile agent environment, such aslocation and weather conditiondata-on-demand policy: send to the cloud a compressedrepresentation of the data, based on a policy defined bythe networkoptionally, crowdsourced deep network training isapplied using a distributed back propagation algorithmthat runs on mobile agents for fine-tuning and adaptingthe embedded models to new environmentsAnnotationonline deep learning cloud-based with a human in theserviceloopactive learner component to minimize the human effortconcept creation component for learning new conceptson-the-flyvehicle-to-vehicle, and/or vehicle-to-infrastructure,and/vehicle-to-pedestrian networkdynamic city heatmaps for generation of city flowpatterns, with the probability of existence of givenobjects in a specific geographical segment in a specifictemporal range, e.g., Monday morning; the objects canbe obstacles detected by the annotation service, e.g., aschool bus, or assets, e.g., a stop sign
Implementation Details

Rules for what data to gather from edge devices are defined as collection queries, which operate on streams of data sourced from the edge devices. Collection queries can refer to a single device, to multiple nearby devices, or to an entire network. Collection queries are written using a specific grammar, which runs on both clients and servers over streams of data events. TABLE III below provides exemplary attributes on which query predicates for collection criteria can relate to.

TABLE IIIEvent attributes on which query predicatesfor collection criteria can relate tocurrent environmenttimeweatherlighting conditionsroad typeposition and motion of vehiclelatitude/longitudealtitudespeedheadingaccelerationposition and motion of other observed vehiclesdistanceposespeedaccelerationactions and maneuvers a driver is performingbrakesharp turnhard accelerationtailgatingtraffic violationcollisionlane changedetections of cameratypedistanceposeconfidence

Embodiments of the present invention:collect, annotate, analyze and sell driving data that is generated;provide a server-side environment to allow automotive customers to semi-automatically annotate and analyze at large scale the data collected from fleets; anddigitize the public space for mapping, and for smart cities.

Embodiments of the present invention offer data including inter alia frames, videos, radar, LiDAR, GPS and IMU, via an application programming interface (API). Users define characteristics of data they request using a query, and delivery of matched data to a user is performed by dropping the data into a centralized storage server that the user has access to. A data analytics tool is provided, which drills down into the data and examines aggregate statistics.

The API provides a simple query language to define the data to be collected. Queries are stored, and used to define what data is to be transferred from devices at the edge to the cloud. As shown in exemplary TABLE IV below, a query can SELECT fields. A query can include a WHERE predicate to specify criteria that the data must meet. A query can optionally specify clauses LIMIT, ORDER BY and GROUP BY, to refine what data is selected.

Exemplary queries are:get 1,000 frames containing garbage trucks every week;get 1,000 frames with bounding boxes for all vehicle types when the driver does a hard brake;get 50 hours of driving from New York in the snow;get 1,000 frames of police cars by night.

The platform components necessary to implement embodiments of the present invention are (i) a client-side platform-independent library (C++) with iOS and Android glue APIs, (ii) a server-side component responsible for managing the lifecycle for collection queries, (iii) a server-side component responsible for indexing and storing the client and server-side output streams, (iv) a server-side component annotation service responsible for indexing actual assets coming in, and (v) a server-side component responsible for indexing and resolving geo-spatial queries in a generic manner.

The client-side library (i) uses as much common code in the shared C++ library as possible, and minimizes the iOS and Android code to platform-specific operations. The client-side library is responsible for:continuously keeping the active client-side collection queries in sync with the server;executing the set of active queries, based on an input sensor stream of events (location, motion, detections), in order to match against the query, with an output stream of matching events;consuming the matched event stream, generating the asset uniform resource name (URN) to be posted to the server, and feeding back the URN to the library;syncing the output stream of matching events to the server.

The server-side component (ii) manages the lifecycle for collection queries, active and non-active, for all vehicles (single vehicle, group of vehicles, network level).

The server-side component (iii) provides a user interface (UI) to explore and query output streams, and resolves the matching assets, via the URN; i.e., to show it as a web UI.

The server-side component annotation service (iv) is responsible for:feeding the stream of incoming assets; namely, the actual frames and videos, through a classification/detection pipeline, inter alia for vehicles, traffic lights, traffic signs and pedestrians;indexing and storing the output stream, including the actual detections;providing a UI to enable exploring the output stream on the actual assets;providing a UI to enable humans for further annotate the assets manually; andstoring the human annotations.

The server-side component (v) indexes and resolves geo-spatial queries in a generic manner, where the document being indexed contains a timestamp, a latitude/longitude, and an array of (document type, confidence) tuples.

Reference is made toFIG. 9, which is a simplified flowchart of an overall DoD method1000, in accordance with the present invention. At operation1010, an edge device that takes a ride makes a decision as to which data is to be transferred. Some data is transferred a priori, from data matched based on collection strategies cached on the client, at operation1020, before the ride begins. Some data is transferred in-ride, at operation1030, during the ride, by sending messages over a vehicle network. Some data is transferred post-ride, at operation1040, after the ride is finished.

The method ofFIG. 9is implemented by the API. The API decides which data to transfer from the edge device to the cloud, when to transfer the data, and how to transfer the data. Data may be transferred post-ride, in-ride and a priori. At operation1020, for a priori transfer, data is cached on the client. Some simple selections are transformed onto DoD client collection strategies and pushed to the client device. Vision-based collection strategies, such as object classification and detection, are performed on the client side.

Reference is made toFIG. 10, which is a simplified flowchart of operation1030for in-ride data transfer, in accordance with an embodiment of the present invention. At decision1031a determination is made whether there is a new signal, corresponding to a query SELECT field, from an edge device. If so, then at operation1032the edge device sends a basic safely message to the V2V manager. At operation1033the V2V manager, in addition to normal V2V responsibilities, pushes the incoming message to a queue. The queue allows multiple consumers for the same message, and relays already consumed messages, e.g., for a given ride ID.

Multiple consumers are subscribed to this queue. Upon consuming a message, at operation1034the queue inserts and indexes the incoming message in a structured format onto an event database. The event database is preferably a column database containing all world events ever encountered while driving with the application. At operation1035the queue executes all pre-defined data-on-demand queries, using the incoming message. At decision1036a determination is made whether there is a match from any query. If so, then at operation1037the edge device marks the desired data; e.g., for a pothole, one or two seconds before the pothole is detected. At operation1038the edge device pushes the requested data onto a requested data input in-memory stack system implementation, such as Redis, which stores the desired data by ride ID and timestamp. At operation1039another process consumes from the stack, and pushes through the vehicle network manager onto an edge device that desires that data.

At operation1040, for post-ride transfer, when the ride metadata is updated, the full ride is observed, to determine if further specific data should be updated.

At operation1050, the client transfers requested data to the cloud. There are two mechanisms for transfer. In the first mechanism, after a decision is made that data is required, either post-ride, V2V message or cached, the client pushes the requested data to a centralized object storage system acting as a message inbox. In the second mechanism, if the client fails to send the message after a V2V message request, when the client uploads the ride metadata at the end of the ride, a consumer checks what outstanding messages are left in the in-memory stack. The server consumer requests the client to upload the missing data.

A consumer of the centralized storage system acting as a message inbox, triggered, for example, by observing the storage file system and generating a notification when a file changes or is added or removed from such storage file system, processes the incoming data. If applicable, the consumer removes the corresponding DoD request from the requested data input message stack. The matching data is moved out of the inbox and stored in a DoD sub-folder in the centralized object storage system. The event in the database is updated with the URN for the data in the centralized object storage system.

At operation1060, as frames enter the DoD sub-folder in the centralized object storage system, a message notification based on observing changes to the object storage system triggers automatic data processing. Reference is made toFIG. 11, which is a simplified flowchart of operation1060processing data, in accordance with an embodiment of the present invention. At operation1061, labelling is automatically performed; e.g., there is a police car in the picture. At operation1062bounding boxes are automatically generated; e.g., around pedestrians. At operation1063all metadata for the frame is stored; i.e., all dictionary fields in the query SELECT. At operation1064the event database is updated. At decision1065a determination is made whether the query requires bounding boxes. If so, then at operation1066the pre-annotated frame, by the automatic process, is sent to a review team. At operation1067the output annotated frame is also stored in the DoD centralized object storage file system sub-folder.

At operation1070the data is shared. The query statements are executed in the event database at the time units exposed in the ORDER BY clause, and the results are collated into an index file, such as JSON. The file is pushed to the customer, namely, to one or more pre-defined HTTP endpoints. The customer uses the JSON file to parse a record at a time, and extract the centralized object storage system's URN, exposed as an HTTP endpoint, which then queries the DoD HTTP server. In turn, the HTTP server retrieves the matched frame from the relevant centralized object storage file system folder.

Reference is made toFIG. 12, which is a simplified flowchart of a method1100for event insertion, in accordance with an embodiment of the present invention. As clients drive around, the cloud continuously decides what to transfer. At operation1105, a V2V worker in the client sends a basic message with position and motion data, at a continuous frequency. At operation1110the V2V manager publishes all incoming basic messages onto a V2V message queue. At operation1115a DoD processor is subscribed to the V2V message queue and consumes incoming basic messages. The DoD processor is non-interactive, and can share code with the DoD controller, but runs in its own memory and compute space.

At operation1120, for each incoming basic message, the DoD processor matches the message against the registered queries in a DoD registered queries database. The operation is similar to how stream databases run, and opposite of a normal database paradigm. Specifically, in a normal paradigm queries are executed on a data corpus to select a number of matching data records. In a stream database, each new data record is matched against the query corpus to select a number of matching queries. In practice, in a stream database, it's not the queries that are executed for every new incoming data record, but rather a dual query in the data space is run matching against a database of queries. For the present embodiment, it is only necessary to determine whether the cloud should ask the client to send data matching the incoming basic message, and it is not necessary to determine which query triggered the collection request.

At operation1125the DoD processor inserts a record into an event detection database, regardless of whether there is a match. At decision1130a determination is made whether there is a match. At operation1135, if there is a match, the DoD processor inserts an event into a frame request message queue. At operation1140, the HTTP server is subscribed to the data request message queue, and is notified of a new data request message. At operation1145, the HTTP server consumes the message and notifies the relevant client of the need to upload data. At operation1150, the client uploads the requested data, based on the policy, either immediately or when the ride ends, to folder in the centralized object storage system for incoming data. At operation1155, the centralized object storage system publishes a message notification to a data uploaded message queue in a queuing system. At operation1160the DoD processor is subscribed to the data uploaded message queue, and consumes the incoming message. At operation1165, the DoD processor performs annotation, labeling and bounding boxes for the incoming frames. At operation1170, the DoD processor stores a pointer to the processed and raw frames into the matching record in the event detection database. At operation1175, the event detection database record is automatically synced with the inverted index in the search cluster.

Reference is made toFIG. 13, which is a simplified flowchart a method1200of ride-end processing, in accordance with an embodiment of the present invention. At ride-end, the client uploads all remaining data. At operation1210the client uploads the ride skeleton to the HTTP server via HTTP. At operation1220the HTTP server stores the ride object into the in-memory stack system implementation. At operation1230stack entries are popped and inserted into the event detection database. At operation1240the event detection database records are synced to the inverted index search cluster. At operation1250the client uploads more data and their time lapse to the centralized object storage system. At operation1260regular processing resumes.

Reference is made toFIG. 14, which is a high-level dataflow diagram for a server-side environment, in accordance with an embodiment of the present invention. Shown inFIG. 14are a plurality of Internet connected devices100, a plurality of systems205-255, and a plurality of databases310-370. The systems include ride services205, vehicle-to-vehicle (V2V) network210, a centralized object storage system215, job executor220, job scheduler225, uniform resource names (URNs)230, training and annotation module240, review tool245, analytics dashboard250and exploration dashboard255. Training and annotation module240includes mobile neural network241, deep neural network242, driver score243and test model244. The databases include processing queue310, ride metadata320, data on-demand queries330, data warehouse340, analytics database350, interactive database360and inverted index search cluster370.

Job scheduler225receives, accepts and runs jobs. Jobs can be run once, at a scheduled time, at regular intervals, or continuously streamed. Each job belongs to a type, and each type defines inputs and output schema. Preferably, a manually curated dictionary captures all possible schema. Jobs determine their input dataset. Batch jobs either provide a URN to a centralized object storage file system folder containing all of the training samples, or provide the URN for a file containing URNs for all of the training samples, or directly provide a list of URNs.

Job scheduler225manages an inference environment. Job scheduler225is connected to a container management system, which are scripts monitoring and managing the lifecycle of virtual server instances, to manage environment scaling. Job scheduler225determines and deploys the appropriate inference engine; namely, container+framework+architecture+model, and triggers a data loader to start feeding. The data loader feeds samples for inference, waits for a response, and stores output into the data warehouse340. The data in the warehouse is then further indexed and made available for human analysis in an in-memory analytics database optimized for interactive queries360, in an inverted index search cluster370, analytics database350, and exposed through an analytics dashboard250and an exploration dashboard255.

The exploration dashboard255enables defining queries that filter data. Query predicates go against the data warehouse, the inverted index search cluster, or the analytics database. Query outputs are refined manually. The final output is downloaded as a CSV, containing URNs to the selected assets.

As an example, consider learning a new concept, “left turns at intersections”. The exploration dashboard is used to define and write a query joining and selecting videos within intersections that contain both detected traffic lights, and where the recording vehicle is turning left. The results are labeled samples, for the new concept. A CSV with URNs to the samples is saved onto a centralized storage file system folder. A new once job is submitted to job scheduler225that triggers model building. The result is a model that allows inference of left turns at intersections from vision data. Going forward, a job is submitted tor recurring streaming, to tag all incoming videos.

Below is provided the overall flow end-to-end for a new use case, e.g., data on demand to train a detector for left turns.

1.Go into the matched events web UI and define a sequence ofkinematics sensor data readings that describe a left-turn.2.Execute this query on the matched events web UI and fetch theunderlying assets from the object store, corresponding to thematched events.3.Insert the matched assets into a neural network training serviceand obtain as output a trained network.4.Push the trained network to the clients.5.Define in the query definitions web UI a query to match eventswhen the confidence of detection in the above network is lowerthan 50%.6.Push this query definition into the clients.7.The client feeds the camera stream into the trained network.8.The network generates detection events.9.Detection events go through the query engine.10.Events where probability <50% of being a left turn are matched.11.Corresponding assets (frames in this case) are uploaded toobject store.12.Object store file insertion raises a message in the notificationqueue.13.Triggered by the notification message, annotation service fetchesmatching asset from object store.14.Annotation service runs neural network and generates detectionevents.15.Detection events from annotation service run through DoD queryengine.16.Matched events go into database.

The V2V use case is analogous to the use case above, with two differences; namely, (i) client-side queries only match events from this client, and only require state from this client, and (ii) if network-wide state across multiple clients is required, these queries run on the V2V server.

Reference is made toFIG. 15, which is a high-level dataflow diagram for a client-side environment, in accordance with an embodiment of the present invention. Shown inFIG. 15are various sensors405-430, including an inertial measurement unit (IMU)405, a geographic positioning system (GPS)410, a camera415, a LiDAR420, a CAN425, and radar430. Also shown inFIG. 8are ride manager435, storage manger440, connection manager445, and autonomous drive and advanced driver assistance system (AD/ADAS)450. Elements405-450are components of a client library. In addition,FIG. 15shows a warning actuator455and cloud460. A key component shared between client and server is a “salience” algorithm, which selects interesting driving scenarios.

Reference is made toFIG. 16, which is a high-level architectural view, in accordance with an embodiment of the present invention.FIG. 16shows that iOS and Android edge devices communicate with V2V manager211, administrators access a DoD controller470via HTTP, and users communicate with an HTTP server500using the HTTP/2 protocol. Administrators create, read update and delete rules in the system that decide where, when and how data is to be retrieved from the clients to the cloud. DoD controller470exposes an API and UI to manage the registry of collection rules. Database330of DoD registered queries stores all the rules for collection data.

Reference is made toFIG. 17, which is a simplified diagram of an HTTP proxy for searching and retrieving frames, in accordance with an embodiment of the present invention. An HTTP/1.1 GET method is used to search and retrieve frames from the inverted index search cluster370. In order to avoid exposing the centralized object storage system215directly, a simple HTTP proxy550is put in front. The HTTP proxy is responsible for authentication using HTTP message headers.

It will be appreciated by those skilled in the art that the subject invention has widespread application to other fields of use in addition to public space management. In fact, the subject invention applied to any situation where there are edge devices with limited network connectivity and limited computing resourcing, which are thus unable to both transfer all data and analyze all data at the edge in depth. Hence, the need to a distributed and collaborative system like the present invention. As such, the subject invention is applicable to security cameras, to CCTV, to any IoT implementation, to fitness tracking devices, and to capturing edge cases; e.g., getting a knee injury while running on grass.

In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident that various modifications and changes may be made to the specific exemplary embodiments without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.