Patent ID: 12205380

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described more thoroughly from now on with reference to the accompanying drawings. Like numerals represent like elements throughout the several figures, and in which example embodiments are shown. However, embodiments of the claims can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. The examples set forth herein are non-limiting examples and are merely examples, among other possible examples. Throughout this specification, plural instances (e.g., “602”) can implement components, operations, or structures (e.g., “602a”) described as a single instance. Further, plural instances (e.g., “602”) refer collectively to a set of components, operations, or structures (e.g., “602a”) described as a single instance. The description of a single component (e.g., “602a”) applies equally to a like-numbered component (e.g., “602b”) unless indicated otherwise. These and other aspects, features, and implementations can be expressed as methods, apparatuses, systems, components, program products, means, or steps for performing a function, and in other ways. These and other aspects, features, and implementations will become apparent from the following descriptions, including the claims.

The embodiments disclosed herein describe methods, apparatuses, and systems for people and vehicle analytics on the edge. In some embodiments, a computer vision processor of a camera generates hyperzooms for persons or vehicles from image frames captured by the camera. The hyperzooms include a first hyperzoom and a second hyperzoom. The first hyperzoom is associated with the persons or vehicles. The embodiments disclosed herein perform tracking and counting of people and vehicles by a camera across a series of image or video frames. The term “on the edge” refers to edge-based computing or edge processing. In the embodiments described herein, the execution of aggregation, data manipulation, bandwidth reduction, and other logic is performed on the edge (directly on the wireless camera).

The computer vision processor tracks traffic patterns of the persons or vehicles while obviating network usage by the camera. For tracking traffic of people in an environment, the camera identifies the different detections in a frame and assigns each person an identification (ID). In subsequent frames, the camera carries forward each person's ID. If the person has moved away from the frame, that ID can be dropped. If a new person appears, the camera assigns them a fresh ID. In some embodiments, positions of the persons or vehicles are predicted by a Kalman Filter using the first hyperzoom. The persons or vehicles are detected in the second hyperzoom. The positions of the persons or vehicles are updated based on detecting the persons or vehicles in the second hyperzoom. The first hyperzoom is removed from the camera.

In some embodiments, tracks of the persons or vehicles are generated based on the updated positions. The second hyperzoom is removed from the camera. Track metadata is generated from the tracks for storing in a key-value database located on a non-transitory computer-readable storage medium of the camera. The tracks are removed from the camera. The camera receives a query from a server instance over a full-duplex communication channel. The query is encoded as a key and a message. The message specifies a characteristic of the traffic patterns. The computer vision processor generates a response to the message by filtering the track metadata using the key and the characteristic of the traffic patterns. The camera sends the response to the server instance over the full-duplex communication channel. The response is for sending to a user device by the server instance.

Some embodiments disclosed herein describe methods, apparatuses, and systems for people and vehicle analytics on the edge. In some embodiments, a computer vision processor of a camera extracts attributes of persons or vehicles from hyperzooms generated from image frames captured by the camera. The hyperzooms represent traffic patterns of the persons or vehicles. In some embodiments, hyperzooms are used to detach a moving object from the background and differentiate between the objects in a video or image frame. A hyperzoom refers to a bounding box that the camera has zoomed into to describe the spatial location of an object, e.g., a person or a vehicle. The hyperzoom is rectangular and can be specified by the x and y coordinates of the upper-left corner and the lower-right corner of the hyperzoom. In other embodiments, the camera specifies a hyperzoom by the (x,y) coordinates of the hyperzoom center, and the width and height of the hyperzoom. The camera disclosed herein generates hyperzooms to detect persons and vehicular activity from images captured by the camera. The hyperzoom that bounds each object in the image frames is used to track and detect activities of the moving objects in further frames. The hyperzooms are used to detect humans and human activities, e.g., crowd traffic.

In some embodiments, extracting attributes is performed using a feature extractor of an on-camera convolutional neural network (CNN) including an inverted residual structure. The attributes include at least colors of clothing of the persons or colors of the vehicles. The computer vision processor generates mobile semantic segmentation models of the CNN using the hyperzooms and the attributes. The computer vision processor generates an index table for a NoSQL key-value structure on cloud storage. The index table is referenced by the attributes. The camera stores the hyperzooms in the NoSQL key-value structure on the cloud storage via a server instance. The index table includes pointers to the hyperzooms in the NoSQL key-value structure. The hyperzooms are removed from the camera.

In some embodiments, the computer vision processor generates attribute analytics by executing the mobile semantic segmentation models based on the attributes while obviating network usage by the camera. The computer vision processor stores the attribute analytics in a key-value database located on a non-transitory computer-readable storage medium of the camera. The attribute analytics are indexed by the attributes. The computer vision processor receives a query from the server instance over a full-duplex communication channel. The query specifies one or more of the attributes. The computer vision processor filters the attribute analytics using the one or more of the attributes to obtain a portion of the traffic patterns corresponding to a particular person or vehicle. The camera sends the portion of the traffic patterns corresponding to the particular person or vehicle over the full-duplex communication channel for sending to a user device by the server instance.

The advantages and benefits of the methods, systems, and apparatuses disclosed herein include detecting and differentiating people that look similar to prevent the computer vision model switching IDs. The computer vision processing is performed on the camera and metadata is stored on the camera, such that no network access is needed for the image processing. The performance of the embodiments on the edge provides improvements in response times and bandwidth. Such embodiments significantly reduce network traffic and communication latencies. For example, the camera disclosed can perform computation at around 10 frames per second, which a network server connected to the camera would not be able to do because of bandwidth and networking constraints. Performing the embodiments on the edge thus provides significant improvements in response times.

The edge-based processing disclosed enhances computation speed and is able to deliver much faster response times from a performance standpoint. Moreover, the camera disclosed provides bandwidth relief. The server instances and cloud database disclosed provide improved data management, better security, and improved reliability compared to traditional methods. The edge computing camera embodiments further decrease response time and latency to respond to queries instead of going through the server instances or cloud for computation. The camera provides higher security and less risk compared to traditional methods. The embodiments also provide smaller transmission costs, greater scalability and versatility, lower infrastructure costs, and reduced data loss compared to traditional methods.

Moreover, the embodiments disclosed herein address situations when people are occluded, e.g., when a pedestrian or vehicle is hidden behind someone or something else, and situations when objects disappear and reappear in later frames. The camera disclosed herein implements carefully sized bounding boxes, such that too much background is not “captured” in the features, thus increasing the effectiveness of the people tracking algorithm. In addition, the camera can distinguish between people that are dressed similarly, e.g., at a sporting event, to prevent the extraction of similar features that could result in ID switching. The advantages of the convolutional neural network (CNN) used for machine learning (ML) in the disclosed embodiments include the obviation of feature extraction and the use of shared weight in convolutional layers, which means that the same filter (weights bank) is used for each node in the layer; this both reduces memory footprint and improves performance.

FIG.1is a block diagram illustrating an example system100for people and vehicle analytics on the edge, in accordance with one or more embodiments. The system100includes a camera128, a server132, and a user device124. The system100is implemented using portions of the example computer system1300illustrated and described in more detail with reference toFIG.13. Likewise, embodiments can include different and/or additional components, or be connected in different ways.

The server132and the user device124can communicate with many different cameras at the same time. The many different cameras can be located in different areas of a building or in different geographical locations entirely. The camera128combines edge processing and storage to provide advanced physical security across numerous sites and rapid sharing of video feeds via SMS text or weblink, such as with offsite law enforcement personnel or company management, and it enables security personnel to remotely monitor entryways, provide badges, and unlock doors without requiring IT involvement. The camera128includes a computer vision processor108and a memory card or embedded non-transitory computer-readable storage medium104. The computer vision processor108generates hyperzooms for persons or vehicles from image frames captured by the camera128. The camera can capture image frames at 10 frames per second (fps). In some embodiments, the computer vision processor108is a system-on-a-chip (SoC) that performs image processing, video encoding, and CVflow™ computer vision processing. In some embodiments, the architecture of the computer vision processor108provides deep neural network (DNN) processing for computer vision applications and the processes illustrated and described in more detail with reference toFIGS.10and11. The computer vision processor108can extract maximum image detail in high contrast scenes, e.g., to extract attributes of people and vehicles from hyperzooms.

The computer vision processor108tracks traffic patterns of the persons or vehicles on the edge while obviating network usage by the camera128. An example network adapter1312performing this function and an example network1314are illustrated and described in more detail with reference toFIG.13. The term “on the edge” refers to edge-based computing or edge processing. In the embodiments described herein, the execution of aggregation, data manipulation, bandwidth reduction, and other logic is performed on the edge (directly on the wireless camera128). Network usage is obviated because the image processing is not performed on the server132or other external device.

For edge computing, the first hyperzoom is removed from the camera128so that storage in the camera128is freed up and the processing, e.g., inFIGS.10and11is performed on the edge without sending data to the server132for processing. Thus network use is obviated. For example, “Core DB Flush Logic” is implemented on the camera128, such that as the computer vision processor108ingests image frames, analysis is performed and a list of open tracklets is maintained. Previous tracklets can be removed from memory and/or the storage medium104. In an example, a minimum query increment is 300 seconds. The camera128's logic starts a timer fired every 300 seconds and executes the flush logic to remove hyperzooms, image frames, tracks, and track visualizations from the camera128. In some embodiments, the camera128executes a script periodically (e.g., running once every day) to clean expired entries in a database on the storage medium104. For example, the camera128can have a retention period of 30 days. The data items in the database on the storage medium104can thus be periodically scanned for selective deletion.

The computer vision processor108generates tracks of persons or vehicles based on updated positions after a next hyperzoom is processed. In some embodiments, the computer vision processor108generates the tracks using deep sort or centroid tracking by comparing the similarity of detected objects with each other in each processed hyperzoom. If a person or vehicle has the same similarity metric throughout the frame, the computer vision processor108generates a track for the object throughout a sequence of hyperzooms (seeFIG.3) and retains the same identifier for the object. The consistent identifier for a particular object enables the people and vehicle counting operation.

In some embodiments, the computer vision processor108generates the tracks using the deep sort method, which increases the number of image frames processed per second to about 10 frames per second (fps). In other embodiments, centroid tracking is used. In centroid tracking, the computer vision processor108computes a centroid from coordinates of a hyperzoom. For example, the computer vision processor108uses coordinates of the hyperzoom (xmin, ymin, xmax, and ymax) and determines (x_center, y_center) coordinates for each of the detected objects in each frame. The centroid can be determined as X_cen=((xmin+xmax)/2) and Y_cen=((ymin+ymax)/2), where xmin, ymin, xmax, and ymax are the hyperzoom coordinates. The computer vision processor108determines a Euclidian distance between the second hyperzoom and the first hyperzoom. After determining the Euclidian distance, the computer vision processor108updates the position of the centroid in the frame, thereby tracking the object. The second hyperzoom is removed from the camera128to reduce the amount of data stored and enable edge processing.

In some embodiments, the computer vision processor108expresses the tracks as image positions measured in pixel units relative to the upper-left corner of the image frame. For example, in a 320×240 pixel image, the upper-left corner is at image position (0.0, 0.0) and the lower-right corner is at (320.0, 240.0). Each physical object within the image frame is assigned world coordinates measured in scaled world units (e.g., meters) relative to a specified reference frame. In some embodiments, a coordinate system or set of transformations used to convert image positions into world coordinates is used. The coordinate system is used to define the tracks for each frame of the video as the scale (image units per world unit), origin (image position of the reference frame origin), and angle (counterclockwise angle from the image x-axis to the world x-axis).

The computer vision processor108generates track metadata168from the tracks for storing in a database located on the non-transitory computer-readable storage medium104of the camera128. The track metadata168can be generated as structural metadata (e.g., metadata about containers of data, how compound objects are put together, types, versions, relationships, etc.), statistical metadata (e.g., the processes that collect, process, or produce statistical data, etc.), etc. The track metadata168can be stored in the form of a data structure including tables, fields, data types, indexes, or partitions in a relational engine, or as databases, dimensions, measures, and data mining models. In some embodiments the track metadata168defines a data model and the way it is accessed and presented to the users, with the reports, schedules, distribution lists, and user security rights.

The non-transitory computer-readable storage medium104is any lightweight, mobile, embedded memory or memory card in the camera that has rapid read/write rates, enabling edge processing. In some embodiments, a secure digital (SD) card or micro SD card is used. SD is a proprietary non-volatile memory card format developed by the SD Association (SDA) for use in portable devices. A microSD card is a removable miniaturized SD flash memory card, sometimes called T-Flash or TransFlash. TransFlash cards are electrically compatible with larger SD cards. In other embodiments, Compact Flash (CF) cards that can store 256 GB+ data with 160 Mb/second access times are used. In other embodiments, XQD memory cards designed for high speed read and write, and suitable for high resolution (4K) video recording are used. In other embodiments, Memory Sticks (MS) that provide flexible storage, such as Memory Stick Duo, Memory Stick PRO, Memory Stick PRO-HG, or Memory Stick Micro M2, are used.

The database is an embedded database to satisfy resource constraints, and is small and lightweight. The database supports frequent writes, batched reads, time-series data, and a C/C++ interface. In some embodiments, the database is a key-value database located on the non-transitory computer-readable storage medium104. A key-value database stores the track metadata168as a collection of key-value pairs in which a key (e.g., an alphanumeric string) serves as a unique ID. The key-value database or key-value store is implemented for storing, retrieving, and managing the track metadata168as associative arrays in a data structure, more commonly as a dictionary or hash table. The track metadata168is stored and retrieved using a key (e.g., an alphanumeric string) that uniquely identifies the record, and is used to find the data within the database. In some embodiments, the database is a relational database (RDB). An RDB predefines the data structure in the database as a series of tables containing fields with well-defined data types. Exposing the data types to the database program allows it to apply a number of optimizations.

In some embodiments, the database stores the tracks as byte arrays on the non-transitory computer-readable storage medium104. For example, the database can be a lightning memory-mapped database (LMDB) that stores arbitrary key/data pairs as byte arrays, has a range-based search capability, supports multiple data items for a single key, and has a special mode for appending records without checking for consistency. The database can also be used concurrently in a multi-threaded or multi-processing environment, with read performance scaling linearly by design. Multiple queries can simultaneously open and use the database, as a means to scale up performance.

In some embodiments, the database stores the tracks in immutable row partitions in sorted order. For example, the database can include an SSTable, which provides a persistent, ordered immutable map from keys to values, where both keys and values are arbitrary byte strings. Operations can be provided to look up the value associated with a specified key, and to iterate over all key-value pairs in a specified key range. A block index (stored at the end of the SSTable) can be used to locate blocks; the index is loaded into memory when the SSTable is opened.

In some embodiments, the database stores the tracks in a relational database management system (RDBMS) on the non-transitory computer-readable storage medium104. For example, the database can be a relational database that organizes data into one or more data tables in which data types may be related to each other. The relations structure the data. The queries create, modify, and extract data from the relational database, as well as control user access to the database. As described, the tracks are removed from the camera128to reduce storage requirements on the camera128after metadata generation. Thus edge processing is enabled.

The server132is implemented using computer hardware or software, e.g., using some of the components of the example computer system1300illustrated and described in more detail with reference toFIG.13. The server132provides functionality for the server instances112,116and application programming interfaces (APIs) that run on the server instances112,116. For example, the server132sends track visualizations to and retrieves them from the cloud storage. The camera132also sends queries148from the user device124to the camera128, and responses152from the camera128to the user device124. A single server132can serve multiple cameras128. In some embodiments, the tracks are stored by the camera128on a NoSQL key-value structure on cloud storage via the server instance112(sometimes referred to as a “second server instance”).

The server instances112,116are virtual machines running workloads, e.g., in the cloud. The server instance116is sometimes referred to as a “first server instance.” For example, the server instances112,116can be launched in a specified cloud infrastructure. In some embodiments, the server instance112is a collection of Structured Query Language (SQL) Server databases run by the server132. The details of each server instance112,116can be viewed on a service console which can be web-based or command-line based. The server instances112,116can be controlled or managed separately. Each SQL Server instance has its own ports, databases, and logins. The multiple server instances112,116provide advantages such as stability and backup for the camera128. The system100itself can transfer the load in case of unprecedented issues such as data loss or system crashes. For example, the server instance116can take up the work of the server instance112if affected.

In some embodiments, the NoSQL key-value structure is a proprietary NoSQL database service that supports key-value and document data structures. In some embodiments, the NoSQL key-value structure resolves version conflicts and uses synchronous replication across multiple data centers for high durability and availability. In some embodiments, the computer vision processor108generates a visualization of the tracks. The visualization provides options for displaying or viewing the object tracks. For example, the color of each track can be a constant color for all tracks, a random color from a color map can be chosen for each track, a track color can be selected based on the x velocity, y velocity, or magnitude of the velocity, etc. A number of points of each track can be changed or every point of every track can be displayed. Lines can be drawn through points and the width of the lines can be changed. The camera128stores the tracks and the visualization in the NoSQL key-value structure on cloud storage via the server instance116. The visualization is for sending to the user device124by the server instance112. For example, the data160is read from the in-memory data structure store120by the server instance112and sent to the user device124.

In some embodiments, the camera128receives a query148from the server instance116over a full-duplex communication channel. The query148is generated from information136sent by the user device124to the server instance112to initiate the query148. The server instance112pushes a message140based on the information136to the in-memory data structure store120to initiate the query148. The server instance116receives a message144including the query148from the in-memory data structure store120. In some embodiments, the query148from the server132includes a date range and an increment. In some embodiments, the query148from the server132includes a start time, an end time, and an increment. The server132can query the database on the storage medium104by fetching rows and columns of data corresponding to the query148. For example, if the query148is “startTime=100, endTime=200, increment=10,” the camera128will return the column count10 of key100,110,120, . . . ,180,190. In some embodiments, both the start time and the end time are multiples of the increment.

The queries are initiated by the user device124. For example, the user device can initiate the query148in SQL or another language to the server instance112for the server instance112to place it in an in-memory data structure store120. The in-memory data structure store120is a distributed, in-memory key-value database and can be implemented as a cache or message broker. The in-memory data structure store120supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, bitmaps, streams, and spatial indices. In another example, the query148can be initiated in a natural language for the server instance112to encode it into a structured language. In another example, the query148can be initiated using a graphical user interface as shown byFIGS.4and5, or by menu controls as inFIGS.6and7. The queries are encoded by the server instance112. For example, the queries can be encoded into SQL or another language. In some embodiments, the server instance112encodes the query string as a part of a uniform resource locator (URL) that assigns values to specified parameters. A query string can include fields added to a base URL by an API running on the server instance112.

To enable the persisting of track metadata and the attribute analytics on camera, the server132is implemented to support proactively sending messages to the camera128. In some embodiments, the server132uses a combination of a full-duplex communication channel (e.g., WebSocket) and an in-memory data structure store120(e.g., Redis) to implement and maintain a connection to the camera128. The camera128is implemented to connect to the full-duplex communication channel endpoint when the camera128is restarted or after the network connected to the camera128is disconnected and re-connected. Once the connection to the full-duplex communication channel is established, the camera128periodically pings the server132to determine whether the connection is still live. The full-duplex communication channel between the server instance116and the camera128operates according to a computer communications protocol, providing full-duplex communication over a single TCP connection. For example, the WebSocket protocol standardized by the Internet Engineering Task Force (IETF) as RFC 6455 can be used. In other embodiments, the full-duplex communication channel uses SSE, which provides a way to push data from the server132to the camera128over HTTP. The query148is a request for data or information from the database, e.g., generated as results returned by SQL or as pictorials, graphs, or complex results, e.g., trend analyses from data-mining tools.

The query148is encoded as a key and a message. The key is a string (e.g., an alphanumeric string) or array of structured data used to filter the track metadata stored in the database on the non-transitory computer-readable storage medium104. In some embodiments, the key has multiple components, specified as an ordered list. A “major” key identifies a portion of the track metadata and consists of the leading components of the key. The subsequent components are called “minor keys.” The organization can be similar to a directory path specification in a file system (e.g., /Major/minor1/minor2/). For example, the key can specify a time range of the hyperzooms, a geographic location where the hyperzooms were shot, a set of days in a week (e.g., Mondays and Wednesdays only), coordinates of a line that people or vehicles are crossing (seeFIG.9), etc. The message specifies a characteristic of the traffic patterns of people and vehicles captured by the camera128and used to generate the results164of filtering the database on the storage medium104. For example, the message can refer to a count of people, a count of vehicles, a heatmap of people or vehicles (seeFIG.7), a count of people or vehicles crossing a line in a particular direction (seeFIG.9), etc. The message can be encoded into one of several different query languages, e.g., SQL, DMX, Datalog, and AQL, etc.

In some embodiments, the computer vision processor108generates a response152to the message by filtering the track metadata using the key and the characteristic of the traffic patterns. The results164of filtering the database on the storage medium104are part of the response152. For example, the response152can contain detections of particular people and faces (seeFIG.6) or filtered results156based on clothing color, apparent sex, and the presence of backpacks. The response152can display a number indicating the number of people detected in an image frame. The response152can display lines that provide a sense of which parts of a clip contain the most people activity. The camera128transmits the response152to the server instance116over the full-duplex communication channel. The response152is for sending to a user device124by the server instance112. The user device124is a smartphone, laptop, desktop, tablet, etc., implemented using at least some parts of the computer system1300ofFIG.13. The user device124thus initiates queries, plots bar diagrams from the response152, and retrieves track visualizations from the cloud storage via the server132.

In some embodiments, the computer vision processor108extracts attributes of persons or vehicles from hyperzooms generated from image frames captured by the camera128. The extraction of features and attributes (e.g., features1212) is described in more detail with reference toFIG.12. The attributes can include clothing color, apparent sex, the presence of backpacks, a make or model of vehicles, vehicle color, etc. The hyperzooms represent traffic patterns of the persons or vehicles (seeFIG.3). In some embodiments, the extracting is performed using a feature extractor of an on-camera convolutional neural network (CNN) (seeFIG.12) including an inverted residual structure. For example, in the CNN, the shortcut connections are between thin bottleneck layers. An intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. An example feature extraction module1208is illustrated and described in more detail with reference toFIG.12.

In some embodiments, the computer vision processor108generates mobile semantic segmentation models of the CNN using the hyperzooms and the attributes. In some embodiments, the mobile semantic segmentation models cluster parts of an image frame or hyperzoom together which belong to the same object class. Semantic segmentation is a form of pixel-level prediction because each pixel in the hyperzoom image is classified according to an attribute or category. The models are referred to as mobile because they are executed on the computer vision processor108itself to provide edge computing capabilities. In some embodiments, generating the mobile semantic segmentation models includes training, by the computer vision processor, the CNN using the hyperzooms and the attributes. Training an example machine learning model1216is illustrated and described in more detail with reference toFIG.12. For example, the attributes can be used as the features1212, the hyperzooms can be used as the training data1220, and the image frames can be used as the validation set1232, as illustrated and described in more detail with reference toFIG.12. In some embodiments, generating the mobile semantic segmentation models comprises classification (making a prediction for a whole input), localization/detection (providing information regarding the spatial location of the classes), or fine-grained inference by making dense predictions inferring labels for every pixel, so that each pixel is labeled with the class of its enclosing object or region.

In some embodiments, the computer vision processor108generates an index table for a NoSQL key-value structure on cloud storage. The index table can use primary keys to uniquely identify each hyperzoom or track, and secondary indexes to provide more querying flexibility. The index table is referenced by the attributes. In some embodiments, the hyperzooms are keyed or indexed by a camera_id and time (milliseconds). More than one hyperzoom can be persisted for the same person. In other embodiments, only one hyperzoom is persisted for each tracked person. In some embodiments, the camera128stores the hyperzooms in the NoSQL key-value structure on the cloud storage via the server instance116. The index table includes pointers to the hyperzooms in the NoSQL key-value structure. For example, the index table can include a partition key and a sort key. To free up storage on the camera128, the hyperzooms are removed from the camera128.

In some embodiments, the camera executes a computer vision (CV) algorithm and generates people analytics. For example, the computer vision processor108generates attribute analytics by executing the mobile semantic segmentation models based on the attributes while obviating network usage by the camera128. After executing the mobile semantic segmentation models for people and vehicles, the camera128stores the attribute analytics in a database on the storage medium104, e.g., on a relational database such as SQLite. To reduce latency, the camera128generates indexing for the attributes, e.g., using SQLite. The camera128uploads hyperzooms to cloud storage. In some embodiments, a cross-camera search using a server132query to all cameras in the system is performed. Thus edge processing is enabled. In some embodiments, a “Person History” feature is enabled such that the camera128performs people detection as well as more complex analytics, such as the ability to search for faces, and for people by clothing color. In some embodiments, all detected people across different cameras can be displayed (seeFIG.6). In some embodiments, person-of-interest (POI) alerts can be enabled that provide the ability to identify and search for individuals in the camera footage, allowing a user device124to quickly find meaningful events and save time during critical investigations.

In some embodiments, the computer vision processor108stores the attribute analytics in a key-value database located on the non-transitory computer-readable storage medium104of the camera128. The attribute analytics are indexed by the attributes. In some embodiments, the camera128can go into a sleep mode, disconnect from the network, or be rebooted, or reset. When the camera128is active again after being woken up, reset, or rebooted, the camera128receives a signal that the network connection to the camera128is active again. An example network adapter1312performing this function and an example network1314are illustrated and described in more detail with reference toFIG.13. Responsive to receiving a power-on signal, the computer vision processor108sends a descriptor of the camera128to the server instance1116. For example, when the camera128is powered on or resumes an Internet connection, the camera128sends an HTTP request to the server132with a camera_id. In another example, the descriptor can be a geographical location where the camera128is deployed, a floor of a building, or an alphanumeric descriptor denoted by camera_id.

In some embodiments, the camera128connects to a full-duplex communication channel that is generated by the server instance116. For example, the server instance116receives a request from the camera, and initiates the full-duplex communication channel (e.g., WebSocket) between the camera128and the server based using HTTP. Further, the server instance116subscribes to a topic with camera_id as the name in the in-memory data structure store120(e.g., Redis). The full-duplex communication channel operates according to a computer communications protocol, providing full-duplex communication over a single TCP connection. For example, the WebSocket protocol standardized by the IETF as RFC 6455 can be used. In other embodiments, the full-duplex communication channel uses SSE, which provides a way to push data from the server132to the camera128over HTTP. The full-duplex communication channel generated references the descriptor and is for transmission of queries generated by the server instance112. The queries are initiated by the user device124. For example, the user device can initiate the query148in SQL or another language to the server instance112for the server instance112to place it in the in-memory data structure store120. In another example, the query148can be initiated in a natural language for the server instance112to encode it into a structured language. In another example, the query148can be initiated using a graphical user interface as shown byFIGS.4and5, or by menu controls as inFIGS.6and7. The queries are encoded by the server instance112. For example, the queries can be encoded into SQL or another language. In some embodiments, the server instance112encodes the query string as a part of a uniform resource locator (URL) that assigns values to specified parameters. A query string can include fields added to a base URL by an API running on the server instance112.

When the user device124initiates a query148, the server instance112receives it and encodes the query148as a key and a message. In some embodiments, the server instance112publishes the key and message to the in-memory data structure store120with the descriptor (e.g., camera_id) as the topic. The server instance116“listens” for the key from the in-memory data structure store120. The server instance116receives the key and the message, and sends it to the camera128, e.g., via WebSocket. The camera128receives the query148from the server instance116over the full-duplex communication channel. The query148specifies one or more of the attributes. For example, the query can specify clothing colors, apparent sex, the presence of backpacks, a particular person or vehicle (e.g., person616inFIG.6), etc. For example, a query can specify “show when Matt Peterson entered the building & all instances of a person with a red shirt in the building.”

In some embodiments, the computer vision processor108filters the attribute analytics using the one or more of the attributes to generate a response152. The response152includes a portion of the traffic patterns (e.g., results164) corresponding to a particular person or vehicle (e.g., person616inFIG.6). The filtering is in response to the query and removes redundant or unwanted information from the attribute analytics prior to generating the response152. The filtering thus increases the semantic signal-to-noise ratio. The results164of the filtering is used to generate the response152ofFIG.1(e.g., portion of the traffic patterns corresponding to a particular person or vehicle).

The camera128sends the response152to the server instance116. The response152includes the portion of the traffic patterns corresponding to the particular person or vehicle over the full-duplex communication channel for sending to the user device124by the server instance112. The query-response process is asynchronous. The process for the camera128to respond with the response152can be out of order, e.g., the camera128can receive two messages and then respond with two messages out of order. The server instance116receives the response152and adds the result in the in-memory data structure store120(e.g., Redis) using received information (e.g., an alphanumeric string). The step of adding the result in the in-memory data structure store120is also asynchronous and does not block the step of the server instance116receiving the key and the message, and sending it to the camera128. The server instance112receives the response152(e.g., including the results164) and responds to the user device124. In some embodiments, the server132implements the WebSocket protocol, using a raw WebSocket or Socket.IO.

When the camera128responds to the server instance116with a message containing the results156, the response process is asynchronous and can be out of order, i.e., the camera128can receive two or more queries and return two or more responses. For example, the query148can be a first query, and the portion of the traffic patterns (response152) is transmitted asynchronously with respect to a second query received by the camera128from the server instance116over the full-duplex communication channel. The server instance116receives the response152and adds the results156to the in-memory data structure store120using received information (e.g., an alphanumeric string). Similarly, the process of the server instance116receiving the response152and adding the results156to the in-memory data structure store120is also asynchronous and does not interfere with or block the step of the server instance116sending the queries148to the camera128. The server instance112retrieves the results156from the in-memory data structure store120and sends them to the user device124for viewing by a user.

FIG.2Aillustrates a computer program for people and vehicle analytics on the edge, in accordance with one or more embodiments. The computer program is executed by the camera128to store hyperzooms by the camera128on a NoSQL key-value structure on cloud storage via the server instance112. In some embodiments, the NoSQL key-value structure is a proprietary NoSQL database service that supports key-value and document data structures. In some embodiments, the NoSQL key-value structure resolves version conflicts and uses synchronous replication across multiple data centers for high durability and availability.

FIG.2Billustrates an example database for people and vehicle analytics on the edge, in accordance with one or more embodiments. The example database is implemented on the storage medium104to store track metadata168, counts of people and vehicles, attribute analytics, etc. The computer vision processor108generates track metadata168from the tracks for storing in the database located on the non-transitory computer-readable storage medium104of the camera128. The track metadata168can be generated as structural metadata (e.g., metadata about containers of data, how compound objects are put together, types, versions, relationships, etc.), statistical metadata (e.g., the processes that collect, process, or produce statistical data, etc.), etc. The track metadata168can be stored in the form of a data structure including tables, fields, data types, indexes, or partitions in a relational engine, or as databases, dimensions, measures, and data mining models. In some embodiments the track metadata168defines a data model and the way it is accessed and presented to the users, with the reports, schedules, distribution lists, and user security rights. When a hyperzoom is stored for each detected person, a list of person IDs is stored in the database; the hyperzooms are persisted on cloud storage keyed by person ID, as described in more detail with reference toFIGS.1,10, and11.

FIG.3is a drawing illustrating example hyperzooms for people and vehicle analytics on the edge, in accordance with one or more embodiments.FIG.3includes an environment300captured by image frames of the camera128. The camera128disclosed herein generates hyperzooms to detect persons and vehicular activity from images captured by the camera128.

The hyperzoom320refers to a bounding box that the camera128has zoomed into to describe the spatial location of an object, e.g., a person or a vehicle. InFIG.3, the object is the person324. The hyperzoom is generated from an image frame captured at time304. Similarly, hyperzoom316depicts person328, hyperzoom312depicts person332, and hyperzoom308depicts person336. Each hyperzoom is rectangular and can be specified by the x and y coordinates of the upper-left corner and the lower-right corner of the hyperzoom. In other embodiments, the camera128can specify the hyperzoom320by the (x,y) coordinates of the hyperzoom center, and the width and height of the hyperzoom320. Each hyperzoom is used to track and detect activities of the moving objects in further image frames. The hyperzooms are used to detect humans and human activities, e.g., crowd traffic.

In some embodiments, the computer vision processor108assigns an identifier to the person324in the hyperzoom320. The identifier is used to register new objects. For example, when a new object (e.g., person324) enters or the same object (person324) is detected again, the centroid tracking will register the new object with a unique ID, so that it becomes helpful for different applications. Once the object (person324) is not in the image frame, the algorithm will de-register the object ID, stating that the object is not available or left the image frame. In some embodiments, the computer vision processor108predicts positions of the person324, e.g., using a Kalman Filter from the hyperzoom320. The Kalman Filter can be executed on the computer vision processor108and uses a series of measurements observed over time (e.g., the hyperzooms) to produce estimates of unknown variables (e.g., positions of persons) by estimating a joint probability distribution over the variables for each time frame (e.g., each image frame or hyperzoom). The computer vision processor108therefore produces estimates of the current state variables (e.g., positions of persons324,328, etc.) along with their uncertainties.

The computer vision processor108further detects instances of semantic objects (e.g., humans or cars) in the image frames or hyperzooms. In some embodiments, the detecting is performed using neural network-based approaches. For example, end-to-end object detection can be performed without specifically defining features using a convolutional neural network (CNN), as described in more detail with reference toFIGS.11and12. In other embodiments, features are defined (e.g., using the Viola-Jones object detection framework based on Haar features, a Scale-invariant feature transform (SIFT), or Histogram of oriented gradients (HOG) features, etc.) using a support vector machine (SVM) to perform the classification, as illustrated in more detail with reference toFIG.12. In some embodiments, responsive to detecting the person324in the hyperzoom320, the computer vision processor108persists the identifier assigned to the person324in the hyperzoom320. The computer vision processor108associates the hyperzoom320with the person324.

FIG.4illustrates an example graphical user interface (GUI) for people and vehicle analytics on the edge, in accordance with one or more embodiments. The query148can be initiated using a graphical user interface as shown byFIGS.4and5, or by menu controls as inFIGS.6and7. The queries are encoded by the server instance112. For example, the queries can be encoded into SQL or another language. In some embodiments, the server instance112encodes the query string as a part of a uniform resource locator (URL) that assigns values to specified parameters. A query string can include fields added to a base URL by an API running on the server instance112. The query148can include search terms relating to high resolution analytics capture, people history, vehicle history, face matching and search, and person of interest notifications.

FIG.5illustrates an example GUI for people and vehicle analytics on the edge, in accordance with one or more embodiments. A query generated using the GUI shown byFIG.5can include search terms relating to high resolution image capture, face detection, and vehicle analytics.

FIG.6is a block diagram illustrating an example graphical representation of results156returned by a query, in accordance with one or more embodiments. The example graphical representation is displayed on a screen600of the user device124. The screen600of the user device124displays the graphical representation as a GUI that can be used to view the results156as well as initiate further queries. A menu bar620is displayed on the left side of the screen600to initiate queries. In the top row of the example graphical representation, a set of detected persons604,608,612is displayed. In some embodiments, intermediate results, such as inFIG.6, can be displayed. The persons shown inFIG.6have been detected from recent hyperzooms. Some of the detections are duplicates, i.e., the persons604,608are the same human. The computer vision processor108can filter out these duplicates and assign the same identifier to them, such that no duplicates are displayed. The person612is a different human than the person604,608. The user device124can be used to initiate a query searching for the known person616from among all detected persons in a particular time frame. Similarly, the bottom row inFIG.6shows functionality for searching for vehicles.

FIG.7is a drawing illustrating a heatmap700for people and vehicle analytics on the edge, in accordance with one or more embodiments. In some embodiments, to respond to the query148, the computer vision processor108generates a heatmap using the track metadata. The heatmap700is a data visualization technique that shows the density of people or vehicles as color in two dimensions. The variation in color can be by hue or intensity, giving obvious visual cues to the user device124about how the phenomenon is clustered or varies over space. In some embodiments, the heatmap provides information on where people have been on a floor plan (e.g., of the buildings704) during a selected time range. For example, a darker color (e.g., in shape708) indicates more people traffic, while a lighter color (e.g., in shape712) indicates less people traffic. The time range being viewed on the people heatmap can be changed, e.g., by moving a timeline back and forth, changing the date and time using a calendar function, or skipping forward and backward with skip buttons. The time interval can be changed using buttons on the right side of the timeline. The camera128sends the heatmap700to the server instance116over the full-duplex communication channel. The heatmap is for sending to the user device124by the server instance112.

FIG.8is a drawing illustrating an example hyperzoom for vehicle analytics on the edge, in accordance with one or more embodiments. The camera128extracts attributes of a vehicle808from the hyperzoom804generated from the image frame800captured by the camera128. The hyperzoom804represents traffic patterns of the vehicle808. The extracting can be performed using a feature extractor of an on-camera convolutional neural network (CNN) including an inverted residual structure. The attributes include at least a color of the vehicle808. The camera128generates attribute analytics by executing mobile semantic segmentation models of the CNN based on the attributes while obviating network usage by the camera128. The camera128stores the attribute analytics in a database located on the camera128. The attribute analytics are indexed by the attributes. The camera128receives a query148from the server instance116(seeFIG.1) over a full-duplex communication channel. The query148specifies one or more of the attributes. The camera128transmits a portion of the traffic patterns filtered by the one or more of the attributes over the full-duplex communication channel for sending to the user device124by the server instance112.

FIG.9is a diagram illustrating an example of line-crossing detection for people and vehicle analytics on the edge, in accordance with one or more embodiments. The camera128(seeFIG.1) implements line-crossing detection using tracklets of people or vehicles and a line924defined by the user device124. For example, a user can use a GUI to define coordinates of the line924on the user device124(seeFIG.1).

In some embodiments, the computer vision processor108generates tracks912of a person or vehicle (e.g., person904) from hyperzooms (e.g., hyperzoom908) depicting the person or vehicle. Example methods for track generation are described in more detail with reference toFIGS.1and10. The hyperzooms are generated from image frames (e.g., image frame900) captured by the camera128. For example, the person904is detected and a centroid is marked on the detected person904. The movement of the marked centroid is tracked. In some embodiments, object tracking methods, such as deep sort, centroid tracker, channel and spatial reliability tracker (CSRT), kernel correlation filter (KCF), or camshift, that track a detected object by comparing the similarity of detected objects with each other in each processed frame are used.

The computer vision processor108receives coordinates of the line924from the server instance116(seeFIG.1) over a full-duplex communication channel. The coordinates of the line924are defined by the user device124communicably coupled to the server instances112,116. The computer vision processor108defines a first parallel line916and a second parallel line920spaced from the line924and positioned on different sides of the line924. The computer vision processor108determines a first position of the person904relative to the line from the image frames captured by the camera128. For each trajectory of a person or vehicle currently in memory on the camera128, the camera128keeps track of which side of the line924the person or vehicle is on. The camera128captures an image frame of the person or vehicle. The image frame is captured after the image frames. For each detection box (e.g., hyperzoom908), the mid-point of the lower edge is tracked.

The computer vision processor108determines a second position of the person904relative to the line924from the image frame captured after the image frames. If, in the image frame, the person904is close to the line924or on the same side as before, the camera128does not record a change. If the side of the detection changes, the camera128records the crossing in the database on the storage medium104(seeFIG.1). When the tracked box crosses the line924and is farther from the line924than the parallel line920, a crossing detection is recorded. For example, the computer vision processor108determines that the person904has crossed the line924from a first side of the first parallel line916to a second side of the second parallel line920based on a difference between the first position and the second position and/or the tracks912. Responsive to determining that a first distance between the second position and the second parallel line920is smaller than a second distance between the second position and the line924, the computer vision processor108stores track metadata168describing that the person904has crossed the line924in a key-value database located on the storage medium104(seeFIG.1) of the camera128.

In some embodiments, the computer vision processor108determines the bounding box coordinates and computes a centroid for the person904. For example, the bounding box coordinates are determined as xmin, ymin, xmax, and ymax. The (xcenter, ycenter) coordinates for the person904in the image frame900are determined. The centroid is determined as: Xcen=(xmin+xmax)/2 and Ycen=(ymin+ymax)/2, where xmin, ymin, xmax, and ymax are the bounding box coordinates. The camera128determines a Euclidian distance between a new detected bounding box (from the image frame captured after the image frames) and the person904. The centroid for the person904is updated. After determining the Euclidian distance between the new detected bounding box and the last position, the camera128updates the position of the centroid in the image frame, thus tracking the person904. In some embodiments, the camera128determines a direction of centroid movement (e.g., track912). The camera128further counts the number of people crossing the line924. Based on the counting, the camera128increments a line-crossing count in the track metadata168.

The camera128receives a query148from the server instance116over the full-duplex communication channel. The query148requests a count of a number of times the person904has crossed the line924. The computer vision processor108generates a response152to the query148from the track metadata168stored in the key-value database. The camera128sends the response152to the server instance116over the full-duplex communication channel. The response152is for sending to the user device124by the server instance112.

FIG.10is a flow diagram illustrating an example process for people and vehicle analytics on the edge, in accordance with one or more embodiments. In some embodiments, the process ofFIG.10is performed by the camera128illustrated and described in more detail with reference toFIG.1. In other embodiments, the process ofFIG.10is performed by a computer system, e.g., the example computer system1300illustrated and described in more detail with reference toFIG.13. Particular entities, for example, the computer vision processor108or the server132, perform some or all of the steps of the process in other embodiments. The computer vision processor108and the server132are illustrated and described in more detail with reference toFIG.1. Likewise, embodiments can include different and/or additional steps, or perform the steps in different orders.

In step1004, the computer vision processor108of the camera128generates hyperzooms for persons or vehicles from image frames captured by the camera128. In some embodiments, the computer vision processor108is a system-on-a-chip (SoC) that performs image processing, video encoding, and CVflow™ computer vision processing. In some embodiments, the architecture of the computer vision processor108provides deep neural network (DNN) processing for computer vision applications and the processes ofFIGS.10and11. The computer vision processor108can extract maximum image detail in high contrast scenes, e.g., to extract attributes of people and vehicles from hyperzooms.

A hyperzoom refers to a bounding box that the camera128has zoomed into to describe the spatial location of an object, e.g., a person or a vehicle. The hyperzoom is rectangular and can be specified by the x and y coordinates of the upper-left corner and the lower-right corner of the hyperzoom. In other embodiments, the camera128specifies a hyperzoom by the (x,y) coordinates of the hyperzoom center, and the width and height of the hyperzoom. The camera128disclosed herein generates hyperzooms to detect persons and vehicular activity from images captured by the camera128. The hyperzoom that bounds each object in the image frames is used to track and detect activities of the moving objects in further frames. The hyperzooms are used to detect humans and human activities, e.g., crowd traffic. The hyperzooms captured include at least a first hyperzoom and a second hyperzoom. The first hyperzoom is associated with the persons or vehicles.

In some embodiments, the computer vision processor108assigns identifiers to the persons or vehicles in the first hyperzoom. The identifiers are used to register new objects. For example, when a new object enters or the same object is being detected, the centroid tracking will register the new object with a unique ID, so that it becomes helpful for different applications. Once the object is not in the frame, the algorithm will de-register the object ID, stating that the object is not available or left the image frame.

In step1008, the computer vision processor108tracks traffic patterns of the persons or vehicles on the edge while obviating network usage by the camera128. An example network adapter1312performing this function and an example network1314are illustrated and described in more detail with reference toFIG.13. The term “on the edge” refers to edge-based computing or edge processing. In the embodiments described herein, the execution of aggregation, data manipulation, bandwidth reduction, and other logic is performed on the edge (directly on the wireless camera128). Network usage is obviated because the image processing is not performed on a server or other external device. The traffic patterns are tracked as follows.

In step1012, the computer vision processor108predicts positions of the persons or vehicles, e.g., using a Kalman Filter from the first hyperzoom. The Kalman Filter can be executed on the computer vision processor108and uses a series of measurements observed over time (e.g., the hyperzooms) to produce estimates of unknown variables (e.g., positions of persons) by estimating a joint probability distribution over the variables for each time frame (e.g., each image frame or hyperzoom). In step1012, the computer vision processor108therefore produces estimates of the current state variables (e.g., positions of persons) along with their uncertainties.

In step1016, the computer vision processor108detects the persons or vehicles in the second hyperzoom. The computer vision processor108detects instances of semantic objects (e.g., humans or cars) in the image frames or hyperzooms. In some embodiments, the detecting is performed using neural network-based approaches. For example, end-to-end object detection can be performed without specifically defining features using a convolutional neural network (CNN), as described in more detail with reference toFIGS.11and12. In other embodiments, features are defined (e.g., using the Viola-Jones object detection framework based on Haar features, a scale-invariant feature transform (SIFT), or histogram of oriented gradients (HOG) features, etc.) and using a support vector machine (SVM) to perform the classification, as illustrated in more detail with reference toFIG.12. In some embodiments, responsive to detecting the persons or vehicles in the second hyperzoom, the computer vision processor108persists the identifiers in the second hyperzoom. The computer vision processor108associates the second hyperzoom with the persons or vehicles.

In step1020, the computer vision processor108updates the positions of the persons or vehicles based on detecting the persons or vehicles in the second hyperzoom. Once the outcome of the next measurement (from step1016) is observed, the estimates of step1012are updated using a weighted average, with more weight being given to estimates having greater certainty. The first hyperzoom is removed from the camera128so that storage in the camera128is freed up and the processing inFIG.10is performed on the edge without sending data to the server132for processing. Thus network use is obviated.

In step1024, the computer vision processor108generates tracks of the persons or vehicles based on the updated positions. In some embodiments, the computer vision processor108generates the tracks using deep sort or centroid tracking by comparing the similarity of detected objects with each other in each processed hyperzoom. If a person or vehicle has the same similarity metric throughout the frame, the computer vision processor108generates a track for the object throughout a sequence of hyperzooms and retains the same identifier for the object. The consistent identifier for a particular object enables the people and vehicle counting operation.

In some embodiments, the computer vision processor108generates the tracks using the deep sort method, which increases the number of image frames processed per second to about 10 frames per second (fps). In other embodiments, centroid tracking is used. In centroid tracking, the computer vision processor108computes a centroid from coordinates of a hyperzoom. For example, the computer vision processor108uses coordinates of the hyperzoom (xmin, ymin, xmax, and ymax) and determines (x_center, y_center) coordinates for each of the detected objects in each frame. The centroid can be determined as X_cen=((xmin+xmax)/2) and Y_cen=((ymin+ymax)/2), where xmin, ymin, xmax, and ymax are the hyperzoom coordinates. The computer vision processor108determines a Euclidian distance between the second hyperzoom and the first hyperzoom. After determining the Euclidian distance, the computer vision processor108updates the position of the centroid in the frame, thereby tracking the object. The second hyperzoom is removed from the camera128to reduce the amount of data stored and enable edge processing.

In some embodiments, the computer vision processor108expresses the tracks as image positions measured in pixel units relative to the upper-left corner of the image frame. For example, in a 320×240 pixel image, the upper-left corner is at image position (0.0, 0.0) and the lower-right corner is at (320.0, 240.0). Each physical object within the image frame is assigned world coordinates measured in scaled world units (e.g., meters) relative to a specified reference frame. In some embodiments, a coordinate system or set of transformations used to convert image positions into world coordinates is used. The coordinate system is used to define the tracks for each frame of the video as the scale (image units per world unit), origin (image position of the reference frame origin), and angle (counterclockwise angle from the image x-axis to the world x-axis).

In step1028, the computer vision processor108generates track metadata168from the tracks for storing in a database located on a non-transitory computer-readable storage medium104of the camera128. The track metadata168can be generated as structural metadata (e.g., metadata about containers of data, how compound objects are put together, types, versions, relationships, etc.), statistical metadata (e.g., the processes that collect, process, or produce statistical data, etc.), etc. The track metadata168can be stored in the form of a data structure including tables, fields, data types, indexes, or partitions in a relational engine, or as databases, dimensions, measures, and data mining models. In some embodiments the track metadata168defines a data model and the way it is accessed and presented to the users, with the reports, schedules, distribution lists, and user security rights.

The non-transitory computer-readable storage medium104is any lightweight, mobile, embedded memory or memory card in the camera that has rapid read/write rates, enabling edge processing. In some embodiments, a secure digital (SD) card or micro SD card is used. SD is a proprietary non-volatile memory card format developed by the SD Association (SDA) for use in portable devices. A microSD card is a removable miniaturized SD flash memory card, sometimes called T-Flash or TransFlash. TransFlash cards are electrically compatible with larger SD cards. In other embodiments, Compact Flash (CF) cards that can store 256 GB+data with 160 Mb/second access times are used. In other embodiments, XQD memory cards designed for high speed read and write, and suitable for high resolution (4K) video recording are used. In other embodiments, Memory Sticks (MS) that provide flexible storage, such as Memory Stick Duo, Memory Stick PRO, Memory Stick PRO-HG, or Memory Stick Micro M2, are used.

In some embodiments, the database is a key-value database located on the non-transitory computer-readable storage medium104. A key-value database stores the track metadata168as a collection of key-value pairs in which a key serves as a unique ID. The key-value database or key-value store is implemented for storing, retrieving, and managing the track metadata168as associative arrays in a data structure, more commonly as a dictionary or hash table. The track metadata168is stored and retrieved using a key that uniquely identifies the record, and is used to find the data within the database. In some embodiments, the database is a relational database (RDB). An RDB predefines the data structure in the database as a series of tables containing fields with well-defined data types. Exposing the data types to the database program allows it to apply a number of optimizations.

In some embodiments, the database stores the tracks as byte arrays on the non-transitory computer-readable storage medium104. For example, the database can be a lightning memory-mapped database (LMDB) that stores arbitrary key/data pairs as byte arrays, has a range-based search capability, supports multiple data items for a single key, and has a special mode for appending records without checking for consistency. The database can also be used concurrently in a multi-threaded or multi-processing environment, with read performance scaling linearly by design. Multiple queries148can simultaneously open and use the database, as a means to scale up performance.

In some embodiments, the database stores the tracks in immutable row partitions in sorted order. For example, the database can include an SSTable, which provides a persistent, ordered immutable map from keys to values, where both keys and values are arbitrary byte strings. Operations can be provided to look up the value associated with a specified key, and to iterate over all key-value pairs in a specified key range. A block index (stored at the end of the SSTable) can be used to locate blocks; the index is loaded into memory when the SSTable is opened.

In some embodiments, the database stores the tracks in a relational database management system (RDBMS) on the non-transitory computer-readable storage medium104. For example, the database can be a relational database that organizes data into one or more data tables in which data types may be related to each other. The relations structure the data. The queries148create, modify, and extract data from the relational database, as well as control user access to the database.

The tracks are removed from the camera128to reduce storage requirements on the camera128after metadata generation. Thus edge processing is enabled. In some embodiments, the tracks are stored by the camera128on a NoSQL key-value structure on cloud storage via the server instance112(sometimes referred to as a “second server instance”). The server instances112,116are virtual machines running workloads, e.g., in the cloud. The server instance116is sometimes referred to as a “first server instance.” For example, the server instances112,116can be launched in a specified cloud infrastructure. In some embodiments, the server instance112is a collection of SQL Server databases run by the server132. The details of each server instance112,116can be viewed on a service console which can be web-based or command-line based. The server instances112,116can be controlled or managed separately. Each SQL Server instance has its own ports, databases, and logins. The multiple server instances112,116provide advantages such as stability and backup for the camera128. The system100itself can transfer the load in case of unprecedented issues such as data loss or system crashes. For example, the server instance116can take up the work of the server instance112if affected.

In some embodiments, the NoSQL key-value structure is a proprietary NoSQL database service that supports key-value and document data structures. In some embodiments, the NoSQL key-value structure resolves version conflicts and uses synchronous replication across multiple data centers for high durability and availability. In some embodiments, the computer vision processor108generates a visualization of the tracks. The visualization provides options for displaying or viewing the object tracks. For example, the color of each track can be a constant color for all tracks, a random color from a color map can be chosen for each track, a track color can be selected based on the x velocity, y velocity, or magnitude of the velocity, etc. A number of points of each track can be changed or every point of every track can be displayed. Lines can be drawn through points and the width of the lines can be changed. The camera128stores the tracks and the visualization in the NoSQL key-value structure on cloud storage via the server instance116. The visualization is for sending to the user device124by the server instance112.

In step1032, the camera128receives a query148from the server instance116over a full-duplex communication channel. The query is initiated by the user device124. For example, the user device124can initiate the query148in SQL or another language to the server instance112for the server instance112to place it in the in-memory data structure store120. In another example, the query148can be initiated in a natural language for the server instance112to encode it into a structured language. In another example, the query148can be initiated using a graphical user interface as shown byFIGS.4and5, or by menu controls as inFIGS.6and7. The queries are encoded by the server instance112. For example, the queries can be encoded into SQL or another language. In some embodiments, the server instance112encodes the query string as a part of a uniform resource locator (URL) that assigns values to specified parameters. A query string can include fields added to a base URL by an API running on the server instance112.

The full-duplex communication channel operates according to a computer communications protocol, providing full-duplex communication over a single TCP connection. For example, the WebSocket protocol standardized by the IETF as RFC 6455 can be used. In other embodiments, the full-duplex communication channel uses SSE, which provides a way to push data from the server132to the camera128over HTTP. The query148is a request for data or information from the database, e.g., generated as results returned by SQL or as pictorials, graphs, or complex results, e.g., trend analyses from data-mining tools.

The query148is encoded as a key and a message. The key is a string or array of structured data used to filter the track metadata stored in the database on the non-transitory computer-readable storage medium104. In some embodiments, the key has multiple components, specified as an ordered list. A “major” key identifies a portion of the track metadata and consists of the leading components of the key. The subsequent components are called “minor keys.” The organization can be similar to a directory path specification in a file system (e.g., /Major/minor1/minor2/). For example, the key can specify a time range of the hyperzooms, a geographic location the hyperzooms were shot in, a set of days in a week (e.g., Mondays and Wednesdays only), coordinates of a line that people or vehicles are crossing (seeFIG.9), etc. The message specifies a characteristic of the traffic patterns of people and vehicles captured by the camera128. For example, the message can refer to a count of people, a count of vehicles, a heatmap of people or vehicles (seeFIG.7), a count of people or vehicles crossing a line in a particular direction (seeFIG.9), etc. The message can be encoded into one of several different query languages, e.g., SQL, DMX, Datalog, and AQL, etc. The results164come from filtering the database on the storage medium104using a key and the message.

In some embodiments, to respond to the query148, the computer vision processor108generates a heatmap using the track metadata. The heatmap is a data visualization technique that shows the density of people or vehicles as color in two dimensions. The variation in color can be by hue or intensity, giving obvious visual cues to the user device124about how the phenomenon is clustered or varies over space. In some embodiments, the heatmap provides information on where people have been on a floor plan during a selected time range (seeFIG.7). For example, a darker color indicates more people traffic, while a lighter color indicates less people traffic. The time range being viewed on the people heatmap can be changed, e.g., by moving a timeline back and forth, changing the date and time using a calendar function, or skipping forward and backward with skip buttons. The time interval can be changed using buttons on the right side of the timeline (seeFIG.7). The camera128sends the heatmap to the server instance116over the full-duplex communication channel. The heatmap is for sending to the user device124by the server instance112.

In step1036, the computer vision processor108generates a response152to the message based on results164of filtering the track metadata using the key and the characteristic of the traffic patterns. For example, the response152can contain detections of particular people, faces (seeFIG.6) or filtered results156based on clothing color, apparent sex, and the presence of backpacks. The response152can display a number indicating the number of people detected in an image frame. The response152can display lines that provide a sense of which parts of a clip contain the most people activity. In step1040, the camera128transmits the response152to the server instance116over the full-duplex communication channel. The response152is for sending to a user device124by the server instance112. The user device124is a smartphone, laptop, desktop, tablet, etc., implemented using at least some parts of the computer system1300ofFIG.13.

FIG.11is a flow diagram illustrating an example process for people and vehicle analytics on the edge, in accordance with one or more embodiments. In some embodiments, the process ofFIG.11is performed by the camera128illustrated and described in more detail with reference toFIG.1. In other embodiments, the process ofFIG.11is performed by a computer system, e.g., the example computer system1300illustrated and described in more detail with reference toFIG.13. Particular entities, for example, the computer vision processor108or the server132, perform some or all of the steps of the process in other embodiments. The computer vision processor108and the server132are illustrated and described in more detail with reference toFIG.1. Likewise, embodiments can include different and/or additional steps, or perform the steps in different orders.

In step1104, the computer vision processor108of the camera128extracts attributes of persons or vehicles from hyperzooms generated from image frames captured by the camera128. Hyperzooms are described in more detail with reference toFIG.10. The extraction of features and attributes (e.g., features1212) is described in more detail with reference toFIG.12. The attributes can include clothing color, apparent sex, the presence of backpacks, a make or model of vehicles, vehicle color, etc. The hyperzooms represent traffic patterns of the persons or vehicles. In some embodiments, the extracting is performed using a feature extractor of an on-camera convolutional neural network (CNN) (seeFIG.12) including an inverted residual structure. For example, in the CNN, the shortcut connections are between thin bottleneck layers. An intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. An example feature extraction module1208is illustrated and described in more detail with reference toFIG.12.

In step1108, the computer vision processor108generates mobile semantic segmentation models of the CNN using the hyperzooms and the attributes. In some embodiments, the mobile semantic segmentation models cluster parts of an image frame or hyperzoom together when they belong to the same object class. Semantic segmentation is a form of pixel-level prediction because each pixel in the hyperzoom image is classified according to an attribute or category. The models are referred to as mobile because they are executed on the computer vision processor108itself to provide edge computing capabilities. In some embodiments, generating the mobile semantic segmentation models includes training, by the computer vision processor, the CNN using the hyperzooms and the attributes. Training an example machine learning model1216is illustrated and described in more detail with reference toFIG.12. For example, the attributes can be used as the features1212, the hyperzooms can be used as the training data1220, and the image frames can be used as the validation set1232, as illustrated and described in more detail with reference toFIG.12. In some embodiments, generating the mobile semantic segmentation models comprises classification (making a prediction for a whole input), localization/detection (providing information regarding the spatial location of the classes), or fine-grained inference by making dense predictions inferring labels for every pixel, so that each pixel is labeled with the class of its enclosing object or region.

In step1112, the computer vision processor108generates an index table for a NoSQL key-value structure on cloud storage. An example NoSQL key-value structure is described in more detail with reference toFIG.10. In some embodiments, the index table uses primary keys to uniquely identify each hyperzoom or track, and secondary indexes to provide more querying flexibility. The index table is referenced by the attributes. In some embodiments, the index table accelerates application accesses and data retrieval, and supports faster performance by reducing application lag. In some embodiments, a secondary index holds an attribute subset and an alternate key. The second index can be used through either a query or scan operation.

In some embodiments, the hyperzooms are keyed or indexed by a camera_id and time (milliseconds). More than one hyperzoom can be persisted for the same person. In other embodiments, only one hyperzoom is persisted for each tracked person. An index table, e.g., a DynamoDB table, is used and called “object_tracking_hyperzoom” that is keyed by camera_id and object_id. The index table includes a path of the hyperzoom on cloud storage.

In step1116, the camera128stores the hyperzooms in the NoSQL key-value structure on the cloud storage via the server instance116. The index table includes pointers to the hyperzooms in the NoSQL key-value structure. For example, the index table can include a partition key and a sort key. To free up storage on the camera128, the hyperzooms are removed from the camera128.

In step1120, the computer vision processor108generates attribute analytics by executing the mobile semantic segmentation models based on the attributes while obviating network usage by the camera128. Thus edge processing is enabled. In some embodiments, a “Person History” feature is enabled such that the camera128performs people detection as well as more complex analytics, such as the ability to search for faces, and for people by clothing color. In some embodiments, all detected people across different cameras can be displayed (seeFIG.6). In some embodiments, person-of-interest (POI) alerts can be enabled that provide the ability to identify and search for individuals in the camera footage, allowing a user device124to quickly find meaningful events and save time during critical investigations.

In step1124, the computer vision processor108stores the attribute analytics in a key-value database located on the non-transitory computer-readable storage medium104of the camera128. Example key-value databases are described in more detail with reference toFIG.10. The attribute analytics are indexed by the attributes. The steps1104,1108,1012,1120, and1124are performed at the edge, on the camera128. In some embodiments, the camera128can go into a sleep mode, disconnect from the network, or be rebooted, or reset. When the camera128is active again after being woken up, reset, or rebooted, the camera128receives a signal that the network connection to the camera128is active again. An example network adapter1312performing this function and an example network1314are illustrated and described in more detail with reference toFIG.13. Responsive to receiving the signal, the computer vision processor108sends a descriptor of the camera128to the server instance116. For example, the descriptor can be a geographical location where the camera128is deployed, a floor of a building, or an alphanumeric descriptor denoted by camera_id.

In some embodiments, the camera128connects to a full-duplex communication channel that is generated by the server instance116. The full-duplex communication channel operates according to a computer communications protocol, providing full-duplex communication over a single TCP connection. For example, the WebSocket protocol standardized by the IETF as RFC 6455 can be used. In other embodiments, the full-duplex communication channel uses SSE, which provides a way to push data from the server132to the camera128over HTTP. The full-duplex communication channel generated references the descriptor and is for transmission of queries generated by the server instance112. The queries are initiated by the user device124. For example, the user device can initiate the query148in SQL or another language to the server instance112for the server instance112to place it in the in-memory data structure store120. In another example, the query148can be initiated in a natural language for the server instance112to encode it into a structured language. In another example, the query148can be initiated using a graphical user interface as shown byFIGS.4and5, or by menu controls as inFIGS.6and7. The queries are encoded by the server instance112. For example, the queries can be encoded into SQL or another language. In some embodiments, the server instance112encodes the query string as a part of a uniform resource locator (URL) that assigns values to specified parameters. A query string can include fields added to a base URL by an API running on the server instance112.

In step1128, the computer vision processor108receives a query148from the server instance116over a full-duplex communication channel. The query148specifies one or more of the attributes. For example, the query can specify clothing colors, apparent sex, the presence of backpacks, a particular person or vehicle (e.g., person616inFIG.6), etc. Example query structures are described in more detail with reference toFIG.10. For example, a query can specify “show when Matt Peterson entered the building & all instances of a person with a red shirt in the building.”

In step1132, the computer vision processor108filters the attribute analytics using the one or more of the attributes to generate a response152. The response152includes a portion of the traffic patterns corresponding to a particular person or vehicle (e.g., person616inFIG.6). The filtering is in response to the query and removes redundant or unwanted information from the attribute analytics prior to generating the response152. The filtering thus increases the semantic signal-to-noise ratio. The results164of the filtering make up the response152ofFIG.1(e.g., portion of the traffic patterns corresponding to a particular person or vehicle).

In step1136, the camera128sends a response152to the server instance116. The response152includes the portion of the traffic patterns corresponding to the particular person or vehicle over the full-duplex communication channel for sending to the user device124by the server instance112. The query-response process is asynchronous. When the camera128responds to the server instance116with a message containing the results156, the response process is asynchronous and can be out of order, i.e., the camera128can receive two or more queries and return two or more responses. For example, the query can be a first query, and the portion of the traffic patterns (response152) is transmitted asynchronously with respect to a second query received by the camera128from the server instance116over the full-duplex communication channel. The server instance116receives the response152and adds the results156to the in-memory data structure store120using a received key. Similarly, the process of the server instance116receiving the response152and adding the results156to the in-memory data structure store120is also asynchronous and does not interfere with or block the step of the server instance116sending the queries148to the camera128. The server instance112retrieves the results156from the in-memory data structure store120and sends them to the user device124for viewing by a user.

FIG.12is a block diagram illustrating an example machine learning (ML) system1200for people and vehicle analytics on the edge, in accordance with one or more embodiments. The ML system1200is implemented using components of the example computer system1300illustrated and described in more detail with reference toFIG.13. For example, the ML system1200can be implemented on the camera128using instructions programmed in a memory of the camera128illustrated and described in more detail with reference toFIG.1. Likewise, embodiments of the ML system1200can include different and/or additional components or be connected in different ways. The ML system1200is sometimes referred to as an ML module.

The ML system1200includes a feature extraction module1208implemented using components of the example computer system1300illustrated and described in more detail with reference toFIG.13. In some embodiments, the feature extraction module1208extracts a feature vector1212from input data1204. For example, the input data1204can include images, features, frames, or metadata from a camera. The feature vector1212includes features1212a,1212b, . . . ,1212n. The feature extraction module1208reduces the redundancy in the input data1204, e.g., repetitive data values, to transform the input data1204into the reduced set of features1212, e.g., features1212a,1212b, . . . ,1212n. The feature vector1212contains the relevant information from the input data1204, such that events or data value thresholds of interest can be identified by the ML model1216by using this reduced representation. In some example embodiments, the following dimensionality reduction techniques are used by the feature extraction module1208: independent component analysis, Isomap, kernel principal component analysis (PCA), latent semantic analysis, partial least squares, PCA, multifactor dimensionality reduction, nonlinear dimensionality reduction, multilinear PCA, multilinear subspace learning, semidefinite embedding, autoencoder, and deep feature synthesis.

In alternate embodiments, the ML model1216performs deep learning (also known as deep structured learning or hierarchical learning) directly on the input data1204to learn data representations, as opposed to using task-specific algorithms. In deep learning, no explicit feature extraction is performed; the features1212are implicitly extracted by the ML system1200. For example, the ML model1216can use a cascade of multiple layers of nonlinear processing units for implicit feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The ML model1216can thus learn in supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) modes. The ML model1216can learn multiple levels of representations that correspond to different levels of abstraction, wherein the different levels form a hierarchy of concepts. In this manner, the ML model1216can be configured to differentiate features of interest from background features.

In alternative example embodiments, the ML model1216, e.g., in the form of a CNN, generates the output1224, without the need for feature extraction, directly from the input data1204. The output1224is provided to the user device1228, the computer vision processor108, the server132, the API running on server instance116, or the user device124illustrated and described in more detail with reference toFIG.1. In some embodiments, the steps performed by the ML system1200are stored in memory on the camera128for execution. In other embodiments, the output1224is displayed on the video display1318illustrated and described in more detail with reference toFIG.13.

A CNN is a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of a visual cortex. Individual cortical neurons respond to stimuli in a restricted region of space known as the receptive field. The receptive fields of different neurons partially overlap such that they tile the visual field. The response of an individual neuron to stimuli within its receptive field can be approximated mathematically by a convolution operation. CNNs are based on biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing.

The ML model1216can be a CNN that includes both convolutional layers and max pooling layers. The architecture of the ML model1216can be “fully convolutional,” which means that variable sized sensor data vectors can be fed into it. For all convolutional layers, the ML model1216can specify a kernel size, a stride of the convolution, and an amount of zero padding applied to the input of that layer. For the pooling layers, the model1216can specify the kernel size and stride of the pooling.

In some embodiments, the ML system1200trains the ML model1216, based on the training data1220, to correlate the feature vector1212to expected outputs in the training data1220. As part of the training of the ML model1216, the ML system1200forms a training set of features and training labels by identifying a positive training set of features that have been determined to have a desired property in question, and, in some embodiments, forms a negative training set of features that lack the property in question.

The ML system1200applies ML techniques to train the ML model1216, that when applied to the feature vector1212, outputs indications of whether the feature vector1212has an associated desired property or properties, such as a probability that the feature vector1212has a particular Boolean property, or an estimated value of a scalar property. The ML system1200can further apply dimensionality reduction (e.g., via linear discriminant analysis (LDA), PCA, or the like) to reduce the amount of data in the feature vector1212to a smaller, more representative set of data.

The ML system1200can use supervised ML to train the ML model1216, with feature vectors of the positive training set and the negative training set serving as the inputs. In some embodiments, different ML techniques, such as linear support vector machine (linear SVM), boosting for other algorithms (e.g., AdaBoost), logistic regression, naïve Bayes, memory-based learning, random forests, bagged trees, decision trees, boosted trees, boosted stumps, neural networks, CNNs, etc., are used. In some example embodiments, a validation set1232is formed of additional features, other than those in the training data1220, which have already been determined to have or to lack the property in question. The ML system1200applies the trained ML model1216to the features of the validation set1232to quantify the accuracy of the ML model1216. Common metrics applied in accuracy measurement include: Precision and Recall, where Precision refers to a number of results the ML model1216correctly predicted out of the total it predicted, and Recall is a number of results the ML model1216correctly predicted out of the total number of features that had the desired property in question. In some embodiments, the ML system1200iteratively re-trains the ML model1216until the occurrence of a stopping condition, such as the accuracy measurement indication that the ML model1216is sufficiently accurate, or a number of training rounds having taken place.

FIG.13is a block diagram illustrating an example computer system1300for people and vehicle analytics on the edge, in accordance with one or more embodiments. Components of the example computer system1300can be used to implement the camera128, the server132, or the user device124illustrated and described in more detail with reference toFIG.1. In some embodiments, components of the example computer system1300are used to implement the ML system1200illustrated and described in more detail with reference toFIG.12. At least some operations described herein can be implemented on the computer system1300.

The computer system1300can include one or more central processing units (“processors”)1302, main memory1306, non-volatile memory1310, network adapters1312(e.g., network interface), video displays1318, input/output devices1320, control devices1322(e.g., keyboard and pointing devices), drive units1324including a storage medium1326, and a signal generation device1330that are communicatively connected to a bus1316. The bus1316is illustrated as an abstraction that represents one or more physical buses and/or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus1316, therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as “Firewire”).

The computer system1300can share a similar computer processor architecture as that of a desktop computer, tablet computer, personal digital assistant (PDA), mobile phone, game console, music player, wearable electronic device (e.g., a watch or fitness tracker), network-connected (“smart”) device (e.g., a television or home assistant device), virtual/augmented reality systems (e.g., a head-mounted display), or another electronic device capable of executing a set of instructions (sequential or otherwise) that specify action(s) to be taken by the computer system1300.

While the main memory1306, non-volatile memory1310, and storage medium1326(also called a “machine-readable medium”) are shown to be a single medium, the term “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized/distributed database and/or associated caches and servers) that store one or more sets of instructions1328. The term “machine-readable medium” and “storage medium” should also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computer system1300.

In general, the routines executed to implement the embodiments of the disclosure can be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically include one or more instructions (e.g., instructions1304,1308,1328) set at various times in various memory and storage devices in a computer device. When read and executed by the one or more processors1302, the instruction(s) cause the computer system1300to perform operations to execute elements involving the various aspects of the disclosure.

Moreover, while embodiments have been described in the context of fully functioning computer devices, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms. The disclosure applies regardless of the particular type of machine or computer-readable media used to actually effect the distribution.

Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable-type media such as volatile and non-volatile memory devices1310, floppy and other removable disks, hard disk drives, optical discs (e.g., Compact Disc Read-Only Memory (CD-ROMS), Digital Versatile Discs (DVDs)), and transmission-type media such as digital and analog communication links.

The network adapter1312enables the computer system1300to mediate data in a network1314with an entity that is external to the computer system1300through any communication protocol supported by the computer system1300and the external entity. The network adapter1312can include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and/or a repeater.

The network adapter1312can include a firewall that governs and/or manages permission to access proxy data in a computer network and tracks varying levels of trust between different machines and/or applications. The firewall can be any number of modules having any combination of hardware and/or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and/or applications and applications (e.g., to regulate the flow of traffic and resource sharing between these entities). The firewall can additionally manage and/or have access to an access control list that details permissions including the access and operation rights of an object by an individual, a machine, and/or an application, and the circumstances under which the permission rights stand.

In additional embodiments, a computer vision processor of a camera generates tracks of a person or vehicle from hyperzooms depicting the person or vehicle, the hyperzooms generated from image frames captured by the camera. The computer vision processor receives coordinates of a line from a server instance over a full-duplex communication channel. The coordinates of the line are defined by a user device communicably coupled to the server instance. The computer vision processor defines a first parallel line and a second parallel line spaced from the line and positioned on different sides of the line. The computer vision processor determines a first position of the person or vehicle relative to the line from the tracks. The camera captures an image frame of the person or vehicle, the image frame captured after the image frames. The computer vision processor determines a second position of the person or vehicle relative to the line from the image frame captured after the image frames. The computer vision processor determines that the person or vehicle has crossed the line from a first side of the first parallel line to a second side of the second parallel line based on a difference between the first position and the second position. Responsive to determining that a first distance between the second position and the second parallel line is smaller than a second distance between the second position and the line, the computer vision processor stores metadata describing that the person or vehicle has crossed the line in a key-value database located on a memory card of the camera. The camera receives a query from the server instance over the full-duplex communication channel. The query requests a count of a number of times the person or vehicle has crossed the line. The computer vision processor generates a response to the query from the metadata stored in the key-value database. The camera sends the response to the server instance over the full-duplex communication channel. The response is for sending to the user device by the server instance.

The functions performed in the processes and methods can be implemented in differing order. Furthermore, the outlined steps and operations are only provided as examples, and some of the steps and operations can be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.

The techniques introduced here can be implemented by programmable circuitry (e.g., one or more microprocessors), software and/or firmware, special-purpose hardwired (i.e., non-programmable) circuitry, or a combination of such forms. Special-purpose circuitry can be in the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.

The description and drawings herein are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of the disclosure. However, in certain instances, well-known details are not described in order to avoid obscuring the description. Further, various modifications can be made without deviating from the scope of the embodiments.

The terms used in this specification generally have their ordinary meanings in the art, within the context of the disclosure, and in the specific context where each term is used. Certain terms that are used to describe the disclosure are discussed above, or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the disclosure. For convenience, certain terms can be highlighted, for example using italics and/or quotation marks. The use of highlighting has no influence on the scope and meaning of a term; the scope and meaning of a term is the same, in the same context, whether or not it is highlighted. It will be appreciated that the same thing can be said in more than one way. One will recognize that “memory” is one form of a “storage” and that the terms can on occasion be used interchangeably.

Consequently, alternative language and synonyms can be used for any one or more of the terms discussed herein, nor is any special significance to be placed upon whether or not a term is elaborated or discussed herein. Synonyms for certain terms are provided. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any term discussed herein is illustrative only and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various embodiments given in this specification.

It is to be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of this invention and that various modifications can be implemented by those skilled in the art.