CLASSIFICATION AND RE-IDENTIFICATION

Aspects of the present disclosure include methods, systems, and non-transitory computer readable media that perform the steps of receiving one or more snapshots, extracting one or more features from the one or more snapshots, and providing the one or more features to a first classification layer for classifying a first target and a second classification layer for re-identifying a second target.

BACKGROUND

In surveillance systems, numerous images (e.g., more than thousands or even millions) may be captured by multiple cameras. Each image may show people and objects (e.g., cars, infrastructures, accessories, etc.). In certain circumstances, security personnel monitoring the surveillance systems may want to locate and/or track a particular person and/or object through the multiple cameras. However, the process may be computationally intensive for the surveillance systems to accurately track the particular person and/or object by searching through the images. Further, during the training of the neural network used for re-identification, the computer resources may be allocated for both classification and re-identification. Therefore, improvements may be desirable.

SUMMARY

An aspect of the present disclosure includes a method including receiving one or more snapshots, extracting one or more features from the one or more snapshots, and providing the one or more features to a first classification layer for classifying a first target and a second classification layer for re-identifying a second target.

Aspects of the present disclosure includes a neural network including feature layers configured to: receive one or more snapshots, extracting one or more features from the one or more snapshots, and providing the one or more features to a first classification layer and a second classification layer, the first classification layer configured to re-identify a first target, and the second classification layer configured to classify a second target.

Certain aspects of the present disclosure includes a non-transitory computer readable medium having instructions stored therein that, when executed by a processor, cause the processor to cause feature layers to: receive one or more snapshots, extracting one or more features from the one or more snapshots, and providing the one or more features to a first classification layer and a second classification layer, cause the first classification layer configured to re-identify a first target, and cause the second classification layer configured to classify a second target.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting.

The term “processor,” as used herein, can refer to a device that processes signals and performs general computing and arithmetic functions. Signals processed by the processor can include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other computing that can be received, transmitted and/or detected. A processor, for example, can include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described herein.

The term “bus,” as used herein, can refer to an interconnected architecture that is operably connected to transfer data between computer components within a singular or multiple systems. The bus can be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others.

The term “memory,” as used herein, can include volatile memory and/or nonvolatile memory. Non-volatile memory can include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM) and EEPROM (electrically erasable PROM). Volatile memory can include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM).

In some aspects of the present disclosures, neural networks for the classification and feature extraction may have similar architectures. A time-consuming portion of the training process is the feature layers in the neural network. In some instances, the calculations from the feature layers may be used for both the classification and the feature extraction processes to conserve computational resources. The feature layers may extract visual patterns that are used in both the classification and the feature extraction processes for re-identification.

In some instances, providing the identified features to the layer for classification and the layer(s) for re-identification in parallel, simultaneously, and/or contemporaneously may obviate the need to repeat the feature extraction processes. In an aspect of the present disclosure, the neural network performs the feature extraction processes, and then provides the extracted features to the re-identification layer(s) and the classification layer in parallel (e.g., providing the extracted features to the re-identification layer(s) and the classification layer), simultaneously (e.g., providing the extracted features to the re-identification layer(s) and the classification layer at the same time), and/or contemporaneously (e.g., providing the extracted features to the re-identification layer(s) during a first time and the classification layer during a second time that overlaps at least partially with the first time).

Referring toFIG. 1, an example of an environment100for performing classification and re-identification during training may include a server140that receives surveillance videos and/or images112from a plurality of cameras110. The plurality of cameras110may capture the surveillance videos and/or images112of one or more locations114that include targets such as people and/or objects (e.g., cars, bags, etc.).

In certain instances, the server140may include a processor120and/or a memory122. The processor120and/or the server140may include a communication component142that receives and/or sends data (such as the captured surveillance videos and/or images112) from and to other devices, such as a data repository150. The processor120and/or the server140may include an identification component144that performs the re-identification process. The processor120and/or the server140may include a classification component146that classifies one or more images or objects in the images. The processor120and/or the server140may include an artificial intelligence (AI) component148that performs AI operations during the re-identification and/or classification processes. The communication component142, the identification component144, the classification component146, and/or the AI component148may be implemented via software, hardware, or a combination thereof. For example, the communication component142, the identification component144, the classification component146, and/or the AI component148may be programs stored in the memory122being executed by the processor120. In another example, the communication component142, the identification component144, the classification component146, and/or the AI component148may be implemented in one or more microprocessors, microcontrollers, programmable logic devices, field programmable gate arrays, or other hardware devices.

In some implementations, the captured surveillance videos and/or images may include snapshots (i.e., frames or portions of frames). For example a one minute surveillance video and/or images may include 30, 60, 120, 180, 240, or other numbers of snapshots. During the classification and re-identification process, the communication component142may receive the surveillance video and/or images112from the plurality of cameras110. The identification component144may perform the re-identification process of the targets in the surveillance video and/or images112. The classification component146may classify the targets in the surveillance video and/or images112. The AI component148may perform the feature extraction process.

Turning toFIG. 2, an example of a neural network200for classification and re-identification may include feature layers202that receive the surveillance videos and/or images112as input. The feature layers202may be a deep learning algorithm that includes feature layers202-1,202-2. . . ,202-n-1,202-n. Each of the feature layers202-1,202-2. . . ,202-n-1,202-nmay perform a different function and/or algorithm (e.g., pattern detection, transformation, feature extraction, etc.). In a non-limiting example, the feature layer202-1may identify edges of the surveillance videos and/or images112, the feature layer202-bmay identify corners of the surveillance videos and/or images112. . . the feature layer202-n-1may perform a non-linear transformation, and the feature layer202-nmay perform a convolution. In another example, the feature layer202-1may apply an image filter to the surveillance videos and/or images112, the feature layer202-2may perform a Fourier Transform to the surveillance videos and/or images112. . . the feature layer202-n-1may perform an integration, and the feature layer202-nmay identify a vertical edge and/or a horizontal edge. Other implementations of the feature layers202may also be used to extract features of the surveillance videos and/or images112.

In certain implementations, the output of the feature layers202may be provided as input to classification layers204a,204b,204c.The classification layer204amay be configured to identify a person and/or provide a person identification (ID) label associated with the identified person. The classification layer204bmay be configured to identify an object (e.g., a car, a person . . . ) and/or provide an ID label associated with the identified object. The classification layer204cmay be configured to identify a class (e.g., person or car) and/or provide a class label associated with the identified class.

AlthoughFIG. 2illustrates an example having three classification layers204, aspects of the present disclosure may include neural networks having different number of classification layers and different types of classification layers. For example, another example of a neural network may include 4 classification layers (e.g., person, vehicle, personal accessory, and class). In another example, a neural network may include a vehicle classification layer only. Some of the classification layers204may perform classification and/or re-identification.

In some implementations, the classification layer204amay output a person ID label. The classification layer204bmay output a car ID label. The classification layer204cmay output a class label. A classification error component206amay receive the person ID label and a ground truth person ID as input. A classification error component206bmay receive the car ID label and a ground truth car ID as input. A classification error component206cmay receive the class label and a ground truth class as input. The ground truth person ID, ground truth car ID, and ground truth class may be the “correct answer” provided by a trainer (not shown) to the neural network200during training. For example, the neural network200may compare the car ID label to the ground truth car ID to determine whether the classification layer204bproperly identifies car associated with the car ID label. Other types of ID labels are possible.

In some instances, the neural network200may include a combined error component208. Based on the person ID label and the ground truth person ID, the classification error component206amay output a person error into the combined error component208. Based on the car ID label and the ground truth car ID, the classification error component206bmay output a car error into the combined error component208. Based on the class label and the ground truth class, the classification error component206cmay output a class error into the combined error component208. The combined error component208may receive one or more of the person error, the car error, and/or the class error, and provide one or more updated parameters220to the feature layers202and/or the classification layer204. The one or more updated parameters220may include modifications to parameters and/or equations to reduce the one or more of the person error, the car error, and/or the class error.

In some examples, the neural network200may include a flatten function230that generates a final output of the feature extraction step. For example, the flatten function230may be an operator that transforms a matrix of features into a vector.

During operation, the feature layers202of the neural network200may receive the surveillance videos and/or images112. The feature layers202-1,202-2. . . ,202-n-1,202-nmay identify features in the surveillance videos and/or images112. The feature layers202may send the identified features to the classification layers204. In certain instances, the feature layers202may be implemented by the processor120, the memory122, the communication component142, the identification component144, the classification component146, and/or the AI component148. The classification layers204may receive the identified features. In some implementations, the classification layers204a,204b,204cmay receive the same identified features. In other implementations, the classification layers204a,204b,204cmay receive different identified features (e.g., tailored to person, car, and/or class). In some implementations, the identified features may be numerical representations (e.g., numbers, vectors, matrix, etc.) that enable the classification layers204a,204b,204cto identify a person, a car, and/or a class. In certain instances, the classification layers204may be implemented by the processor120, the memory122, the identification component144, and/or the classification component146.

In some variations, the classification layer204amay receive the identified features from the feature layers202. Based on the received identified features, the classification layer204amay provide a person ID label of a person in the surveillance videos and/or images112. The person ID label may be an identifier (e.g., alpha-numeric) associated with a person in the surveillance videos and/or images112. Based on the received identified features, the classification layer204bmay provide a car ID label of a car in the surveillance videos and/or images112. The car ID label may be an identifier (e.g., alpha-numeric) associated with a vehicle (e.g., car) in the surveillance videos and/or images112. Based on the received identified features, the classification layer204cmay provide a class label of a class (e.g., person class or car class) in the surveillance videos and/or images112. The class label may be an identifier (e.g., alpha-numeric) associated with a class in the surveillance videos and/or images112.

In certain implementations, the classification error component206amay receive the person ID label and the ground truth person ID as input. The classification error component206amay compare the person ID label and the ground truth person ID and generate an person error. The person error may be inversely proportional to a probability that the person ID label matches the ground truth person ID. For example, if there is a high probability (e.g., greater than 95%) that the person ID label matches the ground truth person ID, the person error may be small.

In some implementations, the classification error component206bmay receive the car ID label and the ground truth car ID as input. The classification error component206bmay compare the car ID label and the ground truth car ID and generate a car error. The car error may be inversely proportional to a probability that the car ID label matches the ground truth car ID. For example, if there is a high probability (e.g., greater than 95%) that the car ID label matches the ground truth car ID, the car error may be small.

In non-limiting implementations, the classification error component206cmay receive the class label and the ground truth class as input. The classification error component206cmay compare the class label and the ground truth class and generate a class error. The class error may be inversely proportional to a probability that the class label matches the ground truth class. For example, if there is a high probability (e.g., greater than 95%) that the class label matches the ground truth class, the class error may be small. In certain instances, the classification error component206may be implemented by the processor120, the memory122, the identification component144, the classification component146, and/or the AI component148.

In some instances, a combined error component208may compute a combined error based on one or more of the person error, the car error, and/or class error. For example, the combined error component208may sum the person error, the car error, and class error to determine the combined error. In response to computing the combined error, the combined error component208may transmit the one or more updated parameters220to at least one of the feature layers202, the classification layer204a,the classification layer204b,and/or the classification layer204c.The one or more updated parameters220may adjust the parameters and/or algorithms used by the feature layers202, the classification layer204a,the classification layer204b,and/or the classification layer204c.In certain instances, the combined error component208may be implemented by the processor120, the memory122, the identification component144, the classification component146, and/or the AI component148.

In some examples, the training of the neural network200includes reducing the combined error generated by the combined error component208. Reduction of the combined error may indicate improvements in the ability of the neural network to correctly identify people, objects, and/or classes during the training process. In one aspect, the neural network200may attempt to minimize the combined error.

In some instances, the flatten function230may provide an output of the neural network. For example, the flatten function230may be an operator that transforms a matrix of features into a vector. In certain instances, the flatten function230may be implemented by the processor120, the memory122, the identification component144, the classification component146, and/or the AI component148.

Turning now toFIG. 3, a method300of classification and re-identification may be performed by the server140, the communication component142, the identification component144, the classification component146, and/or the AI component148.

At block305, the method300may receive one or more snapshots. For example, the processor120, the memory122, and/or the communication component142may receive the surveillance videos and/or images112as described above with respect toFIG. 2. The processor120, the memory122, and/or the communication component142may be configured to and/or define means for receiving one or more snapshots.

At block310, the method300may extract one or more features from the one or more snapshots. For example, the processor120, the memory122, and/or the AI component148may extract the features (e.g., a contour associated with a specific car, a height-to-weight ratio of a specific person, etc.) of the surveillance videos and/or images112as described above with respect toFIG. 2. The processor120, the memory122, and/or the AI component148may be configured to and/or define means for extracting one or more features from the one or more snapshots.

At block315, the method300may provide, contemporaneously, simultaneously, or in parallel, the one or more features to a first classification layer for classifying a first target and a second classification layer for re-identifying a second target. For example, the processor120, the memory122, the identification component144, the classification component146, and/or the AI component148may provide the features of the surveillance videos and/or images112to the classification layers204a,204b,204c.In some implementations, the AI component148may provide the features of the surveillance videos and/or images112as described above with respect toFIG. 2. The processor120, the memory122, the identification component144, the classification component146, and/or the AI component148may be configured to and/or define means for providing, contemporaneously, simultaneously, or in parallel, the one or more features to a first classification layer for classifying a first target and a second classification layer for re-identifying a second target.

Aspects of the present disclosures may be implemented using hardware, software, or a combination thereof and may be implemented in one or more computer systems or other processing systems. In an aspect of the present disclosures, features are directed toward one or more computer systems capable of carrying out the functionality described herein. An example of such the computer system400is shown inFIG. 4. In some examples, the server140may be implemented as the computer system400shown inFIG. 4. The server140may include some or all of the components of the computer system400.

The computer system400includes one or more processors, such as processor404. The processor404is connected with a communication infrastructure406(e.g., a communications bus, cross-over bar, or network). Various software aspects are described in terms of this example computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement aspects of the disclosures using other computer systems and/or architectures.

The computer system400may include a display interface402that forwards graphics, text, and other data from the communication infrastructure406(or from a frame buffer not shown) for display on a display unit440. Computer system400also includes a main memory408, preferably random access memory (RAM), and may also include a secondary memory410. The secondary memory410may include, for example, a hard disk drive412, and/or a removable storage drive414, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, a universal serial bus (USB) flash drive, etc. The removable storage drive414reads from and/or writes to a removable storage unit418in a well-known manner. Removable storage unit418represents a floppy disk, magnetic tape, optical disk, USB flash drive etc., which is read by and written to removable storage drive414. As will be appreciated, the removable storage unit418includes a computer usable storage medium having stored therein computer software and/or data. In some examples, one or more of the main memory408, the secondary memory410, the removable storage unit418, and/or the removable storage unit422may be a non-transitory memory.

Alternative aspects of the present disclosures may include secondary memory410and may include other similar devices for allowing computer programs or other instructions to be loaded into computer system400. Such devices may include, for example, a removable storage unit422and an interface420. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units422and interfaces420, which allow software and data to be transferred from the removable storage unit422to computer system400.

Computer system400may also include a communications circuit424. The communications circuit424may allow software and data to be transferred between computer system400and external devices. Examples of the communications circuit424may include a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. Software and data transferred via the communications circuit424are in the form of signals428, which may be electronic, electromagnetic, optical or other signals capable of being received by the communications circuit424. These signals428are provided to the communications circuit424via a communications path (e.g., channel)426. This path426carries signals428and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, an RF link and/or other communications channels. In this document, the terms “computer program medium” and “computer usable medium” are used to refer generally to media such as the removable storage unit418, a hard disk installed in hard disk drive412, and signals428. These computer program products provide software to the computer system400. Aspects of the present disclosures are directed to such computer program products.

Computer programs (also referred to as computer control logic) are stored in main memory408and/or secondary memory410. Computer programs may also be received via communications circuit424. Such computer programs, when executed, enable the computer system400to perform the features in accordance with aspects of the present disclosures, as discussed herein. In particular, the computer programs, when executed, enable the processor404to perform the features in accordance with aspects of the present disclosures. Accordingly, such computer programs represent controllers of the computer system400.

In an aspect of the present disclosures where the method is implemented using software, the software may be stored in a computer program product and loaded into computer system400using removable storage drive414, hard drive412, or communications interface420. The control logic (software), when executed by the processor404, causes the processor404to perform the functions described herein. In another aspect of the present disclosures, the system is implemented primarily in hardware using, for example, hardware components, such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).