Patent ID: 12260674

DETAILED DESCRIPTION

In the drawings, like reference numerals designate identical or corresponding parts throughout the several views. Further, as used herein, the words “a,” “an” and the like generally carry a meaning of “one or more,” unless stated otherwise. The drawings are generally drawn to scale unless specified otherwise or illustrating schematic structures or flowcharts.

Furthermore, the terms “approximately,” “approximate,” “about,” and similar terms generally refer to ranges that include the identified value within a margin of 20%, 10%, or preferably 5%, and any values therebetween.

The present disclosure relates to a person search system and method that involves identifying a person in a case where multiple cameras at different locations are used to capture gallery images. The present disclosure provides a solution that enables accurate detection and re-identification of person instances under challenging scenarios such as pose variation and occlusions and distracting backgrounds.

FIG.1illustrates a layout in which multiple cameras are located at different locations. In some embodiments, multiple cameras may be located in different locations inside a room, along a hallway, or inside multiple rooms in a building, or inside and outside multiple buildings, vehicles, enclosed spaces or open spaces within a local area. Referring toFIG.1, a general room102includes at least one entrance110, and at least one surveillance camera108mounted inside, or facing inside, the room102. There can also be one or more surveillance cameras106mounted outside, or facing outside, the room102. A room102or building floor may contain one or more interior rooms104. The cameras106,108may face different directions D in order to cover a wider field of view F than can be obtained by a single camera.

The video cameras106,108may be stationary, or may be motorized to move in a regular pattern. The video cameras106,108may be started or stopped by motion activation. Some video cameras may be capable of night vision, such as infrared cameras, or may be equipped with different types of cameras and/or lenses for different conditions, such as day and night, and wide angle or near view.

Provided multiple cameras at different locations, a person of interest may appear in multiple video images, possibly in a location where there are other people, in different locations with different lighting conditions, in videos taken at different view angles, as well as in a dynamic environment where people, as well as the person of interest, are moving and partially occluded. There can be video images where the person of interest is facing away from a camera and/or is at different distances from the camera. There may also be video images in which a person of interest is wearing different clothing or has different styling characteristics relating to hair or cosmetics, or has accessories such as bags, walking devices and the like, due to videos being captured at different periods of time. In addition, different cameras may have a variety of resolutions and frame rates.

Despite these conditions, there is a need to accurately identify a person of interest in video images taken using multiple different cameras.

FIG.2illustrates a layout in which multiple cameras are at different locations on a vehicle. A vehicle200is equipped with one or more outward facing video cameras204. In some embodiments, a vehicle200includes one or more cabin video cameras210. The cabin video cameras210are configured to monitor the driver and/or passenger's behavior. For example, for purposes of safety, the cabin video camera210may include a video controller that can monitor a driver's eye movement and/or head movement in order to determine the state of the driver, for example as being alert or tired.

In the case of a cabin video camera210, video images may be obtained for different drivers at different time periods. There is a need to identify the person in the vehicle at different time periods. Also, the exterior cameras204may serve to identify a person in the vicinity of the vehicle as a person that may be entering the car.

In the case of a cabin video camera210, the video camera210can detect a child being placed in the vehicle, in some cases being placed in a child safety seat. In some embodiments, the video camera210can identify the child in the vehicle and can be configured to transmit status information to a mobile device of a parent of the child.

Features of the disclosed invention are explained in terms of two desirable characteristics to be considered that are robust to appearance deformations (e.g., pose variations, occlusions) and background distractions occurring in the query person image. Ideal photos of a person of interest are not always available for use as a query person image. The best person image may be one in which the person is facing or moving away from a camera, or may be facing or moving sideways relative to a camera, not just facing directly into a camera. The camera may be positioned above or below a person such that the view direction is at an angle above or below a person's face. An image of a person may be a partial view of a person due to blockage by other objects, various lighting conditions, and/or shading conditions. A person image will typically include background. In addition to background scenery, background can include moving objects entering and passing through a scene, other people, a person or persons holding hands or performing gestures in the vicinity of the person of interest. A person image may include another person embracing the person of interest, such as giving a hug. Such background is preferably distinguished from the person of interest, but are distractions that make it difficult to determine clear boundaries.

FIGS.3A and3Billustrate examples of appearance deformations and background distractions.FIG.3Aillustrates the qualitative comparison showing different non-limiting query examples302and their corresponding top matching results obtained with and without the ARM module in the same base framework. Here, false304and true306matching results are shown. These examples depict appearance deformations and distracting backgrounds in the gallery images for the query person302. The query person image302in the second row is an image of a person moving away from the camera, i.e., a view of the back of the person. The query person image302in the third row is an image the side view of a person. In each case, a conventional person search resulted in false identification304for the query person image. The ARM module, in306, explicitly captures discriminative relation features to better handle the appearance deformations in these examples.

FIG.3Bis a graph illustrating an accuracy (AP) vs. speed (frames per second) comparison with state-of-the-art person search methods based on the PRW test set (where PRW is an acronym for Person Re-identification in the Wild). All methods are reported based on a Resnet50 backbone. Speed is computed using a V100 GPU as the implementation platform. Referring toFIG.3B, the PS-ARM achieves an absolute mAP gain of 5% over SeqNet while operating at a comparable speed.

Discriminative Relation Features through Local Information Mixing: As stated above, the position of different local person regions within an RoI can vary in case of appearance deformations such as pose variations and occlusions. These deformations can deteriorate the quality of re-id features, leading to inaccurate person matching. Therefore, a dedicated mechanism is desired that generates discriminative relation features by globally mixing relevant information from different local regions within an RoI. To ensure a straightforward integration into existing person search pipelines, such a mechanism is further expected to learn discriminative relation features without requiring fine-level region supervision or topological body approximations.

FG-BG Delineation for Accurate Local Information Mixing: The quality of the aforementioned relation features rely on the assumption that the RoI region only contains foreground (person) information. However, in real-world scenarios the RoI regions contain unwanted background information due to bounding-box locations. Therefore, foreground-background (fg-bg) delineation is essential for accurate local information mixing to obtain discriminative relation features. Further, such a fg-bg delineation can also improve the detection performance.

Overall Architecture

FIG.4is a block diagram of an overall architecture of the disclosed framework. It comprises a person detection branch402and a person re-id branch404. The person detection branch402follows the structure of standard Faster R-CNN, which comprises a ResNet backbone (res1-res4), a region proposal network (RPN414), RoIAlign pooling416, and a prediction head for box regression and classification. The person re-id branch404takes the boxes predicted by the person detection branch402as input and performs RoIAlign pooling432on these predicted box locations. The resulting RoI Align pooled features are utilized to refine the box locations, along with generating a norm-aware embedding436for box classification (person vs. background) and re-id feature prediction. In both person detection402and person re-id404branches, the RoI Align416,432features are passed through a non-shared convolution block (res5424,434) and pooling layers before being used for the predictions. During inference, the person re-id branch404takes only unique boxes (obtained by non-maximum suppression algorithm) from the person detection branch402and performs a context bipartite graph matching for the re-id similar to Li et al. See Li et al. (2021). The above-mentioned standard detection402and re-id404branches serve as a base network to which an attention-aware relation mixer (ARM422) module are introduced that enriches the RoI features for accurate person search.

The focus of the design is the introduction of an ARM module. Specifically, the ARM module422is integrated between the RoIAlign416,432and convolution blocks (res5424,434) in both the person detection402and re-id404branches of the base framework. The ARM module422strives to enrich standard RoI Align416pooled features by capturing discriminative relation features between different local regions within an RoI through global mixing of local information. To ensure effective enrichment of RoI Align pooled features, a FGBG delineation mechanism is introduced into the ARM module422. The ARM module422strives to simultaneously improve both detection and re-id sub-tasks, and therefore utilizes non-shared parameters between the detection402and re-id404branches. Furthermore, the ARM module422is generic and can be easily integrated to other Faster R-CNN based person search methods. Next, are presented the details of the ARM module.

Attention-Aware Relation Mixer (ARM) Module

FIG.5is a block diagram of the ARM module. The ARM module422takes RoI Align pooled features as input and captures inter-dependency between different local regions, while simultaneously suppressing background distractions for the person search problem. The ARM module422comprises a relation mixer block522and a spatio-channel attention layer524. The relation mixer block522constitutes spatially attended token (spatial) mixing, feature (channel) attended channel mixing, and includes an input-output skip connection. The relation mixer block522generates discriminative relation features which are further enriched by suppressing the unwanted background features by the spatio-channel attention layer524.

In particular, the relation mixer block522captures a relation between different sub-regions (local regions) within an RoI. The resulting features are further enriched by a spatio-channel attention layer524that attends to relevant input features in a joint spatio-channel space. The ARM module422takes RoIAlign pooled feature∈C×H×Was input. Here, H, W, C are the height, width and number of channels of the RoI feature. For computational efficiency, the number of channels are reduced to c=C/4 through a point (1×1) convolution layer502before passing to relation mixer522and spatio-channel attention524blocks.

Relation Mixer Block (522): As mentioned earlier, the relation mixer block522is introduced to capture the relation between different sub-regions (local regions) within an RoI. This is motivated by the fact that the local regions of a person share certain standard prior relationships among local regions, across RoIs of different persons and it is desirable to explicitly learn these inter-dependencies without any supervision. One such module that can learn/encode such inter-dependencies is a MLP-mixer that performs spatial token mixing followed by pointwise feature refinement. See Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al.: Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems 34 (2021), incorporated herein by reference in its entirety. The MLP mixer conceptually acts as a persistent relationship memory that can learn and encode the prior relationships among the local regions of an object at a global level. Despite this, it has been empirically observed that a straightforward integration of MLP mixer on the RoIAlign pooled features leads to sub-optimal results for the person search problem likely due to the diverse objectives of person detection402and re-id404. To address this issue, a simple feature re-using strategy is introduced to the MLP mixer. The feature reusing strategy is motivated by DenseNet, which re-uses the input features of the MLP mixer at its output through a skip connection. See Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K. Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700-4708 (2017), incorporated herein by reference in its entirety. This additionally introduced input-output skip connection along with the default skip connections within token and channel mixers provides complete feature re-use within the ARM module422. The simple feature re-using strategy is not only enabled using the MLP mixer for the first time in the problem of person search, but also provides impressive performance gain over the base framework. To this end, the relation mixer522comprises a spatially attended spatial mixer504and a channel-wise attended channel mixer512along with an input-output skip connection510,520for feature re-use.

FIG.6is a block diagram of the relation mixer block within the ARM module. The relation mixer block522comprises a spatially attended spatial mixing operation530where important local spatial regions are emphasized using a spatial attention before globally mixing them across all spatial regions (tokens) within each channel using MLP-1 shared across all channels. Following this spatial mixing, a channel attention540is performed to emphasize informative channels before globally mixing the channels for each local spatial region (token) using MLP-2 shared across all spatial regions. Additionally, feature re-using is performed within the relation mixer using an input-output skip connection, to simultaneously handle diverse detection and re-id objectives of person search problem.

Spatially attended Spatial Mixer (530): While learning the inter-dependencies of local RoI sub-regions using standard MLP mixer, the background regions are likely to get entangled with the foreground regions, thereby adversely affecting the resulting feature embedding used for the re-id and box predictions. In order to disentangle the irrelevant background information, a spatial attention504is introduced before performing token (spatial) mixing within the MLP mixer for emphasizing the foreground regions. In the spatial attention504, pooling operations are employed along the channel axis602, followed by convolution and sigmoid layers to generate a 2D spatial attention weights Ms∈1×H×W. These attention weights are broadcasted along the channel dimension to generate the spatial attention M′s∈c×H×W. For a given feature′∈c×H×W602, the spatially attended feature map″=′⊙M′s604is obtained. Here ⊙ denotes element-wise multiplication. These spatially attended features (″)604disentangle irrelevant (background) spatial regions from the foreground. These features are (″)604input to a shared multi-layer perceptron508(MLP-1) for globally mixing local features (within″)604across all spatial regions (tokens). The spatially attended spatial mixing strives to achieve accurate spatial mixing and outputs the feature map Q608.

Channel-wise attended Channel Mixer (540): To further prioritize the feature channels of Q608that are relevant for detection and re-id of person instances, a channel attention512is introduced before channel mixing. The channel attention weights Mc∈c××1are generated through spatial pooling, fully connected (fc) and sigmoid layers, which are broadcasted along the spatial dimension to generate the channel attention weights M′c∈c×H×W. Similar to spatial attention, these channel weights are element-wise multiplied with the feature map to obtain channel-wise attended featuremap. The resulting features emphasize only the channels that are relevant for effective channel-mixing within the relation mixing block. The channel mixing540employs another shared MLP (MLP-2)516for global mixing of channel information. As mentioned earlier, the resulting mixed features are further refined through a skip connection610from′602(feature re-using), producing output feature K∈c×H×W612from the relation mixer522.

Spatio-channel Attention Layer (524): The relation mixer block522performs the mixing operations by treating the spatial and channel information in a disjoint manner. But, in many scenarios, all spatial regions within a channel and all channels at a given spatial location are not equally informative. Hence, it is desired to treat the entire spatio-channel information as a joint space. With this objective, a joint spatio-channel attention layer524is introduced within the ARM module422to further improve foreground-background (FG-BG) delineation of RoIAlign pooled features. The spatio-channel attention layer524utilizes parameter-free 3D attention weights obtained based on Yang et al. to modulate the 3D spatio-channel RoI pooled features. See Yang, L., Zhang, R. Y., Li, L., Xie, X.: Simam: A simple, parameter-free attention module for convolutional neural networks. In: International Conference on Machine Learning. pp. 11863-11874. PMLR (2021), incorporated herein by reference in its entirety. These spatio-channel attended features are aggregated with the relation mixer output to produce enriched features O for the person search task. These enriched features are projected back to C channels (∈C×H×W) and taken as input to the res5 block434.

In summary, within the ARM module422, the relation mixer522targets FG-BG delineation and capturing of discriminative relation features in disjoint spatial and channel spaces. The resulting features are further enriched by a spatio-channel attentv cion524that performs FG-BG delineation in a joint spatio-channel space.

Training and Inference

For training and inference, the PS-ARM is trained end-to-end with a loss formulation. That is, the person detection branch402is based on Faster R-CNN such that Smooth-L1 and cross entropy losses are employed for box regression and classifications. For the person re-id branch404, three additional loss terms are employed for regression, classification and re-ID. Both these branches are trained by utilizing an IoU threshold of 0.5 for selecting positive and negative samples.

During inference, the re-id feature is first obtained for a given query by using the provided bounding box. Then, for the gallery images, the predicted boxes and their re-id features are obtained from the re-id branch404. Finally, cosine similarity is employed between the re-id features to match a query person with an arbitrarily detected person in the galley.

EXAMPLES

For purposes of comparison with other state-of-the art methods, experiments are performed on two person search datasets (i.e., CUHK-SYSU) and PRW. See Xiao et al. (2017) and Zheng et al. (2017).

Dataset and Evaluation Protocols

CUHK-SYSU: is a large scale person search dataset with 96,143 person bounding boxes from a total of 18,184 images. See Xiao et al. (2017). The training and testing sets contains 11,206 images, 55,272 pedestrians, and 5,532 identities and test set includes 6,978 images, 40,871 pedestrians, and 2,900 identities. Instead of using full gallery during inference, different gallery sizes are used for each query from 50 to 4000. The default gallery size is set to 100.

PRW: is composed of video frames recorded by six cameras that are installed at different location in Tsinghua University. See Zheng et al. (2017). The dataset has a total 11,816 frames containing 43,110 person bounding boxes. In training set, 5,704 images are annotated with 482 identities. The test set has 2,057 frames are labelled as query persons while gallery set has 6,112 images. Hence, the gallery size of PRW dataset is notably larger compared to CUHK-SYSU gallery set.

Evaluation Protocol: Two standard protocols for person search are followed for performance evaluation of mean Average Precision (mAP) and top-1 accuracy. The mAP is computed by averaging over all queries with an intersection-over-union (IoU) threshold of 0.5. The top-1 accuracy is measured according to the IoU overlaps between the top-1 prediction and ground-truth with the threshold value set to 0.5.

Implementation Details: ResNet-50 was used, pretrained over ImageNet, as a backbone network. See He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016); and Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. IEEE (2009), each incorporated herein by reference in their entirety. Li et al. (2021) was followed and utilized Stochastic Gradient Descent (SGD), set momentum and decay to 0.9 and 5×10−4, respectively. The model for 12 epochs are trained over CUHK-SYSU dataset and 10 epochs over PRW dataset. During training, the batch-size of 5 was used with input size 900×1500 and set initial learning rate to 0.003 which is warmed up at first epoch and decayed by 0.1 at 8th epoch. During inference, the NMS threshold value is set to 0.4. The code is implemented in PyTorch. See Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019), incorporated herein by reference in its entirety.

Comparison with State-of-the-Art Methods

The presently disclosed approach is compared with state-of-the-art one-step and two-step person search methods on two datasets: CUSK-SYSU and PRW. CUHK-SYSU Comparison: Table 1 shows the comparison of the PS-ARM with state-of-the-art two-step and single-step end-to-end methods with the gallery size of 100. Among existing two-step methods, MGN+OR and TCTS achieves mAP of 93.2 and 93.9, respectively. See Yao, H., Xu, C.: Joint person objectness and repulsion for person search. IEEE Transactions on Image Processing 30, 685-696 (2020); and Wang et al. (2020), each incorporated herein by reference in their entirety. Among existing single-step end-to-end methods, SeqNet and AlignPS obtains mAP of 94.8%, 93.1% respectively. See Li et al. (2021); and Yan, Y., Li, J., Qin, J., Bai, S., Liao, S., Liu, L., Zhu, F., Shao, L.: Anchor-free person search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7690-7699 (2021), each incorporated herein by reference in their entirety.

To further analyze the benefits of the ARM module, the ARM module is inserted into a Faster R-CNN based method (NAE method) after RoIAlign pooling. See Chen et al. (2020). The ARM module can provide an absolute gain of 1.9% and 1.8% to the mAP and top-1 accuracies over NAE (see Table 1). The PS-ARM outperforms all existing methods, and achieves a mAP score of 95.2. In terms of top-1 accuracy the method sets a state-of-the-art accuracy of 96.1%.

CUHK-SYSU dataset has different range of gallery sizes such as 50, 100, 500, 1000, 2000, and 4000. To further analyze the disclosed method, an experiment was performed by varying the gallery size. The mAP scores across different gallery size are compared with recent one-stage and two-stage methods as shown inFIG.7.FIG.7is a graph illustrating the state-of-the-art comparison of conventional methods over the CUHK-SYSU dataset with varying gallery sizes. Dotted lines represent two-stage methods whereas solid lines represent one-stage methods. The PS-ARM shows consistent improvement compared to other methods as the size of gallery increases.

PRW Comparison: Table 1 shows the state-of-the-art comparison on PRW dataset. Among the existing two stage methods, MGN+OR achieves the best mAP score 52.3, but with a very low top-1 accuracy. See Yao et al. (2020). While comparing the top-1 accuracy, TCTS provides the best performance, but with a very low mAP score. See Wang et al. (2020). To summarize, the performance of most two-step methods are inferior either in mAP score or top-1 accuracy. See Yao et al. (2020); Chen et al. (2018); Han et al. (2019); Girshick, R., Iandola, F., Darrell, T., Malik, J.: Deformable part models are convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 437-446 (2015); Dong et al. (2020); and Lan et al. (2018), each incorporated herein by reference in their entirety.

Among one-stage methods, NAE and AlignPS, achieved mAP scores of 43.3% and 45.9%. See Chen et al. (2020) and Yan et al. (2021). These methods achieved top-1 accuracies of 80.9% and 81.9%. Among the other one-step methods SeqNet, PBNet, DMRN, and DKD also performed well and obtained more than 46% mAP and have more than 86% top-1 accuracy. See Li et al. (2021); Tian, K., Huang, H., Ye, Y., Li, S., Lin, J., Huang, G.: End-to-end thorough body perception for person search. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12079-12086 (2020); Han, C., Zheng, Z., Gao, C., Sang, N., Yang, Y.: Decoupled and memory-reinforced networks: Towards effective feature learning for one-step person search. arXiv preprint arXiv:2102.10795 (2021); and Zhang, X., Wang, X., Bian, J. W., Shen, C., You, M.: Diverse knowledge distillation for end-to-end person search. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 3412-3420 (2021), each incorporated herein by reference in their entirety.

TABLE 1State-of-the-art comparison on CUHK and PRW test sets in termsof mAP and top-1 accuracy. On both datasets, the PS-ARM performsfavorably against existing approaches. All the methods here utilizethe same ResNet50 backbone. When compared with recently introducedSeqNet, the PS − ARM provides an absolute mAP gain of 5%on the challenging PRW dataset. Also, introducing the novel ARMmodule to a popular Faster R-CNN based approach (NAE), providesan absolute mAP gain of 3.6%. See Chen et al. (2020).CUHK-SYSUPRWMethodmAPtop-1mAPtop-1Two-stepCLSA87.288.538.765.0IGPN90.391.412.970.2DPM——20.548.3RDLR93.094.242.970.2MGTS83.083.732.672.1MGN + OR93.293.852.371.5TCTS93.995.146.887.5End-to-endOIM75.578.721.349.9RCAA79.381.3——NPSM77.981.224.253.1IAN76.380.123.061.9QEEPS88.989.137.176.7CTXGraph84.186.533.473.6HOIM89.790.839.880.4BINet90.090.745.381.7AlignPS93.196.045.981.9PGA92.394.744.285.2DKD93.194.250.587.1NAE+92.194.744.081.1PBNet90.588.448.587.9DIOIM88.789.636.076.1APNet88.989.341.281.4DMRN93.294.246.983.3PGSFL90.291.842.583.5CAUCPS81.183.241.786.0ACCE93.994.746.286.1NAE91.592.443.380.9SeqNet94.895.747.687.6Ours (NAE + ARM)93.494.246.981.4Ours (PS − ARM)95.296.152.688.1

See Lan et al. (2018); Dong et al. (2020); Girshick et al. (2015); Han et al. (2019); Chen et al. (2018); Yao et al. (2020); Wang et al. (2020); Ziao et al. (2017); Chang et al. (2018); Liu et al. (2017); Xiao et al. (2019); Munjal et al. (2019); Yan et al. (2019); Chen et al. (2020); Dong et al. (2020); Yan et al. (2021); Kim, H., Joung, S., Kim, I. J., Sohn, K.: Prototype-guided saliency feature learning for person search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4865-4874 (2021); Zhang et al. (2021); Chen et al. (2020); Tian et al. (2020); Dai, J., Zhang, P., Lu, H., Wang, H.: Dynamic imposter based online instance matching for person search. Pattern Recognition 100, 107120 (2020); Zhong, Y., Wang, X., Zhang, S.: Robust partial matching for person search in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6827-6835 (2020); Han et al. (2021); Kim et al. (2021); Han, B. J., Ko, K., Sim, J. Y.: Context-aware unsupervised clustering for person search. arXiv preprint arXiv:2110.01341 (2021); Chen, S., Zhuang, Y., Li, B.: Learning context-aware embedding for person search. arXiv preprint arXiv:2111.14316 (2021); Chen et al. (2020); and Li et al. (2021), each incorporated herein by reference in their entirety.

To further analyze the effectiveness of the ARM module, the ARM module was integrated to NAE and achieved absolute mAP gain of 3.6% mAP, leading to an mAP score of 46.9%. A similar performance gain was observed over top-1 accuracy, resulting in top-1 score of 81.4%. The ARM module was integrated in Han's method. See Han et al. (2021). Compared to the existing methods, Han et al. utilize a different approach, such as an RoI pooling of 24×8 size, instead of 14×14. To this end, PS-ARM was modified to adapt the setting of Han et al., resulting in absolute gains of 2% and 1.3% improvement on PRW dataset and obtained 55.3% mAP and 89.0% top-1 scores, respectively. The PS-ARM achieved state-of-the-art performance compared the existing one-step and two-step methods. A mAP score of 52.6% and top-1 score of 88.1% was achieved.

TABLE 2Ablation study over the PRW dataset by incrementally adding the novelcontributions to the baseline. While introducing a standard MLP mixerto a baseline, both the detection and re-id performance reduces overthe baseline. The feature re-using strategy helped to improve theperformance over the MLP mixer. The integration of spatially attendedspatial mixing and channel-wise attended channel mixing within therelation mixer captures discriminative relation features within RoIwhile suppressing distracting background features, hence providessuperior re-id performance. Finally, the joint spatio-channel attentionremoves distracting backgrounds in a joint spatio-channel space,leading to improved detection and re-id performance.ReIDDetectionMethodmAPtop-1RecallAPBaseline47.687.696.393.1Baseline + Standard MLP-Mixer45.385.695.792.8Baseline + Standard MLP-Mixer +49.186.896.393.3Feature Re-useBaseline + Relation Mixer51.887.996.693.8PS − ARM (Baseline + ARM)52.688.197.494.1

Qualitative comparison:FIG.8illustrates qualitative comparison between the SeqNet (row 2) and the PS-ARM for the same query input (row 1). See Li et al. (2021). Here, true and false matching results are marked in green and red, respectively. The figure shows top-1 results obtained from both methods. It can be observed that SeqNet provides inaccurate top-1 predictions due to the appearance deformations. The PS-ARM provides accurate predictions on these challenging examples by explicitly capturing discriminative relation features within RoI.

FIG.9illustrates the qualitative results from the PS-ARM. Here the top-2 matching results are shown for each query image. It can be seen that the PS-ARM can accurately detect and re-identify the query person in both gallery images.

Ablation Study

An ablation study was performed on the PRW dataset. Table 2 shows the performance gain obtained by progressively integrating the novel contributions to the baseline. It can be seen that a straightforward integration of a standard MLP mixer on the RoIAlign pooled feature leads to inferior detection and re-id performance. The detection and re-id performance reduces over the baseline from 93.1 to 92.8 AP and 47.6 to 45.3 mAP, respectively. It can be observed that a simple skip connection based feature re-use helped to improve the performance over the MLP mixer by achieving 93.1 detection AP and re-id 49.1 mAP score. With the introduction of spatially-attended spatial mixing and channel-wise attended channel mixing within the relation mixer captures discriminative relation features within RoI while suppressing distracting background features. This resulted in superior re-id performance. Introducing the relation mixer comprising of a spatially attended spatial mixing, channel-wise attended channel mixing and an input-output skip connection leads to an overall AP of 93.8 for detection and 51.8 mAP for re-id. To further complement the relation mixer that performs information mixing in the disjoint spatial and channel spaces, a joint spatio channel attention is introduced. The joint spatio-channel attention removes distracting backgrounds in a joint spatio-channel space, leading to improved detection and re-id performance by achieving 94.1 and 52.6, respectively.

FIG.10illustrates a non-limiting display for an output resulting from the PS-ARM. In an embodiment, the output from PS-ARM may be used to review images for a person search. The display is configured to show a timeline of cameras1002that captured images1006of a person search. In cases where the timeline contains multiple images for a camera1002at a point along the timeline1004, the time point can be expanded to show individual images1008. Each image contains a person image corresponding to the person in the person query. The timeline may be scrolled to display sections of the timeline that extend beyond the display boundaries.

In an embodiment, each icon for a camera1002can further display information about the camera, including the camera properties such as resolution, make and model, and the location of the camera including building address, room identification, location in the room. A summary of information about the camera can be displayed by hovering over the camera icon, or by clicking on the camera icon to bring up a text box displaying full or scrollable information about the camera. Icons for images1006may display overlapping thumbnail images. Expansion of the thumbnail images may be accomplished by dragging the top image of a group of overlapping images.

The purpose of the timeline1004and arrangement of cameras1002along the timeline is to trace movement of a person that is in the search query. Each camera1002represents a camera that captured an image containing the person at a particular time point in the timeline1004.

FIG.11illustrates a non-limiting display for an output resulting from the PS-ARM. In an embodiment, the output from PS-ARM may be incorporated into a display of an area containing video surveillance cameras106,108. The display may be a real time display for a query person as they are detected by one or more of the cameras106,108. The area may be rooms in a building, of several buildings. The display1100may show text information associated with cameras, for example as pop-up balloons. The text information may include a camera ID and data/time period when the searched person image was detected by that particular camera. The display1100may display indications of a sequence of cameras that detected the person images, for example, by displaying a number or alphabetical ordered letter in association with the cameras.

In an embodiment, the display1100is for multiple query person images. The display1100may include a section for submitting multiple query person images. Search results for each query person image may be indicated by a distinct identifier, such as a number or letter displayed in a box or circle, or the searched person's name or initials, or other way of quickly identifying a searched person image in the display1100.

In one embodiment, the cameras in the display may cycle through a display sequence of cameras lighting up to show the time order that the cameras identified a person image based on the person query. In one embodiment, the time order that cameras identified a person image may be used to display an estimated path of the person associated with the query person.

FIG.12is a diagram for a multiple camera surveillance system. In one non-limiting embodiment, multiple cameras1212may be connected in a network, which can be either wired or wireless. Multiple cameras1212may be connected to a common server computer1202. The PS-ARM can be performed in the server computer1202as images are received from the cameras1212. The surveillance system1200includes a database system1220. The server computer1202may include a connected to a cloud service1210. The database system1220may be a stand-alone system or may be a service in the cloud service1210. Various mobile devices1204,1206may communicate with the cloud service1210.

The database system1220may maintain data for the multiple cameras1212, including, but not limited to, a camera ID, camera type, location, IP address, owner/operator, building address, camera characteristics. The database system1220may maintain records and logs of camera recordings. The database system1220may store query person images. The database system1220can store camera videos.

In an embodiment, the surveillance system1200may include a software application for monitoring, controlling, and viewing results that may be performed in a mobile device1204, tablet computer, laptop computer1206or desktop computer1208. The software application may display status information in a manner described with regards toFIGS.10,11.

FIG.13is a block diagram of a computer system in the surveillance system ofFIG.12. The example computer system is a server or workstation for implementing the machine learning training and inference methods according to an exemplary aspect of the disclosure. The machine learning training and inference methods may be a computer program stored in a computer readable storage medium, which can be any non-volatile storage medium, including, but not limited to, a Flash drive, CD-ROM, DVD, as well as in a cloud service, for download to the computer system. The computer system may be an AI workstation running an operating system, for example Ubuntu Linux OS, Windows, a version of Unix OS, or Mac OS. The computer system may also be a mobile device, such as a smartphone or tablet, running a mobile operating system. The computer system1300may include one or more central processing units (CPU)1350having multiple cores. The computer system1300may include a graphics board1312having multiple GPUs, each GPU having GPU memory. The graphics board1312may perform many of the mathematical operations of the disclosed machine learning methods. The computer system1300includes main memory1302, typically random access memory RAM, which contains the software being executed by the processing cores1350and GPUs1312, as well as a non-volatile storage device1304for storing data and the software programs. Several interfaces for interacting with the computer system1300may be provided, including an I/O Bus Interface1310, Input/Peripherals1318such as a keyboard, touch pad, mouse, Display Adapter1316and one or more Displays1308, and a Network Controller1306to enable wired or wireless communication through a network99. The interfaces, memory and processors may communicate over the system bus1326. The computer system1300includes a power supply1321, which may be a redundant power supply.

In some embodiments, the computer system1300may include a server CPU and a graphics card, in which the GPUs have multiple cores. In some embodiments, the computer system1300may include a machine learning engine1312having multiple machine learning cores.

Numerous modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.