Benchmarking person re-identification approaches and training datasets for practical real-world implementations

1 INTRODUCTION . As many cameras are being deployed in public places ( e.g. , airports , malls , parks ) , real-time monitoring of the video streams by security agents becomes impractical . Automated video processing appears as a promising solution to analyze the whole network in real time and select only relevant sequences for verification by human operators . This paper deals with person Re-Identification ( ReID ) , a computer vision problem that aims at finding an individual in a network of non-overlapping cameras ( Bedagkar-Gala & Shah , 2014 ) . It has diverse potential security applications , such as suspect searching ( Liao et al. , 2014 ) , identifying owners of abandoned luggage ( Altunay et al. , 2018 ) , or recovering missing children ( Deb et al. , 2021 ) , among others . In the literature , the problem of Re-ID is studied under different settings depending on the application context ( see Section 2.1 ) . On the one hand , the most studied Re-ID paradigm , which we refer to as standard Re-ID , tries to find images representing the query person within a gallery of pre-cropped images of persons , containing at least one correct match ( Lavi et al. , 2020 ) . On the other hand , Sumari et al . ( 2020 ) recently introduced a setting considering specifically the constraints related to implementing Re-ID for use during live operations . We call it live Re-ID , and a first contribution of this work is to formalize the definition and constraints associated to this setting . We also extend the live Re-ID evaluation metrics proposed by Sumari et al . ( 2020 ) in order to facilitate interpretation . Standard Re-ID is not the best suited paradigm for practical implementations , as it does not consider the influence of potential domain shift due to pedestrian detection errors or to deployment in a city with different characteristics than the training dataset . Indeed , in their experiments , Sumari et al . ( 2020 ) showed that training a successful Re-ID model with respect to standard Re-ID metrics does not guarantee good performance when evaluated on specific live Re-ID metrics . Nevertheless , most publicly available large scale datasets for Re-ID focus on the standard Re-ID setting , and many successful approaches have been developed for this specific purpose . For this reason , we believe that it is essential to study if these datasets and approaches can be used to implement and deploy practical applications in different contexts . More specifically , the objective of this paper is to answer the following questions : 1 . Which standard Re-ID approaches can be successfully deployed for practical implementations in the live Re-ID setting ? 2 . Which standard Re-ID dataset is better suited to train standard Re-ID models for the live Re-ID setting ? 3 . Do different Re-ID approaches have different optimal datasets for deployment ? 4 . Can we use a simple cross-dataset evaluation methodology to assess the deployability of a given approach-dataset pair ? To answer these questions , we conducted a study using three standard Re-ID datasets and four recent standard Re-ID approaches . For each approach-dataset pair , the Re-ID model obtained was evaluated against the other two datasets and against a fourth dataset configured for the live Re-ID setting . A conceptual overview of the objectives of our study is represented in Figure 1 . Xiao et al . ( 2017 ) showed that considering pedestrian detection and Re-ID separately is not as good as end-toend approaches for person search , i.e. , galleries of whole scene images . However , our results show that this two step approach can perform well on the live Re-ID setting . In addition , we believe that the results from our study can be very useful to pre-train successful initial live Re-ID models and to guide the development of more complex end-to-end architectures for live Re-ID . This paper is organized as follows : Section 2 discusses the relevant related literature . The methodology for the proposed benchmark experiments is detailed in Section 3 . The results are presented in Section 4 and discussed in Section 5 . Finally , Section 6 presents our conclusions and potential future work . 2 RELATED WORK . A complete literature review about Re-ID approaches is not the purpose of this paper . Instead , we present clear definitions of the different existing Re-ID settings in Section 2.1 . Previous benchmark studies about Re-ID are also discussed in Section 2.2 . 2.1 PERSON RE-IDENTIFICATION SETTINGS . The field of Re-ID was first formalized by Gheissari et al . ( 2006 ) , it consists in retrieving instances of a given individual , called the query person , within a complex set of multimedia content called the gallery . The different settings presented here are defined by how they represent the query person , the format of the gallery items , the constraints on the gallery content , the boundaries of the Re-ID system , and the constraints imposed on the evaluation methodology . 2.1.1 POPULAR SETTINGS . Standard Re-ID In the standard Re-ID setting , both the query image ( representing the query person ) , and all items in the gallery are well-cropped images representing the entire body of a person . It is sometimes called closed-set Re-ID as it assumes that the query person has at least one representative in the gallery . According to the statistics in Papers with Code ( PwC ) , it is the most studied Re-ID setting by a large margin , in terms of number of papers , datasets and benchmarks published . Some standard Re-ID datasets and successful methods are used for our benchmark study and presented in Section 3 . For a more complete overview of standard Re-ID approaches , we refer the reader to the surveys by Lavi et al . ( 2020 ) and Ye et al . ( 2021 ) . Person search The person search setting was introduced by Xu et al . ( 2014 ) . It consists in replacing the gallery items by whole scene images ( Xiao et al. , 2017 ) . In other words , a person search model must return not only the index of the gallery image where the query is present , but also its location in terms of Bounding Box ( BB ) coordinates . A survey about person search approaches was proposed by Islam ( 2020 ) . Open-set Re-ID The open-set Re-ID setting was first defined by Liao et al . ( 2014 ) . It differs from standard Re-Id in that there is no guarantee that the query person is represented in the gallery , i.e. , an open-set Re-ID model should be able to answer whether the gallery contains the query . The reader can refer to the survey by Leng et al . ( 2019 ) for an overview of recent open-set Re-ID approaches . Video-based Re-ID The video-based Re-ID setting was first studied by Wang et al . ( 2014 ) . In this setting , all images ( query and gallery ) are replaced by image sequences extracted from consecutive frames of a video . Sequences are composed of well-cropped entire body images representing the same person . A complete review of video-based Re-ID was proposed by Ye et al . ( 2021 ) . Others For completeness , we mention the existence of other Re-ID variants in the literature , namely unsupervised Re-ID ( Yang et al. , 2021 ) , semi-supervised Re-ID ( Moskvyak et al. , 2021 ) , human-in-the-loop Re-ID ( Wang et al. , 2016 ) , or federated Re-ID ( Zhuang et al. , 2020 ) . However , their specificities lie in how Re-ID models are trained while the other settings above focus on constraints at inference time . For this reason , these Re-ID paradigms are not presented further here . 2.1.2 LIVE RE-ID SETTING . In this section , we clearly define and formalize the live Re-ID setting , which is inspired by the work of Sumari et al . ( 2020 ) . It takes into account all relevant aspects for deploying Re-ID models in practical real-world applications . When looking for a query person during live operations , whole scene videos need to be processed in near real time , hence the galleries for live Re-ID are composed of the consecutive whole scene frames from short video sequences . The live Re-ID context is also highly open-set as the probability to have the query in a short video sequence from a given camera is low . Hence , this setting combines elements from several of the Re-ID settings mentioned above . Another key characteristic of live Re-ID is that the training context is different from the deployment context . Indeed , building new specialized datasets for deployment in every shopping mall or small city is unrealistic from the perspective of future advances in the field . This highlights the importance of studying cross domain transfer of Re-ID , which was first discussed and highlighted by Luo et al . ( 2020 ) . Finally , this setting also takes into account that Re-ID model predictions need to be processed by a human security agent , who takes the final decision and trigger appropriate actions . This way , very high rank-1 accuracy is not mandatory for live Re-ID , as the operator can find the query in later ranks . On the other hand , false alarm rates must be kept low to avoid overloading the human operators , who have limited processing capacity . To evaluate these two objectives , Sumari et al . ( 2020 ) introduced two evaluation metrics representing both dimensions of the problem ( see Section 3.3.3 ) . The experiments conducted in this paper aim at studying the transferability of standard Re-ID approaches and datasets for deployment in the live Re-ID setting . 2.2 PERSON RE-IDENTIFICATION BENCHMARKS . Most recent research dealing with Re-ID present comparative evaluation of different approaches . While listing all these papers is out of the scope of this work , this section presents several benchmark studies considering different Re-ID settings or specific aspects of the Re-ID pipeline . A large scale benchmark experiment was conducted by Gou et al . ( 2018 ) to compare various approaches for standard Re-ID and video-based Re-ID . By evaluating more than 30 approaches on 16 public datasets , they produced the largest Re-ID benchmark to date . In addition , they built a new dataset to represent several constraints relevant for real-world implementations , such as pedestrian detection errors and illumination variations , among others . However , they do not consider cross domain performance and all evaluations are conducted in the closed-set setting , which are major limitations regarding future deployments . In addition , a smaller systematic evaluation of video-based Re-ID approaches was proposed by Zheng et al . ( 2016 ) . Another extensive set of experiments was conducted by Zheng et al . ( 2017 ) to evaluate different pedestrian detection models on a two-step person search pipeline . They demonstrated that the best performing models on standard object detection metrics are not necessarily the best suited for Re-ID from whole scene frames . In addition , a first benchmark regarding cross-domain transfer of Re-ID approaches was proposed by He et al . ( 2020 ) . Their experiments consisted in training an approach on one standard Re-ID dataset and evaluating on another . Finally , on another note , Zhuang et al . ( 2020 ) compared different approaches for federated Re-ID , i.e . learning Re-ID across decentralized clients to preserve privacy . The studies presented above have brought valuable insights to the Re-ID community . However , none of them allows to assess the performance of a Re-ID model against all the challenges involved during deployment in a new environment for practical use in security applications . This paper contributes to bridging this gap by conducting experiments within the live Re-ID setting , which was designed to take into account all these challenges . In particular , we consider the influence of different standard Re-ID approaches and training datasets on live Re-ID results .

Summary: The authors conduct a systematic analysis of some existing person re-identification approaches. Specifically, they evaluate four publicly available models against three publicly available datasets using standard metrics. They also check how the models generalize to other datasets when not trained on them. Finally, they discuss the results and show which approach and training dataset combination works best in live re-id settings.