Benchmarking person re-identification approaches and training datasets for practical real-world implementations

1 INTRODUCTION . As many cameras are being deployed in public places ( e.g. , airports , malls , parks ) , real-time monitoring of the video streams by security agents becomes impractical . Automated video processing appears as a promising solution to analyze the whole network in real time and select only relevant sequences for verification by human operators . This paper deals with person Re-Identification ( ReID ) , a computer vision problem that aims at finding an individual in a network of non-overlapping cameras ( Bedagkar-Gala & Shah , 2014 ) . It has diverse potential security applications , such as suspect searching ( Liao et al. , 2014 ) , identifying owners of abandoned luggage ( Altunay et al. , 2018 ) , or recovering missing children ( Deb et al. , 2021 ) , among others . In the literature , the problem of Re-ID is studied under different settings depending on the application context ( see Section 2.1 ) . On the one hand , the most studied Re-ID paradigm , which we refer to as standard Re-ID , tries to find images representing the query person within a gallery of pre-cropped images of persons , containing at least one correct match ( Lavi et al. , 2020 ) . On the other hand , Sumari et al . ( 2020 ) recently introduced a setting considering specifically the constraints related to implementing Re-ID for use during live operations . We call it live Re-ID , and a first contribution of this work is to formalize the definition and constraints associated to this setting . We also extend the live Re-ID evaluation metrics proposed by Sumari et al . ( 2020 ) in order to facilitate interpretation . Standard Re-ID is not the best suited paradigm for practical implementations , as it does not consider the influence of potential domain shift due to pedestrian detection errors or to deployment in a city with different characteristics than the training dataset . Indeed , in their experiments , Sumari et al . ( 2020 ) showed that training a successful Re-ID model with respect to standard Re-ID metrics does not guarantee good performance when evaluated on specific live Re-ID metrics . Nevertheless , most publicly available large scale datasets for Re-ID focus on the standard Re-ID setting , and many successful approaches have been developed for this specific purpose . For this reason , we believe that it is essential to study if these datasets and approaches can be used to implement and deploy practical applications in different contexts . More specifically , the objective of this paper is to answer the following questions : 1 . Which standard Re-ID approaches can be successfully deployed for practical implementations in the live Re-ID setting ? 2 . Which standard Re-ID dataset is better suited to train standard Re-ID models for the live Re-ID setting ? 3 . Do different Re-ID approaches have different optimal datasets for deployment ? 4 . Can we use a simple cross-dataset evaluation methodology to assess the deployability of a given approach-dataset pair ? To answer these questions , we conducted a study using three standard Re-ID datasets and four recent standard Re-ID approaches . For each approach-dataset pair , the Re-ID model obtained was evaluated against the other two datasets and against a fourth dataset configured for the live Re-ID setting . A conceptual overview of the objectives of our study is represented in Figure 1 . Xiao et al . ( 2017 ) showed that considering pedestrian detection and Re-ID separately is not as good as end-toend approaches for person search , i.e. , galleries of whole scene images . However , our results show that this two step approach can perform well on the live Re-ID setting . In addition , we believe that the results from our study can be very useful to pre-train successful initial live Re-ID models and to guide the development of more complex end-to-end architectures for live Re-ID . This paper is organized as follows : Section 2 discusses the relevant related literature . The methodology for the proposed benchmark experiments is detailed in Section 3 . The results are presented in Section 4 and discussed in Section 5 . Finally , Section 6 presents our conclusions and potential future work . 2 RELATED WORK . A complete literature review about Re-ID approaches is not the purpose of this paper . Instead , we present clear definitions of the different existing Re-ID settings in Section 2.1 . Previous benchmark studies about Re-ID are also discussed in Section 2.2 . 2.1 PERSON RE-IDENTIFICATION SETTINGS . The field of Re-ID was first formalized by Gheissari et al . ( 2006 ) , it consists in retrieving instances of a given individual , called the query person , within a complex set of multimedia content called the gallery . The different settings presented here are defined by how they represent the query person , the format of the gallery items , the constraints on the gallery content , the boundaries of the Re-ID system , and the constraints imposed on the evaluation methodology . 2.1.1 POPULAR SETTINGS . Standard Re-ID In the standard Re-ID setting , both the query image ( representing the query person ) , and all items in the gallery are well-cropped images representing the entire body of a person . It is sometimes called closed-set Re-ID as it assumes that the query person has at least one representative in the gallery . According to the statistics in Papers with Code ( PwC ) , it is the most studied Re-ID setting by a large margin , in terms of number of papers , datasets and benchmarks published . Some standard Re-ID datasets and successful methods are used for our benchmark study and presented in Section 3 . For a more complete overview of standard Re-ID approaches , we refer the reader to the surveys by Lavi et al . ( 2020 ) and Ye et al . ( 2021 ) . Person search The person search setting was introduced by Xu et al . ( 2014 ) . It consists in replacing the gallery items by whole scene images ( Xiao et al. , 2017 ) . In other words , a person search model must return not only the index of the gallery image where the query is present , but also its location in terms of Bounding Box ( BB ) coordinates . A survey about person search approaches was proposed by Islam ( 2020 ) . Open-set Re-ID The open-set Re-ID setting was first defined by Liao et al . ( 2014 ) . It differs from standard Re-Id in that there is no guarantee that the query person is represented in the gallery , i.e. , an open-set Re-ID model should be able to answer whether the gallery contains the query . The reader can refer to the survey by Leng et al . ( 2019 ) for an overview of recent open-set Re-ID approaches . Video-based Re-ID The video-based Re-ID setting was first studied by Wang et al . ( 2014 ) . In this setting , all images ( query and gallery ) are replaced by image sequences extracted from consecutive frames of a video . Sequences are composed of well-cropped entire body images representing the same person . A complete review of video-based Re-ID was proposed by Ye et al . ( 2021 ) . Others For completeness , we mention the existence of other Re-ID variants in the literature , namely unsupervised Re-ID ( Yang et al. , 2021 ) , semi-supervised Re-ID ( Moskvyak et al. , 2021 ) , human-in-the-loop Re-ID ( Wang et al. , 2016 ) , or federated Re-ID ( Zhuang et al. , 2020 ) . However , their specificities lie in how Re-ID models are trained while the other settings above focus on constraints at inference time . For this reason , these Re-ID paradigms are not presented further here . 2.1.2 LIVE RE-ID SETTING . In this section , we clearly define and formalize the live Re-ID setting , which is inspired by the work of Sumari et al . ( 2020 ) . It takes into account all relevant aspects for deploying Re-ID models in practical real-world applications . When looking for a query person during live operations , whole scene videos need to be processed in near real time , hence the galleries for live Re-ID are composed of the consecutive whole scene frames from short video sequences . The live Re-ID context is also highly open-set as the probability to have the query in a short video sequence from a given camera is low . Hence , this setting combines elements from several of the Re-ID settings mentioned above . Another key characteristic of live Re-ID is that the training context is different from the deployment context . Indeed , building new specialized datasets for deployment in every shopping mall or small city is unrealistic from the perspective of future advances in the field . This highlights the importance of studying cross domain transfer of Re-ID , which was first discussed and highlighted by Luo et al . ( 2020 ) . Finally , this setting also takes into account that Re-ID model predictions need to be processed by a human security agent , who takes the final decision and trigger appropriate actions . This way , very high rank-1 accuracy is not mandatory for live Re-ID , as the operator can find the query in later ranks . On the other hand , false alarm rates must be kept low to avoid overloading the human operators , who have limited processing capacity . To evaluate these two objectives , Sumari et al . ( 2020 ) introduced two evaluation metrics representing both dimensions of the problem ( see Section 3.3.3 ) . The experiments conducted in this paper aim at studying the transferability of standard Re-ID approaches and datasets for deployment in the live Re-ID setting . 2.2 PERSON RE-IDENTIFICATION BENCHMARKS . Most recent research dealing with Re-ID present comparative evaluation of different approaches . While listing all these papers is out of the scope of this work , this section presents several benchmark studies considering different Re-ID settings or specific aspects of the Re-ID pipeline . A large scale benchmark experiment was conducted by Gou et al . ( 2018 ) to compare various approaches for standard Re-ID and video-based Re-ID . By evaluating more than 30 approaches on 16 public datasets , they produced the largest Re-ID benchmark to date . In addition , they built a new dataset to represent several constraints relevant for real-world implementations , such as pedestrian detection errors and illumination variations , among others . However , they do not consider cross domain performance and all evaluations are conducted in the closed-set setting , which are major limitations regarding future deployments . In addition , a smaller systematic evaluation of video-based Re-ID approaches was proposed by Zheng et al . ( 2016 ) . Another extensive set of experiments was conducted by Zheng et al . ( 2017 ) to evaluate different pedestrian detection models on a two-step person search pipeline . They demonstrated that the best performing models on standard object detection metrics are not necessarily the best suited for Re-ID from whole scene frames . In addition , a first benchmark regarding cross-domain transfer of Re-ID approaches was proposed by He et al . ( 2020 ) . Their experiments consisted in training an approach on one standard Re-ID dataset and evaluating on another . Finally , on another note , Zhuang et al . ( 2020 ) compared different approaches for federated Re-ID , i.e . learning Re-ID across decentralized clients to preserve privacy . The studies presented above have brought valuable insights to the Re-ID community . However , none of them allows to assess the performance of a Re-ID model against all the challenges involved during deployment in a new environment for practical use in security applications . This paper contributes to bridging this gap by conducting experiments within the live Re-ID setting , which was designed to take into account all these challenges . In particular , we consider the influence of different standard Re-ID approaches and training datasets on live Re-ID results .

Summary: This paper studies the evaluation methodology to assess if different standard Re-ID approaches and training datasets can be used to build efficient live Re-ID pipeline for real-world deployment. In particular, this paper formalizes the live Re-ID  setting and define two evaluation metrics named mAP (according to True Validation Rate vs Finding Rate) and $F_\gamma$. Extensive experiments of different baselines on three benchmarks under three kinds of evaluation methods, i.e., single-dataset evaluations,  cross-dataset evaluations and live re-ID evaluation, are conducted, which gives insights of building a good live re-ID pipeline.