Benchmarking person re-identification approaches and training datasets for practical real-world implementations

Abstract: Person Re-Identification (Re-ID) is receiving a lot of attention recently. Large datasets containing labeled images of various individuals have been released, allowing researchers to develop and test many successful approaches. However, when such Re-ID models are deployed in a new city or environment, the task of searching people within a network of security cameras is likely to face an important domain shift, thus resulting in decreased performance. Indeed, while most public datasets were collected in a limited geographic area, images from a new city present different features (e.g., people's ethnicities and clothing style, weather, architecture, etc.). In addition, the whole frames of the video streams must be converted into cropped images of people using pedestrian detection models, which behave differently from the human annotators who created the dataset used for training. To better understand the extent of this issue, this paper introduces a complete methodology to evaluate Re-ID approaches and training datasets with respect to their suitability for deployment for live operations. This method is used to benchmark four Re-ID approaches and three datasets and provides interesting insight and guidelines that can help designing better Re-ID pipelines in the future.

Reviews and author responses:

The authors conduct a systematic analysis of some existing person re-identification approaches. Specifically, they evaluate four publicly available models against three publicly available datasets using standard metrics. They also check how the models generalize to other datasets when not trained on them. Finally, they discuss the results and show which approach and training dataset combination works best in live re-id settings. Strengths:

- They explain the existing models and datasets (their advantages and shortcomings) in easy-to-read gists. So it serves as a good overview tutorial for someone new to the field. 


Weaknesses:

- They do not propose any new models, datasets, or ways to improve current datasets/models. While they introduce two new metrics, it is not clear to me what advantage those metrics have over the standard ones and how they change the ranking of the models in a meaningful way. Maybe the authors need to clarify this. However, that being said, see the point below.  

- Even if the contribution is just a systematic survey (which can be helpful), the paper doesn’t provide any exciting insights into what is learned from these experiments. They motivate the problem by saying different datasets have people from different ethnicities, have different geographical features, architectures, climates, etc. How do these features affect person re-id? What do we learn from all these experiments that can be used as insights to create better datasets? How much does the diversity of these features matter for generalization?

- One of the most important questions these days with this domain of work is how it works across various ethnicities, especially under-represented ethnicities. The survey shows no insights or results in this direction.  

- The model that performs well on just the single dataset evaluation (MGN in Table 2) also seems to work best in live settings (Table 4) regardless of dataset. So, why is dataset+model combo choice "paramount for live settings" according to the authors? Seems like the best model on the live dataset is simply the best model according to the single dataset - nothing surprising here.  In what ways are the results unexpected, and why? While the paper provides a thorough evaluation of some publicly available models and datasets, it doesn't offer any interesting insights into what makes a particular approach, dataset, or metric better or worse. For example, they say model_x + dataset_y works best. Why? Is it because of specific trends in model architecture? Is it because of certain geographical features or ethnicities of people in the dataset? Can we use these experimental insights to create better/inclusive datasets, and how? What are the shortcomings of current metrics, if any? How can we use these experimental results to improve the metrics in that case? 
The authors say that the paper's goal is to learn insights on building better re-id pipelines. But, just showing the best model+dataset on a live re-id dataset does not provide enough insights into how to design datasets/metric/models better. Instead, it just shows how to choose the best model for that specific test dataset. Getting the best model for just some specific test dataset isn't that helpful unless some insights into what features of the test dataset make that model+train_dataset a good fit are provided. 

 We thank the reviewer for his extensive and careful review of our paper. The paper was well understood and summarized by the reviewer. Our contributions are indeed the introduction of new metrics for benchmarking live Re-ID approaches, and to use these metrics to benchmark different standard Re-ID approaches and training datasets regarding how well they can be used straightforwardly for live Re-ID in new contexts. 

## Answers to reviewer concerns:

1. The live Re-ID setting is defined specifically for inference. To evaluate a specific Re-ID implementation, we consider we already have a trained model and we want to know how well it performs for the task of finding people in a network of cameras. In this sense the three settings a), b) and c) should be evaluated in the same way because they only differ by how the models are trained, not how they are used at inference. In this sense, we do not think that the statement regarding live Re-ID is wrong as it actually allow to evaluate Re-ID implementations, regardless if they were trained using a), b) or c). Regarding the fact that only setting a) is considered in this paper, the reviewer is right. However, we believe that knowing which methods are good without additional training can be useful to initialize models even for settings b) or c), as explained in the paper. We agree that it would be interesting to compare different adaptation approaches for the live Re-ID setting, but it is not the focus of this paper. However, the proposed methodology and metrics can be used straightforwardly to conduct such study in future work.

2. - The live Re-ID setting is different from the standard Re-ID setting because the bounding boxes are coming directly from a pedestrian detection model (YxCZG
OLO), and are not curated manually as it is the case for most standard Re-ID datasets. The results obtained illustrate that standard Re-ID “approaches” performing well on the cross-dataset setting perform well on the live Re-ID setting. However, it is not true regarding the best “training datasets”, which suggests that the live Re-ID setting introduces additional domain shift than simply considering the cross-dataset setting. In this sense, the m-PRID dataset is the only one currently existing for the live Re-ID setting (which requires having raw videos as well as annotation regarding bounding boxes locations and associated IDs in the video). This is why we only used a single dataset for the live Re-ID evaluation. However, we agree that it would be great to have more such live Re-ID datasets in the future.

    - Regarding the proposal of using several training datasets together, we agree with the reviewer that it would be a great addition to our experiments to try training on all three datasets and evaluating for live Re-ID to see if it could improve performance. We thank the reviewer for this suggestion and we’ll try to get it done before the end of the rebuttal period.

3. We do not have a proper definition of successful as it is probably application dependent. We reformulated this sentence as follows to ensure the paper is free of irrelevant claim: “Although very pessimistic results were reported in Sumari et al. (2020), our experiments showed that it is possible to obtain much better live Re-ID pipelines by properly choosing the standard Re-ID model.”. Thanks for pointing this out.

4. We agree with the reviewer. This sentence has been replaced by: “Proper choice of the standard re-ID approach and training dataset can influence greatly the results when transferring the model to the live Re-ID setting.” We also added the following in the conclusion: “Although we only studied the straightforward transfer strategy without fine-tuning, we believe that the results presented here can serve as a good starting point to develop better live Re-ID models in the future.”

5. From our results, we actually see that the best “approaches” at cross-dataset evaluation are actually the best at live Re-ID. However, the ranking is not the same for “training datasets”. So we do not thinks that the two sentences are contradicting.

6. Unfortunately, m-PRID is the only available dataset to study the live Re-ID setting so far. A cross-dataset methodology was conducted in the paper, but it is not sufficient to study the generalization of the tested models to the real-world inference setting (live Re-ID).

 We thank the reviewer for his extensive and careful review of our paper. The paper was well understood and summarized by the reviewer. Our contributions are indeed the introduction of new metrics for benchmarking live Re-ID approaches, and to use these metrics to benchmark different standard Re-ID approaches and training datasets regarding how well they can be used straightforwardly for live Re-ID in new contexts. 

## Answers to reviewer concerns:

- The proposed study in this paper is indeed empirical and no specific technical contribution is being proposed. In their study, Sumari et al. [4] introduce a new Re-ID setting, better suited to study practical implementations of Re-ID. They conclude from their experiments that standard Re-ID methods cannot be applied straightforwardly to real-world Re-ID implementations. In this paper, we want to study if some approaches from standard Re-ID can still be useful to design such practical Re-ID applications, and compare different approaches and datasets. We believe that the results obtained and the systematic discussion provided could be useful to the community.

- Due to page limit, we did not evaluate any methods to increase robustness in the cross-dataset setting. Instead, our goal was to propose a simple evaluation pipeline to decide which approach and initial training dataset could be used to build good initial models for live Re-ID. These models can obviously be improved and fine tuned on the new target domain to improve final performance, but it is not the focus of our experiments. However, we agree with the reviewer that studying such unsupervised cross-dataset adaptation approaches could be an interesting extension of this paper for future work. We added the following sentence in the last section: “Another interesting research direction would be to see if existing unsupervised cross-dataset adaptation methods could help generalization of standard Re-ID models  for the live Re-ID setting.”

- We could easily add the results using the baseline from Sumari et al., however, the results obtained are bad as their work intended to show that a good standard Re-ID does not necessarily lead to a good live Re-ID pipeline. Based on this result from Sumari et al., we built an experiment to verify if some standard Re-ID approaches could still be used in the live Re-ID setting, thus allowing to use the large body of research in standard Re-ID to build useful applications. We believe that we answered this question successfully, which should give valuable insights to the community for future research directions.

- The reviewer highlights that end-to-end methods can be used to build better live Re-ID settings. As stated in the paper, we agree on this point, but we also believe that having a good two-step approach can serve as a good initialization for such end-to-end approaches. Thus it is important to conduct studies like the one proposed in this paper.

- “It is common that distribution bias exists between datasets. It is hard to find a standard re-ID approach and a training dataset to address the problem unless the dataset is large enough that can cover as much as scenes.” → The objective of this paper is precisely to provide an empirical methodology and first results to make these important choices (training dataset + approach) easier.

- To mitigate the generalization problem, researchers have indeed proposed to study cross-dataset settings, and to propose cross-dataset methods. The experiments conducted in this paper demonstrate that focusing on the cross-dataset setting to account for the difficulties of implementing real-world applications (live Re-ID) is not sufficient. In particular, cross-dataset results were consistent with live Re-ID results regarding Re-ID approaches but not regarding training datasets. However, we agree with the reviewer that it would be interesting to study if unsupervised methods for cross-dataset adaptation could improve live Re-ID results (we added this discussion in the future works section: “Another interesting research direction would be to see if existing unsupervised cross-dataset adaptation methods could help generalization of standard Re-ID models  for the live Re-ID setting.”).

This paper studies the evaluation methodology to assess if different standard Re-ID approaches and training datasets can be used to build efficient live Re-ID pipeline for real-world deployment. In particular, this paper formalizes the live Re-ID  setting and define two evaluation metrics named mAP (according to True Validation Rate vs Finding Rate) and $F_\gamma$. Extensive experiments of different baselines on three benchmarks under three kinds of evaluation methods, i.e., single-dataset evaluations,  cross-dataset evaluations and live re-ID evaluation, are conducted, which gives insights of building a good live re-ID pipeline. 
 Strengths:
   + This paper is well written, which clearly formulates the problem and gives systematic discussions.
   + This paper presents two new evaluation metrics to evaluate the performance of a live re-ID approach.

Weaknesses:

   + The technical contribution of this paper is limited, which is far from a decent ICLR paper. In particular, All kinds of evaluations, i.e.,  single-dataset setting (most of existing person re-ID methods), cross-dataset setting [1, 2,3] and live re-id setting [4], have been discussed in previous works. This paper simply makes a systematic discussion.
   + For cross-dataset setting, this paper only evaluates standard person re-ID methods that train on one dataset and evaluate on another, but fails to evaluate the typical cross-dataset person re-ID methods, e.g., [1, 2, 3].
   +  For live re-ID setting, this paper does not compare with  the particular live re-ID baseline [4]
   + Though some conclusions are drawn from the experiments, the novelty is limited. For example, 1) most of person re-ID methods build on the basis of pedestrian detector (two-step method), and there are also end-to-end method that combines detection and re-ID [5]; 2) It is common that distribution bias exists between datasets. It is hard to find a standard re-ID approach and a training dataset to address the problem unless the dataset is large enough that can cover as much as scenes. 3) cross-dataset methods try to mitigate the generalization problem.

	[1] Hu, Yang, Dong Yi, Shengcai Liao, Zhen Lei, and Stan Z. Li. "Cross dataset person re-identification." In Asian Conference on Computer Vision, pp. 650-664. Springer, Cham, 2014.
	[2] Lv, Jianming, Weihang Chen, Qing Li, and Can Yang. "Unsupervised cross-dataset person re-identification by transfer learning of spatial-temporal patterns." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7948-7956. 2018.
	[3] Li, Yu-Jhe, Ci-Siang Lin, Yan-Bo Lin, and Yu-Chiang Frank Wang. "Cross-dataset person re-identification via unsupervised pose disentanglement and adaptation." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7919-7929. 2019.
	[4] Sumari, Felix O., Luigy Machaca, Jose Huaman, Esteban WG Clua, and Joris Guérin. "Towards practical implementations of person re-identification from full video frames." Pattern Recognition Letters 138 (2020): 513-519.
	[5] Xiao, Tong, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. "Joint detection and identification feature learning for person search." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3415-3424. 2017.
	
 From the discussion above, though the paper gives a systematic evaluation of benchmarking methodology and presents two new metrics to evaluate live re-ID approaches, there are some drawbacks and the technical contribution and novelty are limited. Thus, it is marginally below the acceptable threshold.

This paper proposes a live peron re-identification (Live Re-ID) setting to evaluate person re-id approaches closer to real application scenarios. The paper benchmarks a few state-of-the-art re-id methods, by training on popular re-id datasets (CUHK03, Market-1501, DukeMTMC) and testing on the m-PRID dataset. Experimental results suggest that a proper selection of the training dataset is important. # Strengths
- Person re-id has been studied for long in the research community, but how to bridge the gap between research and real-world application scenrios is still an open problem. This paper aims at reducing the gap by introducing a methodology (Live Re-ID) to benchmark the re-id approaches closer to real applications. This is a great trial and contribution.

- The paper is well structured and easy to follow.

# Weaknesses
One major weakness of the paper is that the proposed Live Re-ID setting was too straightforward to address the question it intended to answer -- how to benchmark re-id approaches in new environments.

1. In real-world person re-id applications, the target test scenario could be

    a) unknown at all, which requires the re-id method to be as generalized as possible

    b) only having some unlabeled data, which allows some unsupervised domain adaptation methods

    c) having labeled data, which allows supervised domain adaptation methods

The proposed Live Re-ID seemed to only consider the condition a), but did not discuss conditions b) and c) at all. It is hard to claim the Live Re-ID setting *"takes into account all relevant aspects for deploying Re-ID models in practical real-world applications"*.

2. Even for condition a), the paper chose to evaluate only on one dataset (m-PRID), and train with one of the three popular datasets (CUHK03, Market-1501, DukeMTMC). This could be biased by the similarity between the training and test sets. Arguably, one better setting could be using K-Fold cross-validation, i.e., leaving one dataset for validation and using all the others together for training. Repeat the experiments with different datasets for validation and take the averaged / worst / P95 metrics as the final metric. I failed to understand why the proposed Live Re-ID setting is better than this baseline K-Fold cross-validation methodology.

Besides, the conclusions of the paper (in the Section 5) may not be rigorous:

3. *"It is possible to build a successful live Re-ID pipeline ..."* -- How to define "successful"? Are the current results shown in Table 4 good enough? What is the minimum requirement for the method to work reasonably well in real applications? What is the upper bound that the methods can achieve?

4. *"Proper choice of the standard re-ID approach and training dataset is paramount to obtain satisfying results when transferring the model to the live Re-ID setting."* -- Again, why not simply combining all the data available for training? How to define "satisfying"? And the paper did not discuss how to transfer the model (e.g., with domain adaptation).

5. *"Simple cross-dataset evaluation can be used to quickly assess the generalization performance of future standard Re-ID techniques for live Re-ID."* -- This claim is contradicting to the discussions *"the cross-dataset results do not allow to choose
the best dataset for training models for the live Re-ID setting"* in the text.

6. *"To properly assess the performance of new training datasets, the complete evaluation methodology proposed in this paper should be conducted."* -- Again, the paper used only the m-PRID as the test set. The methodology proposed was not technically sounding enough to assess the performance of new training datasets. Overall, I recommend rejecting the paper submission. As elaborated above, the paper did not well address the question it intended to answer (how to benchmark re-id approaches in new environments) and its claims were not well supported. These weaknesses were critical.

Meta-review: All reviewers agree that the presented investigation of existing person re-identification approaches is easy to read and can be used as a tutorial in the field. However, the reviewers raised a number of major concerns including inadequate discussion of insights made from the conducted survey, lack of some related important experimental studies and inadequate/ unconvincing conclusions made in the presented work. The authors’ rebuttal addressed some of the reviewers’ questions but failed to alleviate all reviewers’ concerns. Hence, I cannot suggest this paper for presentation at ICLR.

Decision: Reject