Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper studies the robustness of ReID models under a diverse range of image corruptions. The type of corruptions considered in the paper follow the work of [14] with a few additional augmentation. By benchmarking 21 recent ReID models on 6 popular ReID datasets, the study reveals that majority of recent ReID models did not make much progress on corruption robustness. Based on their experiments, the authors propose a strong baseline that is more robust to input corruption compared to other baseline models trained with popular practices. 
strengths: * Experiments are extensive and major recent ReID models have been evaluated on standard ReID benchmark datasets.
* Study is well motivated and addresses important aspect for real-world applications of ReID models
* Makes proper connections and references to previous findings by relevant prior work
* In addition to corruption benchmark, they also study the effect of data augmentation and cross-dataset performance
weaknesses: See comments below in correctness and clarity sections. Please have these concerns addressed in revision.
correctness: Authors mention "Corrupted images can be seen as heterogeneous data for clean images, and our corruption invariant benchmarks as a setting in the open world". However, open world setting specifically mean the setting where the identities differ between query and gallery and this statement is misleading. 

The authors claim and experimentally show that soft random erasing is doing less harm than hard random erasing. However, according to Figure 4, soft random erasing also consistently seem to harm corrupt robustness as erasing degree increases. What is the main motivation to use erasing augmentation to begin with? Do it make model more robust compared to 0 erasing rate? 
clarity: Generally, paper reads well. However, clarity on experiment could be improved.

It would be good to specify somewhere in the paper what kind of the augmentation and evaluation set (corrupt eval or query or gallery) were used for the corrupt input performance reported in Figs 1,3,4. 

Figure 2 - (b) not very clear or convincing that model trained with soft random erasing and random patch mixing capture more discriminative parts. Also readability of the figure could be improved by writing augmentation method used for each activation map instead of putting it in the caption.
relation to prior work: The paper make good connections to many prior works.
additional feedback: n/a
review_title: Review for Paper 49

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new benchmark for studying the performance of re-ID models on corrupted person images. The benchmark builds on top of several most commonly used re-ID datasets and applies various corruption transformations on the datasets. With the benchmark, extensive experiments are conducted and some conclusions have been drawn. Moreover, a strong baseline is proposed which reaches competitive performance for both single and cross-modality re-ID experiments. 

strengths: 1) This paper  is well motivated. In real world, data collected by surveillance cameras often suffer from various corruptions, e.g., occlusion, light change, insufficient light exposure, various weather conditions, etc.  Although existing methods have reached impressive performance on the clean datasets, they might not be satisfactory  when applied in practice. The paper aims to study this problem and provides a benchmark to systematically evaluate the re-ID performance under the more realistic settings. 

2)  The benchmark is comprehensive. It includes 6 datasets with both single and multiple modalities, 20 types of corruptions commonly occurred in real world, and 5 levels of severity.

3) The experiments are extensive. Up  to 21 recent re-ID methods are employed to be evaluated on the benchmark.

4) Some interesting observations have been reached, which could inspire future developments for this topic.

5) A strong baseline has been proposed which reaches competitive performance for different experimental settings.

 
weaknesses: 1) When evaluating methods on the corrupted datasets, it is unclear whether these methods are trained on corrupted data.  If the methods are trained with corruptions, it is surprising that the performance is so low. This is because if the models is trained to be robust with various corruptions, they should perform reasonably well when applied on the test data.

2) Following the above, if the methods are trained with corrupted data, how do you execute the training processing? What is the sampling strategy and how do you apply different corruptions? Are the training processing the same for all the compared methods?

3) All the corruptions are mixed together; it is unclear to see the impacts of each individual corruption. This is interesting because in practice we usually know the types corruptions that are more likely to be suffered considering that mounting location of the surveillance cameras. For example, indoor cameras are unlikely to have raining and foggy problems. 

4) The ablation study of the proposed baseline should be included in the main text, rather than in the supplementary material. They are essential to evaluate the contributing role of each components. 
correctness: The claims are corrected. 
clarity: Overall it is well written.
relation to prior work: Relation to prior work has been clearly presented. 
additional feedback: See the above.
review_title: Interesting study but missing some key evaluations

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper is about the robustness of person re-identification methods with respect to image corruptions. The main contribution is an extensive study of 21 current methods on 6 datasets under the proposed corruption settings. The main findings are that a transformer based model retains significantly more performance under corruption than all other models, and that cross dataset generalization is correlated with corruption robustness.

strengths: S1 - The evaluation is extensive and covers a wide range of methods and datasets. Thus, the benchmark is valuable for the community.

S2 - The findings are interesting. Although it seems somewhat logical, that by design transformers will be more robust to some of the corruptions, the findings should lead to improved models in the future. Further, it is surprising that there seems to be no correlation between the performance on clean data and the performance on corrupted data.

weaknesses: Overall, there is not much technical innovation in this paper, as it uses the corruption strategy of [14] for the person re-identification task (plus a new rain corruption and a soft version of some augmentation strategies). However, since this is a benchmark, I do not see a strong necessity for technical novelty.

W1 - There are some statements that could be quantified numerically. For example, the paper claims that there is no obvious correlation between clean and corrupted performance. This could be actually measured with a correlation test so that this statement becomes stronger. The same applies for other statements such as the correlation between corruption performance and generalization.

W2 - The message of the paper gets slightly diluted by the introduction of modified augmentation strategies and adjusted loss terms. Ideally, a benchmark should only compare existing methods in new settings or under new criteria/data and not introduce novel methodologies. Here, the  problem is not very severe since the introduced modifications are in the same spirit as the benchmark, but a cleaner separation in terms of writing could be possible. 

W3 - The heat maps in Fig2b and the supplement do not really show anything, as far as I can tell. For a proper comparison the visualization should be done on the same image. Even then, heatmaps are a very weak tool to show what a model has learned.

correctness: I am not an expert in person re-identification. As far as I can tell, the findings and reasoning in the paper is correct and founded in experimental evidence.

clarity: The paper is clear in terms of writing and presentation of the findings of the benchmark. 


relation to prior work: The relation to existing dataset and state of the art methods is sufficiently pointed out.


additional feedback: The paper does not need to be anonymized for single blind reviewing.
review_title: Technical Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a new benchmark about corruptions of images in person re-identification datasets.
The paper directly uses the corruptions from ImageNet-C to create the corrupted test set for person reid with a newly added corruption subtype: rain.
The experiments made some observations that the random erasing augmentation, BNNeck helps under corruptions, and the paper established a strong baseline for corruption invariant person reid.
The paper also found that cross-domain generalization may be connected to the corruption invariance.

However, this paper doesn't include documentation for a licensing and maintenance plan, and the dataset about DukeMTMC (and IMO other ReID datasets) may have potential ethical issues, so I tend to vote for rejection at the current stage if the ethical issues are not properly resolved.

Post rebuttal:
I think the author response resolve my concerns about ethical issues, I raised my score to 7.
strengths: 1. The benchmark is important for real-world applications on person reid, and the author presents an extensive comparison of current methods.
2. The connection between cross-domain generalization and corruption invariant seems very interesting, I wonder if that conclusion generalizes to other cross-domain tasks other than reid?
3. The proposed strong baseline for corruption invariant reid seems to work well.
weaknesses: 1. The paper doesn't provide documentation about the licensing and maintenance plan.
2. My biggest concern is the ethical issues that may arise with the use of DukeMTMC dataset, The DukeMTMC dataset's author has already taken the DukeMTMC dataset offline because of ethical issues, https://www.dukechronicle.com/article/2019/06/duke-university-video-analysis-research-at-duke-carlo-tomasi https://freedom-to-tinker.com/2020/10/21/facial-recognition-datasets-are-being-widely-used-despite-being-taken-down-due-to-ethical-concerns-heres-how/, and the derivatives of DukeMTMC dataset is also removed from online e.g. https://github.com/liminxian/DukeMTMC-SI-Tracklet.
I don't know if other ReID datasets (Market 1501 etc.) have similar concerns, but I think this benchmark may enable unintended and unethical researches, so I vote for rejection.
correctness: Yes
clarity: Yes, the paper is well written.
relation to prior work: Yes
additional feedback: No
review_title: Benchmarks for Corruption Invariant Person Re-identification

Decision: Accept