Towards Robustness of Person Search against Corruptions

Authors: Woojung Son, Yoonki Cho, Guoyuan An, Jinhwan Seo, Chanmi Lee, Sung-eui Yoon
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Corruption, Person Search, Robustness

Abstract:
Person search aims to simultaneously detect and re-identify a query person within an entire scene, involving detection and re-identification as a multi-task problem.
While existing studies have made significant progress in achieving superior performance on clean datasets, the challenge of robustness under various corruptions remains largely unexplored.
To address this gap, we propose two benchmarks, CUHK-SYSU-C and PRW-C, designed to assess the robustness of person search models across diverse corruption scenarios.
Previous researches on corruption have been conducted independently for single tasks such as re-identification and detection.
However, recent advancements in person search adopt an end-to-end multi-task learning framework that processes the entire scene as input, unlike the combination of single tasks. 
This raises the question of whether independent achievements can ensure corruption robustness for person search.
Our findings reveal that merely combining independent, robust detection and re-identification models is not sufficient for achieving robust person search. 
We further investigate the vulnerability of the detection and representation stages to corruption and explore its impact on both foreground and background areas.
Based on these insights, we propose a foreground-aware augmentation and regularization method to enhance the robustness of person search models.
Supported by our comprehensive robustness analysis and evaluation framework our benchmarks provide, our proposed technique substantially improves the robustness of existing person search models.
Code will be made publicly available.

Decision: reject
Reviewer scores: 8, 3, 8, 3, 8

Reviews and discussion:
<thread>
creation date:
1730353771421

thread_type:
official_review

<thread_object>
creator: reviewer_ujfi

<content>
summary:
This paper addresses the robustness of person search under various corruption scenarios. The authors introduce two new benchmarks, CUHK-SYSU-C and PRW-C, to evaluate person search models' robustness. Unlike previous studies that focused on independent tasks such as re-identification and detection, this work highlights the inadequacy of merely combining robust detection and re-identification models for robust person search. The authors investigate the sensitivity of detection and representation stages to corruption and its impact on background regions. Based on these insights, they propose a method that significantly enhances the robustness of existing person search models.

strengths:
Overall, this paper takes a practical approach, considering numerous details and unexplored aspects of previous work. It introduces two benchmark datasets and evaluates leading models from both detection and re-identification perspectives. By analyzing the characteristics of foreground and background in the constructed datasets, the authors propose corresponding modules and validate the model's effectiveness.

1.Originality: This paper identifies shortcomings in previous datasets and evaluation methods, proposes a novel data construction approach, and discovers the varying impact of foreground and background on person search. These contributions demonstrate a certain level of originality.

2.Quality: The paper is thorough and provides valuable insights. The introduction and validation of large-scale datasets are of high quality.

3.Clarity: The paper is easy to follow. The problem statement, in particular, is logically clear and easy to understand.

4.Significance: The proposed datasets will assist future researchers in developing more robust end-to-end person search models. Additionally, the identified interference factors and their impacts will guide improvements in detection, re-identification, and other related fields.

weaknesses:
This article has no obvious shortcomings. The author has done a large amount of related work and relevant verification. In my opinion, the workload of this article is sufficient. It also presents visualization analysis in various aspects and in the frequency domain, which is helpful for promoting research in this field.

questions:
1.The author can consider expanding the dataset to the multi-modal re-identification area. Because the corruption types such as darkness and strong contrast constructed in the article are similar to the harsh visual environment in RGBNT201 for the destruction of visible light images. If the corruption can be extended to near-infrared or infrared images, or if LLM is used for text annotation of images and corruption is performed on the text, these are all good expansion ideas.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520175712

thread_type:
official_review

<thread_object>
creator: reviewer_yohg

<content>
summary:
This paper introduces two benchmarks, CUHK-SYSU-C and PRW-C, evaluating the robustness of person search models under corruption scenarios. The authors propose a foreground augmentation and regularization method, improving the robustness.

strengths:
The author examines the shortcomings of existing ReID methods based on experiments, explaining the issues in a clear and easy-to-understand manner. The authors use extensive experiments to demonstrate the drawback of existing methods.

weaknesses:
1.	The five severity levels of the proposed benchmark are not detailed in either the main text or Fig. 1, which is quite important.
2.	The benchmark adds noise to the test sets of existing datasets without collecting additional person images for specific scenarios. I believe this does not qualify as a benchmark that makes a significant contribution. It it better to be considered as an evaluation metric.
3.	The proposed module is an existing augmentation, which I believe is quite incremental in terms of novelty.
4.	CUHK-SYSU and PRW datasets are relatively small, the proposed method has not been validated on larger-scale datasets. Therefore, the scalability of the method is doubted.
5.	Although the author claims that the augmentation method is different from the construction of the benchmark, they use a data augmentation method to solve an augmented test set of existing dataset. This is intuitive and within expectation, and I did not find anything particularly innovative in this approach.
6.	Several related works are missing, including but not limited to “SAT: Scale-Augmented Transformer for Person Search”, and “Making person search enjoy the merits of person re-identification”. Especially, the method of SAT incorporates the augmentation into transformer, whose contribution if believed to be of value.
7.	There are a lot typos in the manuscript, such as the symbols for the loss functions are inconsistent in Eq 5.

questions:
1.	What is the five severity levels of the proposed benchmark, or can you quantatitively describe the differences between them?
2.	It is better to compare more baselines other than the five baselines in Tab. 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528434958

thread_type:
official_review

<thread_object>
creator: reviewer_tury

<content>
summary:
Previous studies have only investigated the corruption issues in person detection or person re-identification separately, whereas this paper is the first to study the corruption robustness in the person search task (detection + re-identification). The authors meticulously constructed two benchmarks in corruption scenarios, CUHK-SYSU-C and PRW-C, and through extensively evaluating models on these two benchmarks, they obtained some meaningful findings. Subsequently, based on these findings, the authors proposed several methods to improve the model's robustness against corruption, and validated the effectiveness of these methods through extensive experiments. Overall, this is a very solid work and I like it.

strengths:
1. The writing of this paper is smooth and easy to understand. The authors structured it by first posing the problem, then constructing the benchmarks, evaluating current models to identify areas for improvement, proposing improvement methods, and finally validating the effectiveness of the methods. I am able to follow the core ideas of this work very well.

2. According to the authors, they are the first to study the issue of corruption robustness in the person search task, a problem which I believe has significant practical relevance. In real-world applications, due to the influence of weather, lighting, and other imaging conditions, the sursurveillance videos are prone to corruption. A person search model that can withstand such corruptions is essential.

3. The contributions of this paper are substantial. The paper introduces two corrupted benchmarks CUHK-SYSU-C and PRW-C (as well as an additional manual annotated test set based on BDD100K), and proposes two modules to enhance the model's resistance to corruption, all of which have positive implications for the development of this field.

4. The paper provides rich details in its experimental procedures, offering good reproducibility.

weaknesses:
Although the entire work is very solid, it relatively lacks novelty at the technical level. Through evaluation and analysis, the authors concluded that foreground corruption and robust person representation are important. Therefore, they proposed two methods: Foreground-Aware Augmentation and Regularization for Robust Person Representation. I carefully reviewed the technical details of these two methods and found them to be somewhat lacking in innovation, as they are based on existing techniques that have already been implemented.

questions:
1. Will you make your entire code, model, and dataset public? I believe this would be significant for the development of the field.

2. Since you pointed out that person re-identification representation is important, have you tried to investigate the effect of some person reid pre-trained models on the robustness of the person search model against corruptions?  To my knowledge, pre-training can learn good person re-identification representations, and I would be very interested to see an experiment where you investigate the impact of pre-trained person representations on your task. For instance, you could conduct an experiment with the pre-trained ResNet50 in LUP, UPReID, LUP-NL, ISR, PLIP and CION. 

I will greatly appreciate it and **raise the score** if you could conduct the aforementioned experiment that interests me.

Reference:

LUP: Unsupervised pre-training for person re-identification (CVPR2021)

UPReID:  Unleashing potential of unsupervised pre-training with intra-identity regularization for person re-identification (CVPR2022)

LUP-NL: Large-scale pre-training for person re-identification with noisy labels (CVPR2022)

ISR: Identity-seeking self-supervised representation learning for generalizable person re-identification (ICCV2023)

PLIP: Plip: Language-image pre-training for person representation learning (NeurIPS2024)

CION: Cross-video Identity Correlating for Person Re-identification Pre-training (NeurIPS2024)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538774781

thread_type:
official_review

<thread_object>
creator: reviewer_d5hd

<content>
summary:
This paper explores the robustness of person search models in degraded data environments. It introduces two new benchmark datasets (CUHK-SYSU-C and PRW-C) for evaluating the performance of person search models under different degraded conditions. By analyzing the sensitivity of both the detection and feature extraction stages under corruption, the authors found that existing models exhibit significant vulnerability when foreground images are damaged. Based on these findings, a foreground-aware data augmentation and regularization method was proposed to enhance model robustness.

strengths:
This paper is the first to investigate the robustness of person search models in degraded data environments. It introduces two novel benchmark datasets (CUHK-SYSU-C and PRW-C) for evaluating the performance of person search models under various degradation scenarios.

weaknesses:
1. The strength of this paper lies in the introduction of two new benchmark datasets for exploring person search in interference scenarios. However, overall, the dataset construction methods are largely adapted from existing approaches, and the proposed solution is relatively simple, focusing on narrowing the feature gap between corrupted and original images. From a theoretical innovation standpoint, the contributions are limited and fall short of ICLR's high standards.

2. While the foreground-aware augmentation approach is effective, it appears to be an incremental improvement rather than a fundamentally new concept. The paper could be strengthened by deeper theoretical insights into why this method improves robustness beyond empirical results.

3. The paper employs various existing data augmentation methods to simulate real-world disturbances. However, the key question is whether these augmentations can effectively benefit real-world scenarios. If their impact is limited, the constructed datasets may hold little significance.

questions:
1. The experiments demonstrate that foreground damage has a greater impact on model performance. However, it is unclear whether this damage affects the detection performance more or the recognition performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730824898046

thread_type:
official_review

<thread_object>
creator: reviewer_fkyp

<content>
summary:
This paper focus on the corruptions of images in person search problems. Two benchmarks are proposed for evaluation, while the analysis is adopted on both detect and reID stages. A method with data augmentation and regularizaiton is further proposed. Good experimental resutls are achieved.

strengths:
1. The motivaion of evaluationg the affects of image corruption for person search is useful.
2. The analysis results and discussion about the performance dropping in both detection and searching stages are acceptable.

weaknesses:
1. It is hard to regard as a contribution via simply post-processing two exisiting datsasets. The human-designed or generated augmentation is far from the real scene. 
2. The further proposed forground-aware augmentation is not novel, which has been a common sense that the key region for accurate person search is the body region.
3. The referred methods in the experiments are most out-of-date, missing some recently proposed methods.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733309311798

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Global Response by Authors

comment:
Dear reviewers and chairs,

We appreciate all reviewers for their reviews and efforts during the rebuttal.

We are glad that the reviewers found our work "unexplored aspects or good motivation" (FKyP, d5hd, tUry, ujfi), "novel benchmarks or assisting future development" (d5hd, tUry, ujfi), "useful analysis for the field" (ujfi, FKyP), "easy to understand" (Yohg, tUry, ujfi), "extensive experiments" (Yohg, tUry, ujfi).
- **Our benchmarks provide the foundation to stimulate further research** in this area, by pioneering an unexplored aspect of person search.
- **We measure the robustness of 12 seminal person search models**, providing valuable insights for the person search community.
- **The analyses and observations** can guide improvements in related fields, with their validity confirmed by consistent results from two experiments conducted during the rebuttal.
- **Our method addresses the robustness vulnerabilities revealed in seminal models**, providing substantial improvements to their performance.
- **We are pleased by positive feedback on our presentation**, such as  'easy to understand', 'logically clear', 'able to follow the core ideas well', which has been further improved by reviewers' valuable efforts during the rebuttal.

---

We are pleased that our responses have addressed concerns and received positive feedback with increased ratings, which has been very encouraging us.

To summarize the points we have addressed:

>### **The value of our task-specialized benchmarks: Beyond traditional real-world new data collection**
1. Although our framework leverages some existing techniques, ***its design is specialized with task-specific insights for evaluating person search framework***, beyond merely employing the existing approach. (Please refer to our response for Reviewer FKyP for more detail)
   - During the rebuttal, we have shown the value of our benchmark that it can bridge the distribution gap towards the real world.
2. Compared to collecting new data from actual real world, our approach offers distinct advantages: ***Cost-effective, Diverse covering, Scalable, and Privacy-issue-free***, where our approach of establishing corruption environments has been extensively demonstrated to serve as a solid foundation across numerous computer vision fields.
   - **Diverse covering**: While the cost becomes easily infeasible when attempting to cover various real-world scenarios
, our framework 18 diverse scenarios.
   - **Scalable**: While the data collection from real world becomes dependent on the specific situation where it was collected, our framework is scalable, leveraging various collecting environments of other datasets, as verified with larger scale dataset conducted by the request of Reviewer Yohg.
   - **Privacy-issue-Free**: As the person search demands careful ethical consideration, introducing our approach of constructing benchmark is especially beneficial for this field.

>### **Simple-yet-effective baseline for robust person search**

Preface: *The birth of our robust person search baseline was possible thanks to the foundations for robustness framework our benchmarks provide. We hope our method, alongside our benchmark, will encourage future endeavors.*

Our method, though relatively simple in its design, derives from our robustness analysis and observations of the person search framework, showing to be simple-yet-effective result. 

Also, its simplicity enables its broad applicability across various person search models. While keeping clean set performance within ±2%, it pushes the robustness to a large margin.

Even though we leverage some existing techniques, we have designed our method tailored to the person search task, based on our experimental observations.

We provide the verification of our robustness solution regarding the scalability, generalizability and deeper analysis: extensive validation across diverse models (Tab 3); generalizable towards real corruption (Tab 5); CAM analysis (Fig 8).

>### **About scalability of validation result of our method requiring more real-world evaluation**

We understand Reviewer Yohg's main remaining concern that real-world validation result of our method might not be scalable due to potential unknown corruptions. To address this, we had analyzed our method from a frequency sensitivity perspective.

The Fig 6 demonstrates that applying our method to the baseline shows improving resilience to a wide range of frequency perturbations including low-, middle- and high-frequency. Yin et al. (2019) analyzed the common corruptions in the frequency domain, revealing that fog and contrast have relatively low-frequency components, noise-related corruptions have high-frequency components, blur and pixelate have middle-frequency components. Given this comprehensive coverage of our method across the frequency spectrum, we believe our approach to be adaptable to unknown corruptions in other frequency ranges as well.

Sincerely,

Authors of submission 7041
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311123403

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Remarks for Final Consideration: Comment by Corresponding Author

comment:
Dear Reviewers and Area Chairs,

Thank you for the thoughtful feedback and valuable discussions throughout the review process. We deeply appreciate the time and
effort you have dedicated to evaluating our work.

We acknowledge the limitations of our study, as noted by some reviewers. Specifically, our benchmarks rely on augmentations and lack
the scale and diversity of large, real-world datasets. Developing such datasets would require substantial resources and significant time
investment—likely one to two years or more.

Nevertheless, we believe our contribution is an important step forward in addressing robustness challenges in person search. As highlighted by reviewers, our work introduces a novel perspective by focusing on the robustness of end-to-end person search under corruptions, a critical yet underexplored area. The benchmarks we propose, while not exhaustive, provide a foundation for evaluating and understanding the vulnerabilities of existing and follow-on models.

It is worth noting that many existing techniques underperformed on our benchmark, not due to intrinsic shortcomings but because they were not designed with such scenarios in mind. The absence of similar benchmarks or evaluation protocols in the past has limited the
development of robustness-aware methods. By addressing this gap, we aim to spark innovation in this direction.

We hope that by sharing our current dataset with the research community, we can catalyze efforts to develop more robust techniques
for person search. These efforts will, in turn, pave the way for the creation of larger and more diverse real-world datasets, driving the
field forward in the coming years.

Once again, we sincerely thank the reviewers for their constructive comments, which have greatly improved the quality of our work. We are committed to incorporating this feedback and contributing further to this important area of research.

Best regards,

Corresponding Author
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received two negative and three positive reviews. Positive reviews praised the paper's valuable exploration of the impact of image corruption on person search tasks, especially its relevance to real-world surveillance affected by weather and lighting. The authors introduce two corrupted benchmarks (CUHK-SYSU-C and PRW-C) and propose modules to enhance model robustness, addressing gaps in previous datasets and evaluation methods. The paper is well-structured, clear, and offers original insights into foreground and background effects, making a notable contribution to person search research. However, the negative reviews raised several concerns: the proposed benchmark lacks detailed severity levels, and the dataset mainly augments existing ones, limiting its originality. The module offers only incremental improvements, with minimal novelty, and has not been validated on larger datasets, questioning its scalability. Additionally, key related works are missing, and there are typographical errors. Overall, while the benchmark introduces some new elements, it lacks significant theoretical innovation and does not fully simulate real-world scenarios, limiting its applicability.
After reviewing the authors’ response, the Area Chair (AC) remains concerned that the benchmarks are not sourced from real-world environments and that the scalability of the method has not been adequately addressed. Therefore, the Area Chair recommends rejecting the paper.