Title: ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Person Re-identification

Multi-Modality

Benchmark
Abstract: In real-word scenarios, person re-identification (ReID) expects to identify a person-of-interest via the descriptive query, regardless of whether the query is a single modality or a combination of multiple modalities. However, existing methods and datasets remain constrained to limited modalities, failing to meet this requirement. Therefore, we investigate a new challenging problem called Omni Multi-modal Person Re-identification (OM-ReID), which aims to achieve effective retrieval with varying multi-modal queries. To address dataset scarcity, we construct ORBench, the first high-quality multi-modal dataset comprising 1,000 unique identities across five modalities: RGB, infrared, color pencil, sketch, and textual description. This dataset also has significant superiority in terms of diversity, such as the painting perspectives and textual information. It could serve as an ideal platform for follow-up investigations in OM-ReID. Moreover, we propose ReID5o, a novel multi-modal learning framework for person ReID. It enables synergistic fusion and cross-modal alignment of arbitrary modality combinations in a single model, with a unified encoding and multi-expert routing mechanism proposed. Extensive experiments verify the advancement and practicality of our ORBench. A range of models have been compared on it, and our proposed ReID5o gives the best performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on person re-identification and proposes a novel task, omni multi-modal person re-identification (OM-ReID), which has been underexplored. The task enables person image search via any modality or their combinations. Based on this, the authors construct ORBench, the first high-quality five-modality dataset (RGB, infrared, color pencil, sketch, text). In addition, the authors also propose a unified model named ReID5o that can effectively achieve multi-modal combined retrieval. Extensive experiments validate the task's rationality, dataset's quality/generalisability, and method's superiority. Overall, the paper makes solid contributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper proposes a novel and underexplored task, omni multi-modal person ReID. Aiming to enable person search via any combination of modalities, this new task holds both research and practical application value. \n2. This paper presents a high-quality five-modality person ReID dataset, ORBench. With rich diversity, it is the first dataset to feature five high-quality modal representations of the same individual, effectively promoting research development in this field.\n3. This paper proposes a simple yet effective multi-modal joint learning framework, ReID5o. If open-sourced, this framework would be of some significance for the community to follow up on this new task.\n4. The authors conducted extensive ablation experiments to verify the effectiveness of each component of the ReID5o method, and performed in-depth evaluations on the high quality and cross-domain generalizability of the proposed dataset.\n5. The paper is fluently written, with clear elaboration of research motivations and technical details, making it easy for readers to follow.\nWeaknesses:\n1. The RGB and infrared modality data in this paper are filtered from existing datasets. If the authors could independently collect new data for these modalities, it might have greater community influence.\n2. To ensure data quality, the data construction process in this paper appears to require substantial manual involvement, which somewhat limits the scalability of the dataset.\n3. The painting styles in ORBench seem somewhat monolithic. Enhancing the diversity of painting styles might better improve generalization ability in real-world scenarios.\nOverall, this paper makes significant contributions, with its strengths outweighing the weaknesses.\n\nQUESTIONS:\n1. In the process of constructing the ORBench dataset, it seems that you filtered out a large number of low-quality RGB images from existing datasets. What was your motivation for doing so? Because low-quality photos often appear in real-world ReID applications.\n2. In Table 1, those cross-modal methods also seem to perform well. Considering that these methods only support RGB and text, how did you train and test these methods on your dataset?\n3. Even though some visualized qualitative comparison examples, such as Figure 7, effectively demonstrate the high quality of ORBench compared to existing multi-modal datasets, could you conduct comparisons using quantitative metrics? This would make the findings more convincing.\nI would greatly appreciate it if you could address these concerns in the rebuttal.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor the real-world challenge of arbitrary single modalities or modality combinations queries in Person ReID, which existing methods hard to deal with, this paper proposes a diverse and practical benchmark: ORBench, with text, RGB, infrared, colored pencil drawing, and sketch modality. In addition, the authors also propose a new method ReID5o for this task, which contains a unified encoder and an expert routing mechanism and achieves flexible queries with arbitrary modality combinations. Extensive experiments demonstrate the advancement and practicality of the proposed benchmarks and indicate the SOTA performance of ReID5o against previous works.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The experiments are sufficient to demonstrate the advancement of the proposed benchmark and method.\n2. The proposed benchmark has great practical and academic significance, which is able to provide a high-quality multi-modal data platform for the community.\n3. The presentation is easy to follow.\n\n**Weaknesses:**\n\nSee questions and limitations.\n\nQUESTIONS:\n* Please discuss whether there is a conflict between learning the complementary information of modalities for fusion queries and fully mining the effective information for single modality queries, and if there is, how to conduct the trade off?\n\n* Please report the the space complexity, and the time cost of training stage and reference stage in different baselines. \n\n* When applying this foundation model on ReID tasks, would more parameters achieve better performance?\n\n*  The authors removed unclear samples from constructed dataset, but didn't literally or empirically analysis the robustness of the proposed model against possible noise in the real world.\n\nLIMITATIONS:\n**1.** The possible approaches of enhancing the robustness of the model to real-world noise should be discussed in detail in A.7.\n\n**2.** The computing cost is relatively large, which may limit its faster and wider application.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the largely overlooked problem of Omni Multi-modal Person Re-identification (OM-ReID), which involves retrieving a person using any combination of five modalities: RGB, infrared, sketch, color pencil, and text. To support this task, the authors present ORBench, a high-quality dataset featuring all five modalities across 1,000 identities. Additionally, they propose ReID5o, a unified framework that employs a multi-modal tokenizing assembler and a multi-expert routing mechanism to facilitate flexible modality fusion and alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1 The paper presents ORBench, a novel five-modality dataset characterized by its high diversity and quality, providing a valuable foundation for advancing research in more practical and versatile multi-modal ReID settings.\n\n2 ReID5o employs a simple yet effective feature mixture strategy with multi-head self-attention and transformer blocks to combine inputs from arbitrary modality combinations. This design facilitates interactive information exchange, allowing the model to harness complementary cues without complex architectural modifications.\n\nWeaknesses:\n\n1 Most modalities in the dataset introduced by this paper are artificially generated rather than captured from real-world sources. Specifically, the Sketch and Color Pencil images are derived from the original RGB images through repeated transformations. This raises questions about the model’s ability to train effectively and generalize well, as the dataset may not fully capture the complexity of real-world multimodal perception.\n\n2 Although the authors introduce a new dataset, ORBench, most of the baseline methods are re-implemented and fine-tuned on this dataset by the authors themselves. However, the paper lacks cross-dataset transfer evaluations on established benchmarks, making it unclear whether ReID5o's superior performance stems from genuine model effectiveness or potential biases inherent in the dataset design.\n\n3 ReID5o incorporates several modality-specific tokenizers, a shared transformer encoder, and a multi-expert routing mechanism, resulting in increased model complexity. However, the paper lacks a thorough discussion of the trade-offs between efficiency and performance during training and inference, as well as the model's scalability under constrained hardware resources—factors that are critical for real-world deployment.\n\n4 To construct multimodal query combinations, the authors randomly sample data from different modalities but of the same identity. This approach can result in overly consistent intra-identity pairs, potentially reducing the realism and difficulty of the evaluation. In particular, when the sketch, color pencil, and text are all derived from the same RGB image, the model may exploit superficial shortcuts instead of truly demonstrating cross-modal generalization capabilities.\n\nQUESTIONS:\nPlease refer to Weakness.\n\nLIMITATIONS:\nPlease refer to Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new dataset for multi-modal person re-identification, ORBench, and a novel multi-modal reidentification framework, ReID5o.\n\nORBench is constructed by combining images from two existing public person re-ID datasets,\nSYSU-MM01 [46] and LLCM [57], and extending these with additional multi-modal annotations:\npencil sketches, color sketches and text descriptions. The pencil and color sketches are created from the original images using (commercial) image-to-image AI software tools. The text descriptions have been manually created by paid workers. Additional manual curation has been applied to remove unreliable samples, resulting in 1000 people, 45k RGB, 45k text, 26k IR and 18k color/grayscale sketches. Compared to other Re-ID datasets, ORBench is unique in its inclusion of all of these modalities together, and its large amount of color and grayscale images. \n\nReID5o is a multi-modal transformer trained to retrieve a person from a reference set using multi-modal observations/descriptions of that person. The multi-modal inputs are first tokenized by modality-specific encoders, and then processed by a transformer. The transformer uses low-rank (LoRA) decomposition for feature extraction, where the LoRA transformation is specific to the input modality associated with each token. Finally, a Feature Mixing stage fuses information from all procided input modalities to produce a single descriptor.\nThe paper presents experiments to show ReID5o outperforms various baseline methods on ORBench. It also shows strong dataset generalization of ORBench-trained models compared to models trained on other datasets, and includes ablation studies on ReID5o components, and a study on the strength of the different modalities.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The proposed ORBench dataset combines more diverse modalities (rgb, ir, text, grayscale & color pencil) compared to other ReID datasets, allowing to test new multi-modal settings.\n-  A new multi-modal architecture is proposed, ReID5o, which combines existing insights in a sensible manner. A single ReID5o model can perform retrieval on a single modality, a few modalities, or all modalities. It outperforms multiple other multi-modal retieval methods on ORBench.\n- An extensive ablation study is included to validate the components of ReID5o, and also inspect the impact of the different modalities in ORBench.\n- Additional experiments show improved generalization of the ORBench-trained model compared to the model trained on other datasets; The paper also includes qualitative results to support the findings. Overall, the evaluation on the new dataset is thorough.\n- In general the paper is clear and easy to follow.\n\n# Weaknesses\n\n## Methodological details are unclear\n\n- the proposed Feature Mixture module\nline 218: \"For each multimodal features Z^mod, we perform a sequential combinatorial traversal.\"; I don't understand this line. As I understand it Z^mod here is the feature vector from a single modality (since mod \\in {R,I,C,S,T}). I don't understand how a single feature can be processed combinatorically.\n\n- As I understand it the network is used to generate feature vectors from the input (multi-modal) inputs; I assume the setup is as follows  as in a standard image retrieval task: the network turns a query into a descriptor, and compares it to descriptors of an available reference dataset. I did not find this explicitly stated in Method section 3. Instead, the section refers to \"identity classification\" which made me think the network was trained to classify a fixed set of person identities known at train time (which would not generalize to new identities). Only at the experimental setting was it made explicit that training identities and test identities are different, hence I believe this is not a classification task but a metric learning task. \nIf it is indeed an retrieval task, then the used retrieval procedure should be stated explicitly (e.g. Euclidean or cosine comparison, any top-k results voting, etc.). Also, the size of the resulting feature size used for retrieval ($D$ from Section 3.4, I believe) should be stated for reproducibility.\n\n## Unclear how laborious creating the dataset was\n\nPapers that introduce new datasets often represent a significant engineering and data curation effort, and sharing such data represents a valuable service to the research community. In return, as reviewer we should expect less methodological novelty for such papers.\n\nAt first glance, this paper appears to introduce a new large dataset representing significant labour that allows the community to better study some real-world phenomena. But upon closer inspection, the new dataset mostly combines and extends existing data sources. The novel annotations that have been added are generated by \"off the shelf\" AI models.\n\nThe most laborious efforts seems to be the text descriptions. It is stated that 45k text descriptions were created from each RGB image by hired workers (line 119), but Figure 7 shows pretty similar description formats. Did the annotators use a tool to help them with this, and did this not introduce other biases and repeated phrasing? Might the workers not produce lower quality descriptions for the last 5k compared to the first 5k descriptions? More details on the annotation setup should be provided. I also wonder why AI tools were used for the drawing modalities, but why no LLMs were used to generate these text descriptions. \n\nIt might sound pedantic, but understanding the effort to build such a dataset is important to judge the significance and value of the dataset towards the community, and also how critical to review the dataset's realism and design choices.\n\nThis brings me to my next concern:\n\n## How realistic is the dataset, and have meaningful choices been made?\n\nI am missing this higher-level motivation for adding new modalities as has been done in this work:\n- What real-world setting(s) are exactly being simulated?\n- Were these AI-generated annotated added simply because they appear to look good, or do these type of sketches reflect the intended real-world use cases?\n- In the intended use cases, can we expect all modalities to have the same level of detail and completeness? What kind of variance within and across modalities can we expect?\n\nCurrently, it is unclear what practical use cases the dataset exactly targets, and if the new annotations represent the these use cases well, also in terms of data variance. My overall impression is that main motivation for the design choices is pragmatic: The experiments show the new dataset improves generalization to other ReID datasets.\n\nThere is some discussion on the realism in the appendix (A.7 \"Limitation\"), but I believe the paper should be explicit in its goals, assumptions, and simplifications when introducing the data in the paper itself.\n\n## Ethical concerns\n\nFirst off, let me acknowledge that the authors address several concerns in the appendices:\n- Appendix A.5 \"Broader Impact Discussion\" discusses risk of such surveillance technology, and encourages researchers using their data to follow the law; It is also explained the data is only released for research purposes, not commercial use.\n- Appendix A.6 \"Controlled Release\" indicates that the authors plan to release the data in a controlled manner, requiring users to \"adhere guidelines and restrictions\" to access the data and code.\n\nNevertheless, I have several concerns that might require closer inspection by an ethics expert:\n\n1. **Privacy of people in the benchmark.**\nThe paper relies on already public benchmarks, and does not make claim about the privacy of the individuals by itsself. However, by extending and updating the benchmark, and publishing it at a highly visible venue as NeurIPS, I believe the paper should make explicit how it established that the used data does follow all guidelines. This may require explicitly stating how the used public benchmarks protected privacy, and how those policies are consistent with the NeurIPS Ethics Guidelines. If those public datasets did _not_ make clear how such privacy concerns were addressed, then extending those datasets might not be suited for a NeurIPS publication. I find the current argument in de appendices A.5 & A.6 too weak: all responsibility to avoid privacy violations is delegated to the end users of the dataset and models complying with law. But the paper does not make any attempt itself to check for such violations.\n\n2. **Bias and discrimination** This research is clearly about identifying human subjects, I did not find any discussion on potential bias in the dataset, and risks for discrimination, risk of predicting protected categories, etc. These areas (bias & fairness, discrimination, predicting protected categories in surveillance) are explicitly mentioned in the ethics guidelines, but not discussed. I expect a minimum would be to indicate what kind of population the data represents, what type of variations in people it contains. This too might require checking the information from the original public datasets, and here too I wonder if those datasets should be extended in a NeurIPS paper if their compliance with its guidelines cannot be verified. Furthermore, I am not sure if the human text annotations would enable to look for \"protected categories\" as the ethics guideline states. Could this research enable surveillance technology that retrieves individuals with a particular ethnicity, or some age, or a disability using a simple text query?\n\n3. **Answers in questions in the NeurIPs checklist do not align with what can be read in the paper submission**\n  - Question 12 \"Licenses of existing assets\"  -> authors answered \"Yes\",\nbut there is no mention of the license of the datasets on which their proposed ORBench builds. There is also no (intended) license mentioned for ORBench.\n  - Question 13 \"New assets\" -> the authors answered \"[N/A]\".\nThe paper _does_ introduce new assets by extending the dataset with new annotations. None of the guidelines to Question 13 have been considered: No details on the license of the dataset, no discussion on how consent was obtained from people in the dataset, no anonymization.\n  - Questions 14 & 15, the authors reply \"Our paper does not involve crowdsourcing nor research with human subjects\", \nBut the paper is about having descriptions of human subjects; Also, the paper stated that external manual labour has been used to generate textual descriptions (line 119) and to perform a public evaluation of the dataset quality (line 348)\n\nI am not saying these necessarily _are_ ethics violations, but my impression is that the answers do not acknowledge what the checklist and NeurIPs guidelines are for. I believe it would be good if an ethics expert reviews the submission.\n\nQUESTIONS:\nSee my discussed Weaknesses for more detail:\n\n- Technical: how are the features exactly fused in the Feature Mixture module?\n- What are the use cases / motivation for adding the selected modalities?\n- How laborious was creating this dataset, and what design choices were considered?\n- To what extent have you actively checked if the public datasets, on which ORBench is built, comply with the NeurIPS ethical guidelines?\n\nLIMITATIONS:\nSome limitations are discussed in the Appendix, which is good.\nStill, I list sev3eral ethical concerns in the Weaknesses above, which also touch on societal issues.\nI would encourage the authors to discuss bias & fairness of the data, and if this research could lead to discrimination e.g. by searching for people with particular race, sex, age, etc.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}