LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification
Authors: Yiding Lu
Mouxing Yang
Dezhong Peng
Peng Hu
Yijie Lin
Xi Peng
Venue: ICML 2025 poster (2025)
Keywords: Person Re-Identification
Interactive Retrieval
Primary area: applications->computer_vision

Abstract: Traditional text-based person ReID assumes that person descriptions from witnesses are complete and provided at once. However, in real-world scenarios, such descriptions are often partial or vague. To address this limitation, we introduce a new task called interactive person re-identification (Inter-ReID). Inter-ReID is a dialogue-based retrieval task that iteratively refines initial descriptions through ongoing interactions with the witnesses. To facilitate the study of this new task, we construct a dialogue dataset that incorporates multiple types of questions by decomposing fine-grained attributes of individuals. We further propose LLaVA-ReID, a question model that generates targeted questions based on visual and textual contexts to elicit additional details about the target person. Leveraging a looking-forward strategy, we prioritize the most informative questions as supervision during training. Experimental results on both Inter-ReID and text-based ReID benchmarks demonstrate that LLaVA-ReID significantly outperforms baselines.

LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-ldentification
Yiding Lu
Mouxing Yang'
Dezhong Peng"
Peng Hu!
Yijie Lin"
Xi Peng" 12
https: httpa//////o//einnngggea SCU/ ReID
Abstract
The who took jewelry tall. perhops fest, wearing plaid shirt and corrying black beg.
Traditional text-based person ReID assumes that person descriptions from witnesses are complete and provided at once. However, in real-world scenarios, such descriptions are often partial or vague. To address this limitation, we introduce a new task called interactive person re-identification (Inter-ReID). Inter-RelD is a dialogue-based re- trieval task that iteratively refines initial descrip- tions through ongoing interactions with the wit- nesses. To facilitate the study of this new task, we construct a dialogue dataset that incorporates multiple types of questions by decomposing fine- grained attributes of individuals. We further pro- pose LLLVA VA- a question model that gener- targeted questions based visual and textual contexts to elicit additional details about the target person. Leveraging a looking forward strategy. we prioritize the most informative questions as su- pervision during training. Experimental results on both Inter-RelD and text-based ReID benchmarks demonstrate that performs baselines.
What color of parts was he wearing?
Witness
dark pants.
LLaVA-ReIb
Witness
Was he carrying onything besides black beg?
Ohi Yes, is carrying purple cloth bag!
LLOVA-ReID
Witness
Gat
LLoVA-ReID
noking-forwarr
Figure I. illustrated example interactive person re- identification. The red circles highlight the distinctive details in candidate images that inquiry process needs focus on.
significantly out-
the suspect: "The man was tall, perhaps 6 feet, wearing plaid shirt and carrying black bag". as depicted in Figure 1. For Holmes, ever the skeptic, vague impressions are never enough and every detail matters. He investigates further: "What color of pants anything besides black bag? Did he limp or walk with peculiar gait?" Much like an artist refining blurry sketch, Holmes sharpens the picture of the suspect through targeted inquiries about details, gradually uncovering the full truth.
1. Introduction
Never trust general impressions, my boy, but concentrate your- self upon details.
he wearing? Was he carrying
Sherlock Holmex
Imagine Sherlock Holmes standing in a dimly lit room, ques- tioning a witness to a recent crime. The witness, nervous but determined, recounts the few fleeting moments he observed
Consider Holmes equipped with an interactive tool, a system designed help him ask increasingly refined and precise questions based on the witness's evolving descrip- tion and the candidate suspects assessed from these clues. Rather than passively accepting general descriptions, this tool empowers Holmes to analyze both the description and the candidate suspects, dynamically adapting questions in real time. By guiding the witness to recall specific, crucial details about the suspect's appearance or behavior, the sys-
'College of Computer Science, Sichuan University, China Naational Key Laboratory of Fundamental Algorithms and Models for Engineering Numerical Simulation. Sichuan University, China. "Correspondence Yijie Lin <linyijic gm cmmm..com> Peng <pengx gm comi.
Proceedings of the at International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

-eeID- Selective Multi-image Questioner for Interactive Person Re-Identification
tem continuously refines the portrayal of the suspect with each response, enabling Holmes to pinpoint the criminal
2. Related Work
2.1. Text-to-Image Person Re-identification
more effectivvly.
Person re-identification has gained significant attention as critical task in surveillance and security systems (Ye et al., 2021), focusing on matching individuals across non- overlapping camera views (Yang et al., 2022) and varying modalities (Ye et al., 2020; Gong et al., 2024; Shi et al., 2024). Among the diverse Re-ID challenges, rext-to-image RelD aims to retrieve the corresponding image of a person based on textual descriptions. The primary challenge lies in effectively aligning fine-grained visual features with lan- guage descriptions in a joint embedding space. Existing methods can be broadly categorized into two approaches: network architecture design and training objective design. The first category (Li et al., 2017; Gao et al., 2021; Shao et al., 2022) focuses on developing fine-grained fusion net- works such as multi-scale interaction (Yan et al., 2023) mechanisms to encourage the alignment of visual details. The second category introduces auxiliary tasks to improve the fine-grained alignment between body regions and textual entities, such random masking (Jiang & Ye, 2023; Bai et al., 2023), color prediction (Wu et al., 2021; Gong et al., 2022), and attribute prediction (Zuo et al., 2023).
In this paper, we introduce interactive person re- identification (Inter-RelD), a framework that learns to do the same: iteratively refines an initial description through ongo- ing interaction with the witness. To the best of our knowl- edge, no prior work has explored this new problem and the closest paradigm is text-based person ReID (T-ReID). However, T-ReID (Ye et al., 2021; Ding et al., 2021; Zuo et al., 2024) is fundamentally different, as it assumes a static, single-shot description provided in isolation. In con- trast, Inter- Reer-RReD closely mirrors real-world scenarios, where initial descriptions are often partial and vague, requiring in- teractive refinement to accurately identify the target person.
To facilitate the study of this novel task, we construct new dataset, Intera, which incorporates: i) coarse- grained descriptions to simulate the initial, partial queries provided by witnesses, ii) fine- geeeeerriedd descriptions that capture rich, detailed visual characteristics, simulating the witness's latent memories, and iii) multi- dialogues de- rived by decomposing fine-grained descriptions into diverse questions, addressing detailed attributes of individuals.
In addition to the dataset, we address the challenge of gen- erating specific and targeted questions to improve accuracy and adaptability, focusing on two core aspects: i) Represen- tative Candidate Selection: we design a hard-pass selection model that identifies a subset of representative candidates from the gallery, emphasizing the critical differences be- tween individuals. ii) Informative Question Generation: leveraging the distinct attribute-based questions provided in Interactive-PEDDER PEDEE, we propose a looking-forward strategy to dynamically select the most informative questions as su- pervision based on their potential information gain in each round of interaction. The contributions of this paper are as follows:
These approaches typically assume that descriptions are complete and well-structured. However, in real-world sce- narios, witnesses rarely provide detailed and comprehen- sive account of the target person's 's appearance, often result- ing in partial and vague descriptions. To address this limita- tion, we propose an interactive ReID framework that incor- porates multiple rounds of dialogue between the witness and the retrieval system. This iterative process enables the sys- tem to refine the initial description by actively engaging the witness with targeted questions. Through active question- ing, our system uncovers missing details, thus improving retrieval performance. While some recent works (Tan et al., 2024; Zuo et al., 2024; 2023) attempt to annotate more fine- grained descriptions, they focus on improving fme-grained alignment rather than guiding the witness to supplement overlooked details through a dialogue-based approach.
We introduce a new task, Interactive Person Re- Identification, which goes beyond traditional T-ReID by incorporating interaction with witnesses to improve both accuracy and adaptability. To support this task, we con- struct a tailored dataset featuring multi-round dialogues, enabling more effective training and evaluation of Inter- RelD systems.
2.2. Interactive Cross-Modal Retrieval
Interactive cross-modal retrieval involves refining the initial query by engaging the user to provide additional informa- tion or clarification. Several studies have introduced inter- active mechanisms in text-to-image retrieval (Levy et al., 2023; Lee et al., 2024; Zhu et al., 2024) and text-to-video retrieval (Madasu et al., 2022; Liang & Albanie, 2023), investigating various interaction formats (Kovashka et al., 2015; Cai et al., 2021; Lee et al., 2021; Ding et al., 2025). Among these formats, question-answering using free-form text dialogue has shown the most promise, as it closely mir- rors the natural way humans interact (Madasu et al., 2022).
We propose LLaVA-RelDD a multi-image question gen- erator capable of identifying fine- ine-grained differences be- tween collections of images, leveraging both visual and textual contexts from candidate persons and dialogue his- tory. Comprehensive experiments demonstrate that our method not only enhances performance in Inter-ReID but also benefits existing T-Rell tasks.
2

---

LLaVA-RelD: Selective Multi-image Questioner for Interactive Person Re- Re-Identification
Step 1:
Step 2: Sub-caption Decomposition
Step 3:
Coarse-to- Captioning
Dialogue Generation
Coarse: & Fine Grained Describing
Initial Description Excluding
Diverse Question Selecting
Onitall Bpcinption Follow up Desoristion
Descriptive Question
Destribing Instruction
Phruses Splitting
Yes/No Question
Muthiple Choice Question
Follow-up Descrpption
Initial Description:
QdA Pair Formulating
The woman wears long black cost with a scarf and carries a bag.
Pants
Shoes
Bigg
Hoir
Ql: Gan you deserbbe her ponts?
Fine-grained Description:
The womon wears olive green, wide- logged pants.
Al: She wears olive green, vide-legged ponnts
woman wears long black with scarf and carries a bag. She wears olive green, wide-legged pontis On her feet is black dlip-on shoes with distingtive lespard print detail. She carries longe tate in shades grey and purple, Her was long, flowing down her back, soomed light brown shode. She has slim build...
Q.2: boes her shoes have specific fecturre?
her feet are block slip-on shoes with a distinctive leapard print detaill
A2: Yes, her shoes have leapard grint detail
(3: Is she carry a nory blue boy?
A3: No. she corvies toote bag shades gnoy purple.
3, She corries a lorgs tate bay in shooles of gray and purple.
Qr: what can you tell me about hor hor??
Her hair long, seemed be a light brown shode.
A) mediun-lengie -lnn--ehhhhh 83 long and brown
c) short and curly D) Not sure.
Figure 2. Illustration of automated dialogue data construction pipeline. Step I: Generate coarse and fine-grained descriptions. Step 2: Decompose follow-up descriptions into distinct attributes. Step 3: Formulate diverse Q&A pairs.
For example, SimIRV (Liang& & Albanie, 2023) designs heuristic question generator that asks about the attributes of objects in the image. Inspired by the context-aware capabil- ities of large language models (LLMs), ChatR (Levy et al., 2023) leverages LLMs to generate the next question based on few-shot examples. More recently, PlugIR (Lee al., 2024) proposes further enhancing performance by filtering redundant questions and rewriting the user query.
3.1. Interactive Person Re-ldentification
Interactive person re-identification is multi-round dialogue and retrieval process where an interactive system collabo- rates with a witness to identify the target person Lgt from an image gallery 9 = {41, 12...... ., Im using dialogue. The process begins with the witness providing an initial textual description T of the target person. However, this initial de- scription is often incomplete vague, necessitating further clarification. To address this, the system employs iterative dialogue: in each round f, it generates a question Qe to guide the witness in recalling additional details about the target person. The witness responds to Q: with an answer A,, and the resulting question-answer pair is incorporated into the dialogue context D, {[T, (Q1, Ai)...., (Qu, A.), which is used to identify the target person.
However, these methods primarily focus on general appli- cation scenarios and lack the domain-specific knowledge required for fine-grate fine-grained retrieval tasks (Lin et al., 2023). To address this, we introduce the first customized dataset for training and evaluating Inter-RelD. The dataset includes subtle clothing features, accessories, and other distinctive attributes, enabling the questioner to effectively handle the nuances between individuals. Furthermore, we propose a question generator based on the large multi-modal model (Li et al., 2024b), which integrates both visual and textual con- texts from candidate images and dialogue inputs. In contrast, existing methods first transform images into textual descrip- tions using image caption models (Li et al., 2022), which might lack the fine-grained visual details and multi-modal perception required for context-aware question generation.
By leveraging the accumulated dialogue context, the system continuously refines its understanding of the target person, improving both retrieval accuracy and the generation of more precise, discriminative questions. This interactive cy- cle repeats until the target person identified or a predefined maximum number of rounds reached.
3.2. Dataset Construction
3. Interactive-PEDES
To simulate real-world scenarios where witnesses provide in- cremental and useful feedback, Interactive- PEDES includes both person images and corresponding dialogue for retrieval tasks. The dataset comprises 54.749 images of 13,051 indi- viduals, collected from CUHK-PEDES (Li et al., 2017) and ICFG-PEDES (Ding et al., 2021). Each image is annotated with a dialogue that includes an initial description and an in- teractive dialogue detailing the person. The construction of the dataset involves three key steps: coarse-to-fing caption-
In this section, we introduce our interactive person re- identification dataset, Interactive-PEDDE. We first define the interactive person re-identification task (Section 3.1) and then describe the creation of the dataset (Section 3.2).
3

---

LLaVA-ReID; Selective Multi-image Questioner for Interactive Person Re-Identification
ing, sub-caption decomposition, and dialogue generation, as illustrated in Figure 2.
witness provide specific information.
Yes/No Questions present an assumption about the target person, prompting the witness to confirm or deny the pres- ence of particular feature. As Agatha Christie wrote in Dumb Witnsss "I have often had occasion to notice how, where a direct question would fail to elicit a response, a false assumption brings instant information in the form of a contradiction." " Even when the assumption is incorrect, such questions can help stimulate the witness's s memory about specific aspects of the target person. To simulate real-world scenarios, we design questions that could clicit either "yes" or "no" answers, ensuring a balanced and realistic mix of both correct and incorrect assumptions.
Coarse-to- to-ree Captioning To generate dia- logues in lnter-RelD, it is essential to define both the general impression and also more specific finer attributes that could recalled by the witness. For this purpose, we propose a coarse-to-fine captioning strategy utilizing GPT-4o (Hurst et al., 2024). Specifically, we first generate goarse-graines descriptions (i.e., initial description T) that primarily fo- cus the overall appearance of the person, using the in- structions randomly sampled from Table 5. Coarse- -ranied descriptions are typically brief' and concise, simulating the initial high-level descriptions provided by the witness. Next, we generate fine-grained descriptions F based on the initial descriptions, using the detailed prompt in Table 6. Fine- grained descriptions delve deeper into the details of the person, including attributes such as distinctive logos, acces- sories, hairstyle, walking posture, and even the environment or surroundings in which the witness encounters the person. Fine- gene-graindd descriptions are designed to simulate the latent memories of a witness, capturing distinguishing features that contribute to identifying the target person.
Step 1:
Multiple-choice Questions provide the witness with set of options choose from. The alternatives are carefully crafted to include subtle differences between features, such as dark brown vs. deep chestnut color. This format simpli- fies the witness's ' decision- process while emphasiz- ing specific and nuanced distinctions.
Finally, we construct the Interactive- PEDDES dataset with an average of dialogue rounds per image. The training set comprises 47,376 images corresponding 1 11,543 identities, while the test set includes 7,373 images representing 1,508 identities. Additional details are provided in Appendix A.
Step 2: Sub-Caption Decomposition This step divides fine-grained descriptions into sub-captions to facilitate the construction of question-answer pairs in Step 3. Two key principles guide the creation of sub-captions: i) they must address specific details not mentioned in the initial descrip- tion, and ii) each sub-caption should focus on a single and distinct aspect of the person. To achieve this, we first remove the information already provided in the initial description to generate a follow-up description that emphasizes the unex- plored details. Next, we decompose the follow-up descrip- tion into non-overlapping sub sub-captions using the prompt in Table 7. Each sub-caption represents a unique aspect of the person's s features and ensures that no redundant questions will be asked.
4. LLeVA-ReID
In this section, we introduce our interactive person re- identification framework in Section 4.1 and provide a de- tailed explanation of our selective multi-image questioner (LLaVA-RelD) in Section 4.2.
4.1. Interactive ReID Framework
The interactive person re-identification framework con- sists of three main components: Retriever, Questioner (LLaVA-ReID, acting as Holmes's 's assistant), and an An- swerer. Retriever identifies the target person by retrieving the person who best matches the descriptions from the im- age gallery. The Questioner generates discriminative and context-aware questions to clicit additional details about the target person. The Answerer simulates the role of the witness by responding to the questions with relevant infor- mation. This framework leverages iterative and interactive refinement of the description to enhance the precision of person re-identification.
Step 3: Dialogue Generation To simulate real-world inquiry scenarios, we design diverse fine-grained ques- tions, categorized into three types: descriptive questions (50%), yes/no questions (40%). and multiple-choiee ques- tions (10%). The questions are carefully crafted to extract detailed information from the sub-captions generated in Step 2, ensuring both diversity and depth in the dialogue. The corresponding prompt for generating each question Q and its answer A is provided in Table and the order of Q&A pairs within the dialogue D, is randomly chosen.
Retriever is a dual-stream network (Radford et al., 2021) that encodes the dialogue descriptions and person images into shared cross-modal space using pre-trained vi- sual encoder di and textual encoder dh. Specifically, the visual encoder d. first encodes the image gallery g = {11 J2,... : Im into embeddings J2,.......
Descriptive Questions encourage the witness to describe details in their own words, promoting a richer and more natural response. For example, a question like "Can you tell me about the style of the young man's 's coat?" allows the
4

---

LLaVA-RRell RelDD: Selective Multi-image Questioner for Interactive Person Re-Identification
Image
Images/Texx feature
Frozen/Trainable Module
Selector
Retriever
Top-k Person
Description
Image Encoder
Gallery Person
Top-k Person
Target Person
Selector Encoder & O
"A mon is wearing gray coat."
* Encoder d1
Text
Description
Questioner
Selector y
Description
Selector Head sha O
T
A1
Dialogue History
Candidates
0
Update
Instruction
Q
A
2
LLavA
LoRA
Witness
Top-k Person
Candidates
Figure 3. (Left) The framework of interactive person re-identification. The Retriever encodes gallery images and the description, providing retrieval results and the relevant candidates the Questioner. The Questioner generates discriminative questions based the description and the candidates. The Witness provides the corresponding information in response to these questions. (Right) The architecture of the selector. The selector chooses most representative candidates from the top-k person based on textual information.
in advance. descriptions at the t-th round into an embedding 2, AI, At)). The matching probability of the
The textual encoder Φ: encodes the context
4.2. Selective Multi-image Questioner
In this work, we develop a Questioner capable of generating increasingly refined and discriminative questions by leverag- ing both visual and textual contexts. Specifically, given the candidate and the dialogue history from round / the Questioner generates the question Qe with the maximum likelihood which is given by:
th person is computed as:
expsim((z)) fi) (((D..) Σn exptim(2t:ff)
p(QuIC- D.-1) II Cl Di-).
where sim(..-) denotes the cosine similarity. The retrieval result I* is the image with the highest similarity, defined as 1* = arg max; (.)..
where qi is the i-th token of the generated question Q, To improve efficiency and relevance, the Questioner processes concatenated input comprising candidate individuals, dia- logue history, and instruction for question generation:
Questioner a large multimodal model (LMM) that gener- ates specific questions Qe based the currently collected information, including the dialogue D,-1 and the candidate set C--1 retrieved from the person gallery. The candidate set should include representative individuals and encourage Questioner to effectively generate refined and contextually relevant questions.
[Images] [Dialogue] [Instruction],
where the instruction is randomly sampled from Table 9,
Notably, effective question generation involves addressing two fundamental challenges:
Answerer a large language model (LLM) that simulates the role of the witness. receives questions and provides answers based on its memory (i.e., the fine- gre-igrinedd descrip- tions F). An LLM is employed instead of an LMM since, in real-world scenarios, witnesses rely on recollections F rather than directly viewing images of the target person. This aligns with the LLM's strength in processing and gen- erating detailed based information. The prompt An- swerer is placed in Table 10.
i) Image Selection: Which images should be chosen as context to help the Questioner identify specific attributes and generate targeted questions?
ii) Question Selection: How to determine the optimal se- quence of questions from the dataset to maximize informa- tion gain and uncover critical details overlooked in previous dialogue rounds?
5

---

RelVA-ReeDD: Selective Multi-image Questioner for Interactive Person Re-Identification
To address these challenges, we propose LLaVA-RelD, selective multi-image Questioner that integrates Selective Visual Context and Booking-Forward Supervision to effec- tively tackle both challenges.
To enable the selector to explore more possible combi- nations during the training stage, we employ differen- tiable random sample strategy, namely, Gumbel-top-k relax- ations (Xie & Ermon, 2019). Specifically, given k persons with weights w, the probabilities of sampling a sequence Sundered of e candidates without replacement are defined as follows:
4.2.1. SELECTIVE VISUAL CONTEXT
The differences between candidate persons offer valuable insights for distinguishing the target person and are essen- tial for enabling the Questioner generate effective and discriminative questions. However, existing interactive re- trieval methods often neglect this aspect, typically relying on t selection (Liang & Albanie, 2023; Madasu et al., 2022) or k-means clustering (Lee et al., 2024) of candi- dates. Specifically, top-A selection can limit the diversity of uncertain candidates in the gallery, reducing the range of potential questions the Questioner can generate while k-means clustering may fail to preserve fine-grained dis- tinctions between candidates. Additionally, existing large multi-moda models struggle to process extensive image collections due to limited context understanding and com- putational constraints (Wu et al., 2024). Applying token reduction methods might damage the fine-grained features.
W
Wi2 7 Wi Z
W
p(Sonereal (w)
7
Wij
where Z Σ3 W; is the normalizing constant and the subscript i1 denotes the first selected candidate in the subset C. The expected probability of sampling C is the sum over all permutations of candidates in C:
C C~p(Clw) = Σ
(8)
where II(.) denotes the set of all permutations. To build differentiable subset, we perturb the weights w by adding Gumbel noise, i.e., w, = w, where 9i ~ Gumbel(O, 1), and apply the re-parameterization trick to ensure the gra- dient flow is preserved throughout the sampling process. The selection model supervised using the loss function derived from the mooking-forward supervision introduced in the following section.
To address the above challenges, we propose a multi-image Questioner built on the LMM framework LLaVA VA (Li et al., 2024b), incorporating novel hard-pass selection model preceding its vision encoder. The selection model identifies a subset of representative candidates from the gallery based on their relevance to the dialogue context, enabling the Questioner to focus on the most informative visual inputs. The structure of our selector is illustrated in Figure 3.
4.2.2. SELECTIVE SUPERVISION BY LOOKING-
Although the constructed Q&A pairs in our interactive dataset enteractive-EEDDs can serve as supervision, they are inherently an unordered set. Different questions yield varying levels of information gain depending on the current retrieval results. For example, if most candidate person are wearing dark pants, asking about the color of the pants might provide little discriminatory power, even though pants are an important attribute for identification. Conversely, if the candidates differ significantly in terms of accessories, asking about accessories could significantly narrow down the candidates. Thus, determining the optimal sequence of questions is critical to maximizing information gain.
Specifically, we first retrieve the top k most relevant person from the gallery given the current dialogue context D, using the Retriever model,
T= U4 i top-k(sim(=r, f:))}.
(4)
Next, we feed the image embeddings and the dialogue em- beddings into shallow Transformer encoder os. enabling deep interaction between visual and textual modalities:
A straightforward approach to this problem is to enumerate all possible permutations of question sequences and evalu- ate their impact exhaustively. However, the complexity of this brute-force strategy grows factorially with the numher of rounds, i.e., Ott),, making it computationally infeasi- ble. To address this, we propose a one-step looking-forward strategy that dynamically evaluates the information gain of each candidate question in the current round. This ap- proach selects the most informative question based on its impact on the retrieval rank of the target person, avoiding the combinatorial explosion of full permutations.
(5)
where fa = [filT E T denotes the embeddings of the candidate set T. Here, represents the conditional selection embeddings of each candidate person based on the textual dialogue. We treat the candidates as an unordered set and do not apply positional embeddings.
We then pass v through a linear layer Φn to predict the final selection weight for each candidate person, i.e., w Softmax (Φr (v)). The final candidates sent to the LMM the top-c candidates with the highest weights:
Specifically, let S be the questions prepared in our dataset, and Ot-1 be the questions asked in the previous f 1 rounds,
C = ( (Teli E top-c(w)).
6

---

LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re-Identification
Table
Comparison state-of-the-art interactive retrieval methods on Interactive-PEDEs. "Initial" denotes using the initial description
without interaction. Our method is marked in gray
A lower BRI indicates better performance.
Round 3
Round
Method
BRI +
R@
R@5
@ 10
mAp
@
@@5
R @ 10
mAP
Initial
35.86
55.17
64.57
32.80
35.86
55.17
64.57
32.80
SimIRV (Liang & Albanie, 2023)
50.45
73.39
82.49
41.00
61.27
82.00
88.36
46.53
1.024
ChatlR (Levy et al., 2023)
57.85
78.69
86.08
44.92
63.86
83.81
89.84
48.05
0.935
PlugIR (Lee et al., 2024)
60.34
81.34
87.89
47.28
65,44
85,33
91.01
49.89
0.849
LLaVA-RelD (Ours)
63.96
84.70
91.27
48.45
73.20
90.62
95.95
53.31
0.719
with 2pre 0 initially. At round i, we iterate through all candidate questions and append the corresponding answer to the description. We evaluate the impact of each question based on the retrieval rank of the target person. The ques- tion that yields the highest retrieval rank is selected as the supervision for the current round. Formally, the optimal question is determined as:
90
80
70
Revall
60
50
Qi = arg max rank (Lg,, ( Q,E(S) 2
40
SimIRV
PlugIR
(9)
30
ChatlR
LLa LL-VAD
where rank represents the retrieval rank of the target person fot based on the context with the lorkinggfoowardd descrip- tion. The Questioner is then optimized using the negative log-likelihood (NLL) loss:
Round
Figure 4. Retrieval performance v.s. Number of queries. The solid line denotes the R@ and the dashed line denotes the R@5.
NLL log (Qi Ct-1, Dr-1).
(10)
responses. We select 4 candidate images using our selector and set the maximum number of interaction rounds to 5, Module structures, training details, and evaluation metrics are provided in Appendix C.
This strategy ensures that the supervision dynamically adapts to retrieval results, prioritizing questions that pro- vide the most information gain.
5. Experiments
5.2. Interactive Person Re-Identification
In this section, we first introduce the details of the implemen- tation of our approach. We then evaluate its performance on Inter-ReID, comparing it with interactive cross-modal retrieval methods. Next, we integrate our method into exist- ing T-RelD frameworks to demonstrate its transferability in enhancing standard text-based retrieval. Finally, we conduct analysis studies and present qualitative results for further evaluation.
We compared our method against three state-of-the-art in- teractive cross-modal retrieval baselines: i) SimIRV (Liang & Albanie, 2023) is heuristic method that extracts objects from the initial description and then generates detailed ques- tions about these objects. ii) ChatIR (Levy al., 2023) instructs pre-trained LLMs using multiple dialogue exam- ples to guide the model to predict the next question. iii) PlugTR (Leee et al., 2024) selects candidates using k-means clustering and generates the next question based on the textual descriptions of these images. A filtering process iter- atively removes redundant questions until suitable one is selected, making this approach computationally inefficient.
5.1. Implementation Details
For the Retriever, we adopt CLIP (Radford al., 2021) as the backbone and train it using the IRRA (Jiang & Ye, 2023) framework with fine- ihe-grained descriptions from Interactive- PEDES. For the Questioner, we build our model on LLaVA- One Mision-Qwen2--e using QLoRA (Dettmers et al., 2023). For the Answerer, we use - (Team, 2024) to emulate witness
As shown in Table 1, LLLVA-RRID achieves superior per- formance compared to basclines. In particular, it improves R@I I by 28.1 and 37.34% after and 5 rounds of inter- action (compared to Initial), respectively. Notably, LLaVA- RelD significantly outperforms the recent state-of-ther
(Li et al., 2024a) and fine-tum tune it

---

LLaVA-RelD; Selective Multi-image Questioner for Interactive Person Re-Identification
Table 2. Performance comparison of integration with existing T-Rell methods on three henchmarks.
CUHK-PEDES R@5
ICFG-PEDES
RSTPReid R@5
Method
R@1
R@ 10
mAP
R@1
R@5
R@ 10
mAP
R@1
R@ 10 mAP
CFine (Yan et al., 2023)
69.57
85.93
91.15
60.83
76.55
82.42
50.55
72.50
81.60
RaSa (Bai et al., 2023)
76.51
76.51
94.25
69.38
65.28
80.40
85.12
41.29
66.90
86.50
91.35
52.31
APTM (Yang et al., 2023)
76.53
90.04
94.15
66.91
68.51
82.99
87.56
41.22
67.50
85.70
91.45
52.56
AUL (Li et al., 2024c)
77.23
90.43
94.41
69.16
83.32
88.37
71.65
87.55
92.05
IRRA (Jiang & Ye, 2023)
73.38
89.93
93.71
66.13
63.46
80.25
85.82
38.06
60.20
81.30
88.20
47.17
+ ALLaVA-ReID
78.51
92.43
95.67
70.61
67.44
82.91
87.69
40.60
69.85
88.10
92.55
54.92
RDE (Qin et al., 2024)
75.94
90.14
94.12
67.56
67.68
82.47
87.36
40.06
65.35
83.95
89.90
50.88
+ +LLAVA-ReID
79.39
93.02
95.96
71.70
69.98
84.32
88.60
42. 12
72.00
88.95
93.55
56.06
method PlugIR, achieving improvements of 7.76% in @@ Moreover, our method attains the lowest BRI (Lee et al., 2024), which measures the interaction efficiency and rank-
Table 3. Analysis of image and question selection strategy.
Settings
R@
R@s
R @@l 10 BRI J
ing improvement. This demonstrates the effectiveness of our method in refining retrieval results with fewer interactions. To further analyze retrieval performance across interaction rounds, we place the visualization results in Figure 4. While all methods benefit from iterative questioning, our approach exhibits the most significant performance gains, highlight- ing its ability to refine retrieval through more effective and targeted inquiries.
k-means
72.51
90.08
94.59
0.727
uniform
72.70
90.06
94.60
0.735
Candidate
top-k
72.53
89.78
94.22
0.740
selector
73.20
90.62
95.95
0.719
random
72.25
89.69
94.48
0.813
Question
look-forward
73.20
90.62
95.95
0.719
5.3. Integration with Text-based ReID
5.4. Analysis of Image and Question Selection
We further integrate LLaVA-ReID into existing T-ReID frameworks and evaluate its transferability on T-ReID bench- marks, including CUHK-PEDES (Li al., 2017), ICFG- PEDES (Ding et al., 2021) and RSTPReid (Zhu et al., 2021). Specifically, we use the dataset annotations as the initial descriptions and conduct a 5-round interaction to refine the retrieval process. After the interaction, we encode the initial description using the existing T-ReID model, process the dialogue with our Retriever, and re-rank the matching scores by averaging the similarity. In this setup, T-RelD models capture the static initial descriptions, while our approach extracts additional details through interactive refinement.
We conduct analysis study to evaluate the impact of dif- ferent selection strategies for candidate images and target questions. For candidate selection, we compare our pro- posed selector with top-k (Liang & Albanie, 2023; Madasu et al., 2022), k-means (Lee et al., 2024), and uniform sam- pling. Top-k selects the k images with the highest similarity scores. k-means clusters the gallery and selects the closest sample from each centroid as representative candidates. Uni- form sampling randomly selects images candidates. For target question selection, we compare our forrirg-forward strategy with random sampling to assess the effectiveness of question selection refining retrieval. As shown in Table 3, our selector outperforms the widely used strategies and highlights the benefits of selecting representative candidates. Meanwhile, our looking- forward strategy demonstrates its advantage in dynamically selecting informative questions.
As shown in Table 2, integrating our method into IRRA (Jiang & Ye, 2023) and RDE (Qin et al., 2024), where 5-round interaction consistently enhances perfor- mance across different methods and benchmarks, achiev- ing an average @ improvement of 5.19%. Notably, our dataset with
significant improvements on ICFG-PEDRE annotatdd fine-grained descriptions, reveal that par- tial and vague descriptions remain inevitable, even when annotators are instructed to provide as much detail as possi- ble. This highlights the necessity of interactive refinement to achieve more accurate and reliable ReID in real-world scenarios.
5.5. Qualitative Results
We visualize several retrieval processes in Figures' 7 and 8. As shown, VA-RRD effectively focuses on fine-grained attributes such as shoes, hair, surroundings, and even makes assumptions about logos and accessories to refine the re- trieval process. Specifically. LLaVA-ReID captures key distinguishing features among candidates, such as the dis-
8

---

LLaVA-ReeD:: Selective Multi-image Questioner for Interactive Person Re-Identification
tinctive logo on the jacket (round 4 in Figure 7) and the length of the shirt (round 3 in Figure 8), further validating the effectiveness of our selection. These examples demon- strate that LLaVA-ReID can effectively identify and em- phasize the fine-grained attributes crucial for differentiating individuals in real-world scenarios.
Cai, G., Zhang, J., Jiang, X., Gong. Y., He, L., Yu, F., Peng. P., Guo, X., Huang, F., and Sun, X. Ask&confirm: Ac- tive detail enriching for cross-modal retrieval with partial query. In 2021 IEEE/CVF International Conference on Computer Vision ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 1815-1824. IEEE, 2021. doi: 10.1 A11100/CC849922200000 100155.
6. Conclusion
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized Ilms. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M.. and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.
This paper introduces Interactive Person Re- Re-leentification new problem that more closely aligns with real-world sce- narios by incorporating multi-round dialogue between the witness and the retrieval system. To support this task, we propose a specific dataset Interactive-PEDD PPDDES and propose a multi-image questioner LLaVA-Relll that leverages both vi- sual and textual contexts identify fine-grained differences. Despite achieving strong performance, challenges remain, such as the limited ability of the retriever to handle incre- mental and dialogue-basee descriptions. Future research could further explore these directions.
Ding, G., Lu, Y., Hu, P., Yang, M., Lin, Y., and Peng, X.
Visual abstraction: A plug-and-play approach for text- visual retrieval. In International conference on machine learning. PmLR, 2025.
Ding, Z., Ding, C., Shao, Z., and Tao, D. Semantically self- self-aligned network for text-to- rext-to-image part-aware person re-identification. CORR, abs/2107. 22211007766600 12666, 2021.
Impact Statement
The proposed interactive person RelI) framework has broad applications in public spaces such as airports, shopping centers, and transit hubs, aiding in locating missing children or elderly individuals. Additionally, this technology can assist law enforcement identifying and tracking suspects, contributing to enhanced public security.
Gao, C., Cai, G., Jiang, X., Zheng, F., Zhang, J., Gong, Y., Peng, P., Guo, X., and Sun, X. Contextual non-local alignment over full-scale representation for text-based person search. CoRR, abs/2101.00 0030666 2021.
Gong, Y., Huang. L., and Chen, L. Person re- iddenifiiiatiin method based color attack and joint defence. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4313 4322, 2022.
However, the deployment of such systems raises critical ethi- cal concerns, particularly regarding privacy and surveillance. Ensuring responsible use requires stringent safeguards, in- cluding informed consent, data protection measures, and transparent governance.
Gong, Y., Zhong, Z., Qu Y., Luo, Z., Ji, R., and Jiang, M. Cross-modality perturbation synergy attack for person re-identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http::///ppeeeeeeeww net/ forum? id=1oNd7ACE
Acknowledgments
This work was supported in part by NSFC under Grant 62176171, 624B2099, 62472295, U24B20174; in part by the Fundamental Research Funds for the Central Universi- ties under Grant CJ202303, CJ202403; in part by Sichuan Science and Technology Planning Project under Grant 24NSFTD0130; and in part by Baidu Scholarship.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21270 2024.
Jiang, D. and Ye, M. Cross-modal implicit relation rea- soning and aligning for text-to-image person retrieval. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 2787-2797. IEEE, 2023, doi: 10.1109/CVPR552229920 000.2227722222.02277777
References
Bai, Y., Cao, M., Gao, D., Cao, Z., Chen, C., Fan, Z., Nie, L., and Zhang, M. Rasa: Relation and sensitivity aware representation learning for bext- person search. In Proceedings of the Thirtt-Second International Joint Con- ference Artificial Intelligence, IJCAI 2023, 15th-25th August 2023, Macao, SAR, China, pp. 555-563. ijcai.org, 2023. doi: (0.24963/UJCAA.202234
Kovashka, A., Parikh, D., and Grauman, K. Whittlesearch: Interactive image search with relative attribute feedback. Int. J. Comput. Vis., 115(2):185- 2015. doi: 10.1007/
9

---

-LaVA-ReeDD: Selective Multi-image Questioner for Interactive Person Re-Identification
Lee, S., Kim, D., and Han, B. Cosmo:
Content-style mod- ulation for image retrieval with text feedback. In IEEE Conference Computer Vision and Pattern Recogni- tion, CVPR 2021, virtual, June 19-25. 2021, pp. 802- 812. Computer Vision Foundation / IEEE, 2021. doi: 10.1 N0.1009CCPRR4443.2222....
Vancouver, Canada, pp. 3172-3180. AAAI Press, 2024c. doi: 10.1609/AAAI 23814.28101.
Liang, K. and Albanie, S. Simple baselines for inter- active video retrieval with questions and answers. In IEEE/CVF International Conference on Computer Vi- sion, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11057-11067. IEEE, 2023. doi: 10.1109//CCV510000. 2023.01018.
Lee, S., Yu, S., Park, J., Yi, J., and Yoon, S. Interactive text- to-image retrieval with large language models: A plug- and-play approach. In Ku, I.,, Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume 1: Long Papers), AC1. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 791-809. Association for Computational Linguistics, 2024. doi: 10. 10.16653W1/202444CLLLLLM 46.
Lin, Y., Zhang. J., Huang, Z., Liu, J., Peng, X., et al. Multi- granularity correspondence learning from long-term noisy videos. In The Twelfth International Conference on Learn- ing Representations, 2023.
Madasu, A., Oliva, J., and Bertasius, G. Learning to retrieve videos by asking questions. In Magalhães, J., Bimbo, A. D., Satoh, S., Sebe, N., Alameda-
X., Jin, Q., Oria, V., and Toni, L.. (eds.), MM '22: The 30th ACM In- ternational Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, pp. 356-365, ACM, 2022. doi:
Levy, M., Ben-Ari, R., Darshan, N., and Lischinski, D. Chatting makes perfect: Chat-based image retrieval. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, and and Levine, S. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Qin, Y., Chen, Y., Peng, D., Peng, X., Zhou, J. T., and Hu, P. koisy-correspoodent learning for text-to-image person re-identification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 27187-27196. IEEE, 2024. doi: 10.1 0..10099CVPP3273.222000222
Li, B., Zhang, Y., Guo, D.. Zhang, R., Li. F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.033266 2024a. doi: 10.48550/ARXIV..0 033226.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748- 8763. PMLR, 2021.
Li, B., Zhang, Y., Guo, D.. Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03322 033226 2024b.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping Linguage-image pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning. pp. 12900. PMLR, 2022.
Shao, Z., Zhang, X., Fang, M., Lin, Z., Wang, J., and Ding, C. Learning granularity- representations for text-to-image person re- re-identification. In Magalhães, J., Bimbo, A. D., Satoh, S., Sebe, N., Alameda-Pinenn X., Jin, Q., Oria, V., and Toni, L. (eds.), MM '22: The 30th ACM International Conference on Multimedia, Lis- boa, Portugal, October 10 14, 2022, pp. 5566 6574. ACM, 2022. doi: 10.1 10.1145/3503161 06000311611100 111.500366.3548228..
Li, S., Xiao, T. Li, H., Zhou, B., Yue, D., and Wang, X. Person search with natural language description. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5187-5196. IEEE Computer Society, 2017. doi: 10.1109/CVPR..007..555..
Li, S., He, C., Xu, X., Shen, F., Yang, Y., and Shen, H. T. Adaptive uncertainty-basel learning for text-based person retrieval. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), Thirty-Fighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on In- novative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2014, February 20-27, 2024,
Shi, J., Yin, X., Chen, Y., Zhang. Y., Zhang, Z., Xie, Y., and Qu, Y. Multi-memory matching for unsupervised visible- infrared person re-identification. In European Conference on Computer Vision, pp. 456-474. Springer. 2024.
Tan, W., Ding, C., Jiang, J., Wang, F., Zhan, Y., and Tao, D. Harnessing the power of mllms for transferable text- to-image person reid. In JEEE/CVF Conference on Com-
10

---

LLaVA- RelID: Selective Multi-image Questioner for Interactive Person Re-Identification
puler Vision and Pattern Recognition, CVPR 2024, Seat- fle, WA, USA, June 16-22, 2024, pp. 17127-17137. IEEE, 2024. doi: 10.1 00.11999CVPP322332200..000
Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L.., and Hoi, S. C. Deep learning for person re-identification: A survey and outlook. IEEE transactions pattern analysis and machine intelligence, 44(6):2872228) 2893, 2021.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https:/ //tp:////w/mmmmmmmmm git.giithhh.... /o/ blog/ hlgg//een.54/
Zhu, A., Wang, Z., Li, Y., Wan, X., Jin, J., Wang, T., Hu, F., and Hua, G. DSSL: deep surroundings-perseee separation learning for text-based person retrieval. In Shen, H. T., Zhuang, Y., Smith, J. R., Yang, Y., César, P., Metze, F., and Prabhakaran, B. (eds.), MM '21: ACM Multimedia Conference, Virtual Event, China, Oc- tober 20 24, 2021, pp. 209-217. ACM, 2021. doi: 10.1145/3474085.3/4
Wu, T., Biamby, G., Quenum, J., Gupta, R., Gonzalez, E., Darrell, T., and Chan, D. M. Visual haystacks: An- swering harder questions about sets of images. CoRR, abs/2407.13766, 2024. doi: 10.48550/ARXIV2400. 13766.
Zhu, H., Huang, J., Rudinac, S., and Kanoulas, E. En- hancing interactive image retrieval with query rewriting using large language models and vision language mod- els. In Gurrin, C., Kongkachandra, R., Schoeffmann, K., DanggNguyen, D., Rossetto, L.., Satoh, S., and Zhou, L. (eds.), Proceedings of the 2024 International Con- ference on Multimedia Retrieval, ICMR 2024, Phuket, Thailand, June 10-14, 2024, pp. 978-987. ACM, 2024. doi: 10. 1445365555833665833333
Wu, Y., Yan, Z., Han, X., Li, G., Zou, C., and Cui, S. Lapscore: Language-guidee person search via color rea- soning. In 202/ IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 1604-1613. IEEE, 2021. doi: 10. 10.11000CCC4499222200000000
Xie, S. M. and Ermon, S. Reparameterizable subset sam- pling via continuous relaxations. In Kraus, S. (ed.), Pro- ceedings of the Twenty-Eighth Eighth International Joint Confer- ence on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 9929.3925. ijcai.org, 2019. doi: 10. (0.24963/UCAA.2219/5544
Zuo, J., Yu, C., Sang, N., and Gao, C. PLIP: language- image pre-training for person representation learning. CoRR, abs/2305.08386. 2023. doi: 10.48550/ARXIV. 2305.08386.
Yan, S., Dong, N., Zhang, L., and Tang. J. Clip-driven fine- grained text-image person re-identification. IEEE Trans. Image Process., 32:6032-6046, 2023. doi: 10.1 109/TTP. 2023.3327924.
Zuo, J., Zhou, H., Nie, Y., Zhang, F., Guo, T., Sang, N., Wang, Y., and Gao, C. Ufinebenc:: Towards text-based person retrieval with ultra-fine granularity. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 22-22, 2024, pp. 22019-22019. IEEE, 2024. doi: 10.1 1V/100/72..2 2024.02078.
Yang, M., Huang, 7.,, Hu, P., L., Lv, J., and Peng, X. Learning with twin noisy labels for visible-infrared per- son re-identification. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 14308-14317, 2022.
Yang, S., Zhou, Y., Zheng, Z., Wang, Y., Zhu, L., and Wu, Y. Towards unified text-based person retrieval: A large-scale multi-attribute and language search benchmark. In El- Saddik, A., Mei, T., Cucchiara, R., Bertini, M., Vallejo, D. P. T., Atrey, P. K., and Hossain, M. S. (eds.), Proceedings of the st ACM International Conference on Multimedia, MM 2023, Ottawa, ON, Canada, 29 October 2023- 3 November 2023, pp. 4492-4501 ACM, 2023. doi: 10. 1145/3581783.885 709.
Ye, M., Shen, J., J. Crandall, D., Shao, L., and Luo, J. Dynamic dual-attentive aggregation learning for visible- infrared person re-identification. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part XVII 16, pp. 229- 247. Springer, 2020.

---

LLaVA-RelD: Selective Multi-image Questioner for Interactive Person Re- Re-Identification
Appendix Overview
This supplementary document is organized as follows:
Sec. A presents the details of our Inter-RelD dataset, Interactive-PEDEE.
Sec. B outlines the instructions for the Questioner and Answerer components in our interactive RelD framework.
Sec. C provides the experimental details, including implementation specifics and evaluation metrics.
A. Interactive-PEDES Details
In this section, we present additional details about our Interactive-PEDES dataset and provide the prompts used for its automatic generation.
A.1. Details of the Dataset
In this section, we provide statistics and visualizations for our dataset and compare its description length and word diversity with other existing T-ReID datasets, shown in Table 4. Specifically, Interact-PEDDE init and Interact-PEDESS fine refer to the initial and fine-grained descriptions in our dataset, respectively. Our dataset exhibits a higher level of text granularity, with average length of 135 words, compared to other datasets, enabling more detailed and nuanced descriptions. The distribution of the description length and the dialogue round are shown in Figure 5. Our dataset provides comprehensive dialogue about the person's 's features.
Additionally, we present further visualizations in Figure 6. The left panel displays word frequency distribution of the first six words in the generated questions, while the right panel illustrates word cloud of the fine-grained descriptions, where the size of each word corresponds to its frequency in the dataset.
Table 4. Statistics of texts in our dataset compared to other existing T-ReID datasets.
Dataset
Maximum Length Minimum Length
Average Length
Unique Words
CUHK-PEDE PEEK-E
112
13
26.6
7470
ICFG-PEDES
84
38.7
3102
SreeractPEDDE-img
34
6
17.2
2394
Intesict-PDDES-fne
193
75
135.4
6255
25%
30%
25%
Frequency
20%
30%
15%
5%
10%
5%
0%
[70, 80)
tw0, 90)
ts0, 100)
1100 119)
1320, 130)
1120
1140 180)
(1s0 160)
E360, 170)
(170, 180)
[180 190)
(190. 200)
120)
140)
0%
8
10
11
122
33
14
Descritiio Length
Dialogua
Figure 5. Distribution of description length (Left) and the dialogue round (Right) in our dataset.
A.2. Prompts for Automatic Dataset Construction
In this section, we present the prompts used for the automatic construction of our dataset, covering the generation of coarse- and fine-grained descriptions, sub-caption decomposition, and interactive question-answer pairs, detailed in Tables 5 8.
12

---

LLaVA-ReIlD RelD: Selective Multi-image Questioner for Interactive Person Re- Re-Identification
about
acces S T
ies
early 30s
re: Laxe
posture SuRcestirg
C
Tuntpaan buii
discinctive
posture
logo
surrourd ings
aeaaaaa holeted
look
aug
the
205
stand
inptearing
sur
visible due ehorg
d
me
aut S
s slim build
with
manc
teit
maan
access
IN aol
angle
mar
Jegn
pants
Ca y ing
han
difficult
the
shirt
celarrrrlyyyyyy Visible
stee
sirple
aetting
desiin
e
dar k ha poters
i
sshor
man
What
kind
tean
Bar
walking
be kg oundoe
late his
Young
the
Is
the
vi
aed pall
ong
dari
ro
ppat
early
ly
twenties
siee
ceand
Carries
Longth
diing
saccomi
length
Hpper
pea
Trae reluun
old
man
V
den'
wea
Tng
Lack
have
hare
appea
coloe
JacKot
redd
walk-
back
persabyy
man look?
ymug man
Stort
cily
Late
teens
whise
aw
sneake
detaill
short
upp
walking
style
oa
Nime
Fand
me
lock?
Light Co
slender
bui
colore
ld
backgack
ed
Vis i
logo
street
short dark
urban environment
Figure (Left) Word frequency statistics question in our dataset. The word order sentences starts at center and extends outward. (Right) Word cloud of fine-grained descriptions. A larger area or font size indicates higher frequency of occurrence.
Table 5. The list of instructions for coarse-grained description.
Prompts for Coarse-grained Description:
Describe the person's 's clothing in one sentence.
Generate a brief caption about the clothing in one sentence.
Generate a brief caption of this person in one sentence. Begin with "I saw a".
Table 6. The instruction for fine-graines description. The <init.query? will be replaced by generated coarse soase--rriined description.
Prompt for Fine-grained Description:
Describe the person in the image using definitive statements. Focus on identifying and describing the following
entities and attributes:
Approximate age range.
Color, type, patterns, and length of their upper and lower clothing, as well as any distinctive logos or details
that stand out.
-Color and type of shoes other footwear.
Any items they are wearing, carrying, holding, along with their colors, types, and patterns.
Color, length, and style of hair.
Build and posture.
Surroundings.
Provide the most direct and lifelike description this person in a paragraph, as if you are watching this person in the scenario. Focus on what is visible and what the person doesn't have that might aid identifying them.
Avoid using phrases like "The image", or "The picture". Use he, she, or descriptive terms (man, woman, boy, girl, lady) as subject rather than "The person""
Begin with "<init query>", and provide more detailed information.
13

---

LLaVA-ReID: Selective Multi-image Questioner for Interactive Person Re- Re-Identification
Table 7. The instruction for sub-caption decomposition.
Prompt for Follow-up Description Splitting:
Given a person's 's description, summarize the content of the description into individual sentences. Each sentence should capture one aspect of the description, ensuring that no information lost in the splitting process.
List the sentence as follows:
1. [first sentence]
2. [second sentence]
3.
Here an example:
Description: A young woman, likely in her late teens to early twenties, is walking with a relaxed posture. She wearing a dark blue puffer jacket that appears to zippered with logo on the back, providing warmth and comfort. Undemmatth, the specifics of her lower clothing are not visible, but she has on dark jeans that fit snugly. Her footwear consists of bright blue athletic shoes, which stand out against the darker tones of his clothing. She carries large, dark-colored backpack, which seems practical for daily use. She had long, dark brown hair that fell past her shoulders, styled straight. The setting is indoors, possibly in lobby entrance area, with glimpse of outdoor activity visible through the glass, indicating a busy environment. Sentences:
A young woman is in his late teens to early twenties.
She walking with a relaxed posture.
She wearing dark blue puffer jacket with a logo on the back, that appears to be zippered.
4. Underneath the jacket, the specifics of her lower clothing is not visible.
5. She has dark jeans that fit snugly.
6. Her footwear consists of bright blue athletic shoes.
She carries large, dark-colored backpack.
She had long, dark brown hair that fell past her shoulders, styled straight.
The setting indoors, possibly in lobby or entrance area, with a glimpse of outdoor activity visible through the glass.
Now split the following description: Description: <description> Sentences:
14

---

-LaVA-ReelD: Selective Multi-image Questioner for Interactive Person Re-Identification
Table The instruction for dialogue generation, where <sub ject> will be replaced by "man" "woman" based the gender of the
person and <question. <queesiion...yee.roomp is sampled from the question prompts below.
Base Prompt for Q&A Pair Formulating:
You given [sentence] describing visible feature of an image of <sub ject>. This may include details about the <sub ject>'s appearance, location, or surroundings. Your task is to generate a question about a feature of the <subject>> described the [sentence] (or details about the location or surroundings) that could help identify this person in a large collection of images.
The question should sound natural, as if you are asking a witness, without implying that you already know the answer. Avoid phrases like "in this image" or "How would you describe" and focus on describing visual features. Use appropriate pronouns (he, she) descriptive terms (e-g-, young man, elderly woman) when referring to the subject. Then, based the [sentence], provide direct and complete answer. <question innastiootttoo pypmmpttppptt>>>>>> [sentence]: <sub.caption>
Question Prompts for Each Type:
Yes/No question with positive answer:
The question should be a yes or no question. Format your response as follows: Question: [question] Answer: Yes, [answer]
Yes/No question with a negative answer:
The question should be yes or question that assumes something incorrect about the person. Then provide corrective answer based on the original sentence. Format your response follows: Question: [question]
Answer: No, [answer]
Descriptive question:
The question should be descriptive question that asks for the feature the sentence describing. Format your response as follows:
Question: [question]
Answer: [answer]
Deseriptive question (Wh- form):
The question should be Wh- question seeking specific piece of information that described in the [sentence]. Format your response follows: Question: [question]
Answer: [answer]
Multiple-choice question:
The question should be a multiple-choice question with four options including:
the correct answer,
three possible options that perturb the type, length, and color. These altered options should not be easily distinguishable from the correct answer.
The answer should be complete sentence, rather than only an option. Format your response as follows: Question: [question]
A) [option A]
B) [option B]
C) [option C]
Joption D]
Answer: [answer]
15

---

LLaVA-RelD: Selective Multi-image Questioner for Interactive Person Re-Identification
B. The instructions of Interactive RelD Framework
In this section, we provide the prompts used for the Questioner and Answerer, as detailed Tables and 10.
Table 9, The instruction for training L (Questioner).
Instructions:
<aadddaatee>><aaalaop
Based on this information:
[Candidate Person] The closest match to the provided description and answers. [Similar Persons] Individuals who also match but are less similar. [Conversation] The initial description from the witness and the ongoing dialogue. Generate question that could help differentiate the candidate from similar individuals. Avoid repeating questions that have already been answered with "I don't know.".
<candi dates><<>> log>
Based on this information:
[Candidate Person] The individual most likely matches the description provided.
[Similar Persons] Individuals with similar characteristics, but they are less likely to be the candidate. [Conversation] The dialogue between the witness and investigator, including all available descriptions. Generate a question that explores a specific detail or feature that differentiates the candidate from others. Ensure the question has not already been answered or ruled out with an "I don't 't know." response.
<candi dates> <dd a log>
The following context applies:
[Candidate Person] The individual who best fits the description so
[Similar Persons] Individuals who have overlapping characteristics but are less likely be the target. [Conversation] The ongoing witness description and responses to questions. Generate a question that focuses on details that could clearly identify the candidate, while avoiding any redundant questions that have already been answered with "I don't 't know.".
Image Tokens Placeholder:
<candi dendidates>>> "[Candidate Person] <i <images> (Similar Persons]"
\n
for i in
L date>>=4"t <images>\n"
Dialog Context:
"aaaloos="(Converstttiono Witness: <init.query>"
for question, answer in previous.dialogs
<dialog> "You: (question))n Witness: (answer)\n"
16

---

LLaVA-RellD Selective Multi-image Questioner for Interactive Person Re-Identification
Table 10. The instruction for Answerer.
Prompt for Answerer:
Imagine you are witness who has seen one person and someone is asking the information about that person. You are answering question about that person and this is what you remember: <context>
Answer the question only according to what you remember.
Instruction:
Answer directly in complete sentence that provides the information to the question, without including any extra information.
Avoid giving answers that are just 'Yes', 'No', or single word or phrase. Always answer in full sentences, even for multiple-choice questions.
3. If an entity or attribute is not included in your memory, assume the person does not possess it.
Question: <questions
Answer:
17

---

LLaVA-ReID; Selective Multi-image Questioner for Interactive Person Re-Identification
C. Experimental Details
In this section, we provide additional details on the model structures, training configurations, and evaluation metrics used in our experiments.
C.1. Model structures
Retriever. We use a pre-trained CLIP model (Radford et al., 2021), employing CLIP-ViT-B/16 as the visual encoder and -.IP-Xformer as the text encoder. The model is fine-tuned using the T-ReID method IRRA (Jiang & Ye, 2023) on the fine-grained descriptions from Interactive-PEDD
for 30 epochs with batch size of 128. All other training parameters follow the original IRRA settings. To handle longer text inputs, interpolate the text positional embeddings from 77 to 192. Once trained, the Retriever is frozen for the remaining tasks.
Questioner. It is built on ALaVA-One Oee Vision- Vision-Qwen!s Qwen2-7B-or (Li et al., 2024a) and fine-tuned using QLoRA (Dettmers et al., 2023). The model is quantized to 4-bit and LoRA weights are applied with r = 128 and a = 256. The learning rate is set to 1 x 10-5, with batch size of 4 and gradient accumulation steps of 4. To reduce the length image patch tokens, x mean pooling applied to each image, reducing the number of tokens to 49 per candidate image. The
for training Questioner are shown in Table 11, and the instruction used for question generation is provided Table 8.
Answerer. We use Swen2.5--BBInnsruue (Team, 2024) for emulating witness responses. The prompt used for the Answerer is provided in Table 10.
Selector. It is implemented using a 5-layer Transformer with latent dimension of 512, followed by a linear layer. The number of candidate images is dynamically calculated as k = | where is the current interaction round, with fixed set of = 4 candidates selected per round. Learnable modality embeddings are added to differentiate between visual and textual inputs, without using positional embeddings.
C.2. Training Details
During training, we first warm up the Questioner and selector by training them separately with top-4 images as context using NLL loss and BCE loss for and 20 epochs, respectively. After this, we combine the Questioner and the selector and train them jointly using NLL loss for 0.5 epoch. During the interaction process, limit the question length to 96 tokens and the answer length to 40 tokens. All experiments are conducted on an Ubuntu 20.04 system with NVIDIA 4090 GPUs.
Table 11. Training configuration of LLaVA-ReID
Parameter
Value
lora_alpha
256
Iora
128
Iora_ urradropout
0.05
deepspeed
zero2
bf16
True
epoch
batch size
4
learning rate
10-5
weight decay
0
warm-up ratio Iv scheduler model max length
0.02
cosine
4096
attn implementation
sdpa
bits
4
double quant
True
quant_type
nf4
18

---

LLaVA-ReID; Selective Multi-image Questioner for Interactive Person Re-Identification
C.3. Evaluation Metrics
We evaluate interactive ReID performance using Recall@k, mAP, and BRI (Lee al., 2024).
@ecall@k measures the percentage of cases where the ground-truth person appears in the retrieved top-k images
(A: = 1,5, 10).
mAP (mean Average Precision) provides overall measure of retrieval accuracy by averaging precision across all queries.
BRI (Best Log Rank Integral) assesses the system's ability to refine retrieval results over multiple interaction rounds. It measures the average area under the log rank curve of all rounds. A lower BRI value indicates a more efficient and effective refinement process.
To analyze the impact of interaction, we report Recall and mAP after 3 and 5 rounds Table to show the performance improvements over time.
Round
Initial Description:
He wearing dark puller jacket, and stylish sneakers with sporty look.
339 54
Rank: 339
eneakers?
Target Person
A: The sneakers have black base with red and while accents.
Round
Round
Rank: -
Rank:
man standing busy street with around him? A: He is walking in busy urban area with parked cars in the background.
Q: young man's jeans? A: The young jeans are medium blue.
Round
Round 5
-
1-1
Is wearing noticeable logo
walking relaxed posture?
A: The man wearing a dark puffer jacket with hood featuring distinctive pattern white stripes left sleeve.
A: Yes. the man is walking with a relaxed posture.
Figure 7. Qualitative results of the dialogue generated by our interactive system. The 4 images in each round are the representative candidates selected hy our selector.
19

---

LLaVA-RReha Selective Multi-image Questioner for Interactive Person Re- Re-Identification
Round
Initial Description:
The dressed all black, featuring fitted top and loose pants.
78 59
Rank:
Q: What is the man carrying?
Target Person
A: The carrying anything.
Round
Round
Rank:
Rank: -
Q: What can you tell me about the man's hair?
Q: the man wearing long-sloeved shirt? A: He wearing long-sleeevdd shirt.
man
Round
Round
Rank: +
Rank: 1 1 1
standing busy stree!?
Is the wearing black shoes that look like casual athletic footwear?
A: No, the surroundings minimalistic with smooth floor and signage.
A: The wearing dark shoes that appear be sturdy.
Figure 8. Qualitative results of the dialogue generated by interactive system. The 4 images in each round the representative candidates selected by our selector.
20


Reviews:
[{"review_id": "02JGMGElBm", "reviewer": "Reviewer_RhLW", "summary": "The paper introduces a new task for person re-identification, an interactive person re-identification (I-ReID) framework that iteratively refines incomplete or vague descriptions through multi-round dialogues. The paper proposed LLaVA-ReID, a selective multi-image questioner that leverages both visual and textual contexts to generate targeted questions aimed at uncovering fine-grained differences among candidate images. The work is also supported by a newly constructed dataset which includes both coarse and fine-grained descriptions along with multi-round dialogue. Extensive experiments demonstrate its effectiveness over existing methods in both interactive and traditional text-based ReID tasks.\n\n## update after rebuttal\n\nAfter rebuttal, I have no further questions. I am inclined to maintain the score as Accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We will answer your questions one by one in the following.\n\n> Q1: Can you provide further analysis or experiments to assess the computational efficiency of your candidate selection module compared to baseline methods? (As well as the question generation time)\n\nWe conduct the experiment on the computation time of different select strategies, including top-k, k-means, and our selector. The experiment is performed on NVIDIA RTX 3090 GPUs, and we report the total selection time over five rounds in the table below:\n\n| Select Strategy | Time (second per person) | R@1/R@5   |\n| --------------- | ------------------------ | --------- |\n| selector        | 0.002195                 | 73.2/90.6 |\n| k-means         | 0.028286                 | 72.5/90.1 |\n| uniform         | 0.000099                 | 72.7/90.0 |\n| top-k           | 0.000187                 | 72.5/89.8 |\n\nOur lightweight transformer-based selector achieves the highest retrieval accuracy while ensuring no significant computational overhead compared to other methods.\n\nWe also provide the inference time for 5-round question generation of LLaVA-ReID and other baseline methods are shown below:\n\n| Method         | Time (second per person) | R@1 |\n| -------------- | ------------------------ | ----------------- |\n| SimIRV         | 0.007                    | 61.3              |\n| ChatIR         | 0.216                    | 63.9              |\n| PlugIR         | 13.06                    | 65.4              |\n| **LLaVA-ReID** | **0.621**                | **73.2**          |\n\nOur multi-image questioner does not add significant computational overhead, indicating its suitability for real-time applications.\n\n> Q2: How do you define the parameter $k$ in the Gumbel-top-k relaxation within your selection strategy?\n\nThe number of candidate images plays a crucial role in providing rich visual context to the Questioner. If too few candidates are given, the Questioner may not receive sufficient fine-grained information between persons. Conversely, an excessive number of images can overwhelm the MLLM’s capacity, introducing redundancy and extra computational costs. To further analyze the impact of $k$, we present retrieval results for different numbers of candidate images in the table below:\n\n| Number of Candidates | R@1       | R@5      | R@10     | mAP       | BRI       |\n| -------------------- | --------- | -------- | -------- | --------- | --------- |\n| 2                    | 72.4      | 89.7     | 94.5     | 53.2      | 0.738     |\n| **4 (ours)**         | **73.2** | **90.6** | **96.0** | **53.3** | **0.719** |\n| 6                    | 71.4      | 89.2     | 94.2     | 52.8      | 0.764     |\n| 10                   | 71.6      | 88.9     | 93.8     | 52.7      | 0.772     |\n\nThe results indicate that setting $k=4$ provides the best balance between retrieval performance and computational efficiency.\n\nIn our previous experiments, the reported BRI was computed using a logarithm with base 2. According to the recent released code of BRI [1], we have corrected this by using the natural logarithm (base $e$). While this change slightly affects the absolute values of BRI, the relative ranking and overall conclusions remain unchanged.\n\n> Q3: In the LLaVA-ReID framework, does the conversation get embedded into the LLaVA module before the candidate images are appended?\n\nAs depicted in Equation 3, candidate images are inserted before the dialogue history, in accordance with the default configuration of LLaVA. For improved visual clarity, we modify the input order in Figure 3.\n\n> Q4: In Figure 3, could you explain the meaning of the term “clue”?\n\nThe term \"clue\" refers to the embeddings of the context description $z_t$. We use this term to indicate that the candidates are selected based on the current \"clue\" we have at hand. We apologize for any confusion caused by this terminology and will revise it in the next version.\n\n[1] Lee S, Yu S, Park J, et al. Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 791-809.", "claims_and_evidence": "The paper claims that interactive refinement via multi-round dialogues can significantly improve person re-identification accuracy compared to static, text-based approaches. These claims are verified by experimental evidence on the Interactive-PEDES dataset and integration with existing T-ReID frameworks.", "methods_and_evaluation_criteria": "The proposed method for multi-round interactive retrieval is reasonable. The evaluation criteria about retrieval metrics and interaction efficiency are appropriate for assessing the system’s performance.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "Experimental evaluations are comprehensive.", "relation_to_broader_scientific_literature": "The work is well-situated within the fields of interactive cross-modal retrieval and text-based person re-identification. It builds upon recent approaches by addressing the fine-grained differences that traditional interactive retrieval methods have not adequately considered, thereby contributing meaningfully to the literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. This manuscript presents an interactive framework for person re-identification that more closely mirrors real-world scenarios by engaging in multi-round dialogue to iteratively refine descriptions, thereby addressing a key limitation in traditional text-based ReID methods.\n2. The construction of the Interactive-PEDES dataset with its detailed coarse and fine-grained descriptions and multiple dialogue rounds, providing a robust benchmark that can drive further research in the area, filling an important gap in available data.\n3. The experimental evaluation is thorough, including integration with existing text-based ReID frameworks and detailed analysis of both image and question selection strategies.\n\nWeaknesses:\n1. While the candidate selection mechanism particularly the use of Gumbel-top-k relaxation is effective, the paper provides limited in-depth analysis of its robustness.\n2. The computational complexity associated with processing multiple images and iterative dialogue rounds may hinder real-time application.", "other_comments_or_suggestions": "Please refer to the \"Strengths And Weaknesses\"", "questions_for_authors": "1. Can you provide further analysis or experiments to assess the computational efficiency of your candidate selection module compared to baseline methods?\n2. How do you define the parameter $k$ in the Gumbel-top-k relaxation within your selection strategy?\n3. In the LLaVA-ReID framework, does the conversation get embedded into the LLaVA module before the candidate images are appended?\n4. In Figure 3, could you explain the meaning of the term “clue”?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EyCMLZmuVi", "reviewer": "Reviewer_fi2Z", "summary": "This paper introduces interactive person re-identification, which refines partial text queries via dialogue interactions to better suit real-world scenarios. The proposed LLaVA-ReID generates discriminative questions using images and dialogue history, enhanced by an image selector and a looking-forward supervision strategy. Experimental results show significant improvements over existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the novelty of our method. We will answer the questions one by one.\n\n> Q1: Details about the ensemble.\n\nIn our integration of LLaVA-ReID with the existing T-ReID framework, we use the human-annotated captions in the T-ReID datasets as the initial queries. We first encode an initial query $T$ using T-ReID model and compute the matching score between $T$ and gallery images:\n$$\n\\operatorname{score}(I_i|T)=\\operatorname{sim}(\\phi^{base}_t(T), \\phi^{base}_v(I_i)),\n$$\nwhere $\\phi_t^{base}$ and $\\phi_v^{base}$ are text and visual encoder of T-ReID model, $I_i$ is the $i$-th image in gallery, and $\\operatorname{sim}(\\cdot, \\cdot)$ denotes cosine similarity. Next, we perform 5 rounds of interaction and encode the dialogue history $D=\\{T, A_1,\\dots, A_t\\}$ using our Retriever, then compute the matching score:\n$$\n\\text{score}(I_i|D)=\\operatorname{sim}(\\phi_t(D), \\phi_v(I_i)).\n$$\nThen we re-rank the matching scores by simply averaging two scores:\n$$\n\\text{final\\\\_score}_i=\\frac{1}{2}(\\text{score}(I_i|T)+\\text{score}(I_i|D)),\n$$\nand the final score is used to compute the retrieval metrics.\n\nIt’s important to note that the T-ReID dataset only provides caption-image pairs for persons. To simulate user interactions, we turn to a multi-modal LLM. Specifically, we use LLaVA-OneVision-Qwen2-7B-ov as the Answerer, which generates follow-up descriptions based on the image of the target person.\n\n\n> Q2: Preprocessing of dialogue.\n\nFor dialogue preprocessing, we remove prefixes such as \"Yes,\" \"No,\" and multiple-choice options like \"A)\" and \"B)\" from the answers. The remaining responses are then directly concatenated and fed into the text encoder. While our current approach is straightforward, we believe that post-hoc techniques, such as dialogue rewriting, could be a promising strategy to enhance the system's ability to generalize to real-world, open-form responses.", "claims_and_evidence": "In the motivation of this paper, the authors claims that users rarely provide a detailed and comprehensive account of the target person’s appearance, often resulting in partial and vague descriptions. This claim is well-founded, as it aligns with observations in real-world applications where initial queries are often incomplete.", "methods_and_evaluation_criteria": "The method makes sense in the context of person re-id and the evaluation metrics are common.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "More analysis about visual and textual context should be provided.", "relation_to_broader_scientific_literature": "Prior work in this field largely assumes that query descriptions are complete and well-structured, which is often not the case in real-world applications. In contrast, the authors introduce an interactive cross-modal retrieval technique that explicitly accounts for the inherent misalignment between dataset queries and practical user inputs, and address the candidate selection and tailored supervision for this fine-grained question generation task.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n It introduces an interactive approach to text-based person re-id that is applicable and user-friendly in real-world scenarios. The new benchmark is thoughtfully constructed to mirror actual questioning methods, thereby enhancing the practical relevance of the work. Moreover, by integrating traditional methods, the work reveals the task's potential to extract more detailed information about individuals, underscoring its innovative contribution to the field.\n\nWeaknesses:\n\nIn Section 5.3, although it highlights the potential benefits of combining two retrieval models through an ensemble, how to apply the ensemble remains unclear. Additionally, the manuscript lacks clarity on how dialog is encoded within the retrieval model; it remains uncertain whether any preprocessing steps or LLM-based rewriting techniques were employed to adapt the training data.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1U3NImoeu3", "reviewer": "Reviewer_Ty38", "summary": "In this paper, a novel task is presented to address the limitations of traditional text-based ReID which rely on complete and one-time descriptions from witnesses. This work introduces interactive person re-identification (I-ReID) that employs multi-round question-answer dialogue to iteratively gather information of person. To achieve this, a question model is designed to generate context-aware questions based on both visual and textual cues. Additionally, the authors construct a dataset tailored to I-ReID and employs a looking-forward strategy to prioritize the most informative questions. Experimental results demonstrate its effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful questions and will address your concerns in the following.\n\n> Q1: Comparison of inference time and ablation study on the number of candidates.\n\nWe conduct these experiments on NVIDIA RTX 3090 GPUs. The inference time for the 5-round question generation of LLaVA-ReID and other baseline methods are shown in the table below:\n\n| Method         | Inference Time (second per person) | R@1 |\n| -------------- | ---------------------------------- | ----------------- |\n| SimIRV         | 0.007                              | 61.3              |\n| ChatIR         | 0.216                              | 63.9              |\n| PlugIR         | 13.06                              | 65.4              |\n| **LLaVA-ReID** | **0.621**                          | **73.2**          |\n\nOur multi-image questioner does not add significant computational overhead and achieves the best performance.\n\nThe retrieval results for different numbers of candidate images are shown in the table below:\n\n| Number of Candidates | R@1       | R@5      | R@10     | mAP       | BRI       |\n| -------------------- | --------- | -------- | -------- | --------- | --------- |\n| 2                    | 72.4      | 89.7     | 94.5     | 53.2      | 0.738     |\n| **4 (ours)**         | **73.2** | **90.6** | **96.0** | **53.3** | **0.719** |\n| 6                    | 71.4      | 89.2     | 94.2     | 52.8      | 0.764     |\n| 10                   | 71.6      | 88.9     | 93.8     | 52.7      | 0.772     |\n\nThe results indicate that setting $k=4$ provides the best balance between retrieval performance and computational efficiency.\n\nIn our previous experiments, the reported BRI was computed using a logarithm with base 2. According to the recently released code of BRI[1], we have corrected this by using the natural logarithm (base $e$). While this change slightly affects the absolute values of BRI, the relative ranking and overall conclusions remain unchanged.\n\n> Q2: Introduction of Gumbel subset sampling.\n\nThanks. To enable the selector to explore more possible combinations during the training stage, we employ a differentiable random sample strategy, namely, Gumbel-top-$k$ relaxations. This approach introduces randomness to increase the diversity of candidates while ensuring that the NLL loss gradient can be properly backpropagated for effective optimization.\n\n> Q3: BRI metric interpretation.\n\nBRI (Best Log Rank Integral) assesses the system’s ability to refine retrieval results over multiple interaction rounds. It measures the average area under the log-rank curve of all rounds. A lower BRI value indicates a more efficient and effective refinement process. We will make this clear in the next version.\n\n> Q4:  Input of Answerer in section 5.3.\n\nThe T-ReID datasets only provide caption-image pairs of individuals, we turn to the multi-modal LLM to simulate the users. Specifically, we use LLaVA-OneVision-Qwen2-7B-ov as the Answerer. It provides the follow-up descriptions, looking at the image of the target person. \n\n> Q5: Ambiguous figure of qualitative results.\n\nThe 4 images of candidates in the figure are the selected candidates and the rank represents the similarity ranking of the target person within the gallery **after** current round. We will improve the clarity of Figures 7 and 8 in the next version.", "claims_and_evidence": "The paper's central claim is that witnesses often struggle to provide complete and clear descriptions of a target person on their own. This claim is well-supported by researches in interactive cross-modal retrieval and is intuitively valid within the text-based ReID domain. Furthermore, experimental results validate that even datasets with human-annotated, fine-grained descriptions inevitably contain partial or ambiguous information.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem setting. Recall and mAP are standard metrics in text-based person ReID, and BRI (introduced in recent interactive retrieval research) is also used to evaluate the effectiveness of the interactive refinement process. Overall, the proposed methodology and evaluation framework align well with the problem context and practical application.", "theoretical_claims": "The paper does not propose new theoretical claims.", "experimental_designs_or_analyses": "I have reviewed the experimental analyses, and the experimental design appears sound and well-structured.", "relation_to_broader_scientific_literature": "This paper introduces an interactive setting for text-based person ReID. Previous studies mainly focus in improve the fine-grained matching capability and test on the static and complete description in existing dataset. However, real-world scenarios often involve incomplete or ambiguous descriptions provided by witnesses in isolation. This paper bridges that gap by proposing an interactive refinement approach, enhancing both applicability and robustness of text-based ReID in practical settings.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths\n1.\tThe interactive framework could be a new research direction for the community, improving the applicability of text-based ReID in real-world cases.\n2.\tThe paper constructs a new dataset with initial descriptions, fine-grained descriptions, and multi-round dialogues, enabling more effective training and evaluation of interactive ReID methods.\n3.\tThe proposed candidate selection model provides representative sample to the model and effectively benefit the question generation. Experimental results demonstrate the effectiveness of the proposed method.\n\n### Weaknesses\n1.\tThe multi-images questioner will lead to additional computational overhead. The authors should provide a comparison of inference time between their method and baseline models. Additionally, the number of candidate images is likely a critical factor in performance; an ablation study on this aspect would strengthen the analysis.\n2.\tThe expression clarity of Gumbel top-k sampling should be improved. While the rationale behind using Gumbel top-k sampling is understandable, its role and the necessity of the Gumbel trick should be more clearly explained.", "other_comments_or_suggestions": "1.\tThere is a typo in section 5.2 (We compared…)\n2.\tThe author should add an interpretation of BRI (e.g. lower is better) in the caption of Table 1.", "questions_for_authors": "1.\tIn section 5.3, what contextual information does the answer model use? Given that existing datasets lack additional fine-grained descriptions, how is this experiment conducted?\n2.\tThe figure 7 seems to be confusing. If the rank is 1, why is the target person not visible among the given images?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plARFPYka2", "reviewer": "Reviewer_1zUH", "summary": "This paper introduces a new task named interactive person re-identification (I-ReID), which aims to address the insufficient details in initial human descriptions. To support this task, the paper contributes a new dataset comprising multi-round dialogues generated using a customized approach with vision-language models (VLMs). Another key contribution is a novel interactive framework tailored for the ReID domain. The well-designed question model effectively identifies the most informative questions from the gallery, significantly enhancing performance. Experimental results demonstrate the superiority of this task-specific interactive framework over existing methods from other domains and show its ability to supplement missing details in text-based ReID methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reviewing our paper and giving us valuable suggestions. We will answer the questions one by one.\n\n> Q1: Visual hallucination in VLM captioning.\n\nThanks for your insightful comments. We acknowledge that GPT-like vision-language models (VLMs) are prone to hallucinations, particularly when generating fine-grained descriptions. Leveraging existing techniques, such as contrastive decoding [1], in data construction could further improve the accuracy and reliability of our descriptions. We will explore these enhancements in future work.\n\n[1] Leng S, Zhang H, Chen G, et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13872-13882.\n\n> Q2: Explanation of coarse-to-fine captioning.\n\nIn interactive ReID, we expect the witness to provide additional fine-grained discriminative information in the subsequent dialogue rounds. This information should not merely repeat what has already been stated. Therefore, we use the coarse description as a basis for generating the follow-up description.\n\nAn alternative approach could be to generate a fine-grained description first and then remove the overlapping content with the initial description. However, our generation strategy simplifies this process of decomposing the description, making it more efficient and structured.\n\n> Q3: Revision of figure 3\n\n(i) Thanks for your advice, we will change it to \"Descriptions from Witness\".\n\n(ii) The term \"clue\" refers to the embeddings of the context description $z_t$. We use this term to indicate that the candidates are selected based on the current \"clue\" we have at hand.\n\nWe apologize for any confusion caused by this terminology and will revise the figure in the next version.\n\n\n> Q4: Long text encoding of CLIP model\n\nAs mentioned in Appendix C, we interpolate the text positional embeddings from 77 to 192 and finetune fine-grained captions in Interactive-PEDES to obtain the Retriever. If the total text length exceeds 192 tokens, we truncate it accordingly. In practice, since the initial description has a maximum length of 48 tokens, the average length per turn in a five-round dialogue is (192-48)/5=28.8. This is sufficient for the current interaction setup, and we can also further interpolate it to longer encoding lengths.\n\n\n> Q5: Details about the question model functions when the interactive framework is integrated with existing text-based methods.\n\nPlease refer to our response to Q1 from reviewer fi2Z.\n\n\n> Q6: Re-organize section 3.2 (dataset construction)\n\nThanks for your valuable suggestions. We will revise this section to improve the clarity.\n\n> Q7: Abbreviation of interactive person re-identification.\n\nThank you for pointing this out. We will update the abbreviation to \"Inter-ReID\" in the new version.", "claims_and_evidence": "Yes. The main technical claims are: (i) **Representative Candidate Selection** emphasizes critical differences between individuals. (ii) **Informative Question Generation** dynamically selects the most informative questions for supervision. Both claims are well-supported by extensive experiments.", "methods_and_evaluation_criteria": "Yes. The paper proposes a multi-round dialogue benchmark for the interactive person ReID task. This benchmark includes three types of Q&A pairs, which could facilitate further research in this area.", "theoretical_claims": "The paper does not involve theoretical claims.", "experimental_designs_or_analyses": "Yes. I have reviewed all experimental designs and results. Table 1 and Figure 1 present performance comparisons against existing interactive retrieval methods in other domains. Table 2 highlights the effectiveness of the proposed method over existing text-based ReID methods, underscoring the necessity of interactive ReID for supplementing missing details. Additionally, ablation studies confirm the indispensability of each module.", "relation_to_broader_scientific_literature": "The contribution of the paper could be summarized in two main aspects.\ni) New task. Considering the practical demands of security and protection, the paper releases a novel task named interactive person reid, which involve progressively revealing the most relevant person according to the witness's description.\nii) Novel approach. The proposed method could not only emphasize the critical differences between individuals, but also dynamically select the most informative questions as supervision. Extensive experiments verify the effectiveness of the method. More importantly, the experiment results show off that the interactive framework could bring performance gain for the existing text-based reid methods, demonstrating the insufficient details in the existing human-annotated data.", "essential_references_not_discussed": "The authors discuss and compare a wide range of related methods.", "other_strengths_and_weaknesses": "**Strengths:**  \n1) The paper introduces a novel and practical task: interactive person ReID.  \n2) It provides the first dialogue-based dataset for this task.  \n3) The proposed approach is novel, and experiments confirm its effectiveness over existing interactive methods from other domains.  \n4) The framework can be integrated into text-based ReID methods to mitigate the issue of insufficient human-annotated details, significantly improving performance.  \n\n**Major Weaknesses:**  \n1) GPT-like VLMs are prone to hallucinations. Have the authors considered this issue in dataset construction? Could leveraging existing dataset annotations help mitigate hallucinations?  \n2) Why does the instruction for generating fine-grained captions start with coarse-grained captions? Would an alternative approach better leverage the VLM’s captioning capabilities?  \n3) Figure 3 needs revision: (i) The input to the text encoder should not be limited to the initial description, and (ii) the meaning of \"clue\" in the figure is undefined.  \n4) Will the text input exceed the maximum input length of the CLIP model?  \n5) It is unclear how the question model functions when the interactive framework is integrated with existing text-based methods.  \n\n**Minor Weaknesses:**  \n1) Steps ii and iii in dataset construction appear somewhat redundant and could be merged for clarity.  \n2) The abbreviation \"I-ReID\" may be confused with the widely recognized image-based ReID. A different abbreviation is recommended.", "other_comments_or_suggestions": "Please see the weaknesses.", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received four positive ratings, with all reviewers generally inclined to accept it. 
The paper proposes a new task, interactive person re-identification (I-ReID), to address the problem of incomplete person description in real-world text-based person ReID. For this task, it constructs a new dataset that includes coarse and fine-grained descriptions and multiple rounds of dialogue, and introduces a question model LLaVA-ReID. All reviewers approved of the proposed I-ReID settings, believing that it more closely reflects real-world scenarios and helps improve the applicability of text-based ReID in real-world cases. They find that the constructed datasets and benchmarks were well thought out, broke through the data limitations of existing benchmarks, and promoted further research in this field. The authors provide extensive experimental evaluations and detailed analyses to demonstrate the effectiveness of the proposed LLaVA-ReID model for solving the I-ReID task. 
Furthermore, all weaknesses have been addressed in the authors' response. Therefore, the Area Chair (AC) recommends accepting the paper.
Decision: Accept (poster)