Optimal Transport-based Labor-free Text Prompt Modeling for Sketch Re-identification
Authors: Rui Li
Tingting Ren
Jie Wen
Jinxing Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Sketch Re-identification; Optimal Transport; Prompt Learning
Primary area: machine_vision

Abstract: Sketch Re-identification (Sketch Re-ID), 
which aims to retrieve target person from an image gallery based on a sketch query, is crucial for criminal investigation, law enforcement, and missing person searches. 
Existing methods aim to alleviate the modality gap by employing semantic metrics constraints or auxiliary modal guidance. However, they incur expensive labor costs and inevitably omit fine-grained modality-consistent information due to the abstraction of sketches.
To address this issue, this paper proposes a novel $\textit{Optimal Transport-based Labor-free Text Prompt Modeling}$ (OLTM) network, which hierarchically extracts coarse- and fine-grained similarity representations guided by textual semantic information without any additional annotations. 
Specifically, multiple target attributes are flexibly obtained by a pre-trained visual question answering (VQA) model. Subsequently, a text prompt reasoning module employs learnable prompt strategy and optimal transport algorithm to extract discriminative global and local text representations, which serve as a bridge for hierarchical and multi-granularity modal alignment between sketch and image modalities.
Additionally, instead of measuring the similarity of two samples by only computing their distance, a novel triplet assignment loss is further proposed, in which the whole data distribution also contributes to optimizing the inter/intra-class distances. Extensive experiments conducted on two public benchmarks consistently demonstrate the robustness and superiority of our OLTM over state-of-the-art methods.

Optimal Transport-based Labor-free Text Prompt Modeling for Sketch Re-identification
Rui Li; Tingting Ren, Jie Wen, Jinxing Lit School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 1rhitestu. hit. edu. en, (rentt0410, (allaio,,,jjniiiiii 1ijiiningg118 @gmail. com, jiewen_pre126.co
Abstract
Sketch Re-identification (Sketch Re-ID), which aims to retrieve target person from an image gallery based on a sketch query, is crucial for criminal investigation, law enforcement, and missing person searches, Existing methods aim to alleviate the modality gap by employing semantic metrics constraints or auxiliary modal guid- ance. However, they incur expensive labor costs and inevitably omit fine-grained modality-consistent information due to the abstraction of sketches. address this issue, this paper proposes novel Optimal Transport-based Labor-free Text P Prompt Modeling (OLTM) network, which hierarchically extracts coarse- and fine- fineggrined similarity representations guided by textual semantic information without any ad- ditional annotations. Specifically, multiple target attributes are flexibly obtained by a pre-trained visual question answering (VQA) model. Subsequently, text prompt reasoning module employs learnable prompt strategy and optimal transport algorithm to extract discriminative global and local text representations, which serve as a bridge for hierarchical and granuligraaularrtyy modal aligmment between sketch and image modalities. Additionally, instead of measuring the similarity of two samples by only computing their distance, a novel triplet assignment loss is further proposed, in which the whole data distribution also contributes optimiz- ing the inter/intra-class distances. Extensive experiments conducted on two public benchmarks consistently demonstrate the robustness and superiority of our OLTM over state-of-the-an methods.
Introduction
With the growing need for urban public safety, traditional person re-identification (Re-ID) methods [ grad- ually becoming inadequate for criminal investigations and missing person tracking, the individuals of interest may not have been captured by surveillance cameras. To bol- ster social security management and combat criminal activ- ities, sketch person re-identification (Sketch Re-ID), which utilizes eyewitness clues to draw professional sketches as queries and match target images in a photo gallery database, has received widespread attention from researchers and scholars 4. 5 6), as shown in Fig. []] Nonetheless, due to the considerable disparity in modal heterogeneity result- ing from the varied sketch styles of different artists and the
witress
police
gallery
The illustration of sketch Re-ID. Different artists create sketches based clues provided by witness to assist the police in identifying targets.
Figure 1:
'Equal Contribution.
1 Corresponning Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

diverse postures of real pedestrians in monitoring, sketch Re-ID remains highly challenging and necessitates further exploration and investigation.
Giving the high generalization and inherent abstraction of person characteristics in sketches, the pedestrian features depicted in single sketch (such clothing and gender) may match multiple similar real images, as illustrated in Fig. [ A viable solution is to leverage the inter-modality interaction within the feature space to achieve hard alignment. Such methods typically employ loss constraints to directly map different modalities into a generic latent space A 7 g However, this hard alignment manner may not fully capture the complex dependencies and correlations that exist within and across modalities. To compensate for the lack of details in the above manner, another branch is introduce an intermediate modality to bridge two source modalities. For instance, [I generate simulated sketches through adversarial learning, but the generated sketches are inevitably corrupted with noise due to limited generation performance. Additionally, [2] construct benchmarks that contain textual information to alleviate modal gap; [6] improves inference efficiency by introducing text only during the training process. Despite the fact that these additional texts do contribute to mitigating the modal gap, they are all manually labeled, requiring significant human labor in real-world applications. Moreover, existing text-guided methods 7 only focus on global text embeddings paper aims to address two key challenges: i) developing sufficient textual information as a transition mechanism without incurring additional costs, and ii) further exploring fine-grained discriminative information for oulti-granularity interaction.
masks, neglecting finer and richer local features. Therefore, this
To address the above issues, we propose the Optimal Transport-based Labor-free Text Prompt Modeling (OLTM) framework, which implicitly incorporates text semantic information during training, facilitating hierarchical and multi- cualiiigrallarttyy modal alignment. In particular, OLTM is composed of three main components: i) text prompt reasoning (TPR); ii) text-injected coarse-grained alignment module (TCA); iii) consensus- uunddd fine-grained interaction module (CFT). On the one hand, to introduce text sequences without additional manual annotation, we dynamically transfer pre-trained language- visual knowledge into the downstream task. Specifically, TPR first generates multi-dimensional person attributes based on real images with a pre-trained visual question answering (VQA) model. Then, these attributes are inserted as fixed parts into learnable prompts to obtain the textual embedding representations. TCA integrates global parts of the embeddings to achieve the ceare-graiine alignment across modalities. On the other hand, to explore fine-grained information, we employ optimal transport theory to enhance deep-level interaction. Concretely, TPR formulates the mapping from local parts of the textual embeddings to more discriminative feature representations, i.e., consensus, as an optimal transport problem. Subsequently, guided by consensus, CFI selectively focuses on key details, extracting fine-grained conceptual representations for sketch-ID. In addition, due to the significant heterogeneity gap between modalities, using Euclidean distance as a sole metric to measure feature similarity is inadequate. Thus, we propose a triplet assignment loss to optimize feature distance measurement and improve model performance. Extensive experiments are conducted on two challenging benchmarks, which demonstrate the favorable comparison of OLTM with other state-of-the the-arrrr methods.
The core contributions of this work summarized as follows: (1) optimal transport-basee based labor-free text prompt modeling framework for sketch Re-ID. To our best knowledge, this is the first attempt to apply VQA-generated text responses as a means achieve modal alignment in sketch Re-ID without any additional annotations. (2) A novel text prompt reasoning module is deployed to dynamically extract global textual embeddings and discriminative fine-grained consensus, which guide the hierarchical multi-granularity alignment module in injecting semantic knowledge into the modeling process. (3) A new triplet assignment loss proposed, which optimizes siierr/intra-llass distance by considering overall data distribution information.
This paper proposes novel
2
Related Work
Sketch Re-identification As an important part of public safety guarantee, sketch Re-ID is a novel and challenging task that aims to match person image with given professional sketches. Existing sketch Re-ID methods could be roughly classified into two groups according to their interaction modes, i.e., hard alignment methods 4 9 and soft alignment methods [12 10 6 The former try to learn modality embeddings in a common latent space by employing some modality interaction operations or semantic metrics. Pang et al. [ pioneered a sketch-photo benchmark and introduced
2

---

cross-domain adversarial learning to narrow the feature gap. Zhang et al. ( proposed an advanced cross-modal learning mechanism for handling correspossgn information between modalities. However, due to significant differences between modalities, this direct alignment paradigm inevitably loses fine-grained modality-specific cues [33. Hence, some of the latter methods investigate gentler alignment techniques through transitional modality. For example, Chen et al. [T0] designed a dynamic updatable auxiliary sketch modality to increases the diversity of training samples; Zhai et al. [12] introduced multi-modal Re-ID task by combining text and sketch query for retrieval, exploiting their complementary advantages. Obviously, auxiliary modalities lacking detailed information may introduce noise, while data annotation is a fabor-intensio task. In this paper, we first attempt to text attributes generated by a reasonable VQA model as guidance for achieving multi -atiiirraatrtty alignment across modalities in sketch Re-ID.
For optimizing the moving cost between distributions, Optimal Transport (OT) was first proposed by Kantorovich [14), which has shown significant potential in machine learning and
Optimal Transport
computer vision, e.g., domain adaptation [5 16 17,, learning with noisy labels [8] 19,, and feature matching [20] 221. Zhang et al. [22] incorporated OT into the re-ranking phase of image retrieval, significantly improving accuracy and efficiency. Similarly, Sergio et al. [23] first applied OT in visual place recognition and introduced novel local feature aggregation method. In semi-supervised person Re-ID, OT often achieves the mapping between pseudo labels and classes as a classifier [24. (5).. In addition, Ling et al. (26) designed a assignment strategy for alleviating the intra-identig variations; Wasserstein distance was used to rectify the original global distance between samples and provides aligned distance estimation for local features (27]. Considering the enormous potential of OT in feature aggregation and distribution mapping, our study adopts OT to assist fine-grained alignment between modalities and guide the model in extracting the overall sample distribution pattern.
Prompt Learning Prompt learning initially garnered widespread attention and extensive research in natural language processing (28, 29
30), which has gradually demonstrated significant potential
in vision-language (V-L) models [3
32 [
and pure vision models 34 355 36]. Prompt learning
provides a flexible way to adapt pre-trained models to downstream tasks by training only additional parameters. This enables prompts to capture task-specific information while guiding the fixed model's s performance (37 38). In sketch-based image retrieval, [39] innovatively learns unified prompt for different branches in CLIP's 440 visual encoding layer, fully exploiting CLIP's zero-shot learning potential. In text-to-image person Re-ID,, [41] introduces a multi-prompt strategy to integrate text prompts from various sources for fine-grained interaction. Furthermore, Li et al. (42) first attempt to conduct in-depth research on zero-shot multi-modal RelD through large foundational model. In this paper, we delve into the significant application of prompt learning in sketch Re-ID, innovatively generating global text representations by integrating fixed and learnable prompts, and utilizing OT to reason consensus for effectively guiding detailed interactions across modalities.
3 Preliminaries
3.1 Problem Statement
To ensure clarity, we represent the gallery containing m images as G == W: and the query set containing sketches S as S = {S y }ー=1- where y E (1...., C} are the identity labels for C' distinct pedestrian entities. Notably, each entity may include multiple images and sketches. The goal of Sketch Re-ID is to retrieve pedestrian images from the gallery & that match one multiple given sketch. Like [6], there exist two types of query methods: single sketch query and multiple sketches query. This section will use single sketch query as an example, and the same applies to multiple sketches query. Formally, we define a matching function M : S x G - Raxm that assigns similarity score to each pair S,). The objective learn function M such that for any sketch Sj and image 1,,
S,), > M(I,, S2) if ys = yj and yk + yj- where 4, and 4 are images from the gallery set, and yı and y are their respective identity labels.
(1)
3.2 Optimal Transport
Optimal Transport is mathematical theory that focuses on finding an efficient solution between two probability distributions, minimizing the cost of transporting one distribution into another. We briefly review the theoretical derivation of optimal transport. Let Ir, == fo Ri 1, = 1} represents
3

---

Hlerarchical and Multi-graullarlt Modal Alignment
Crose moss-nebbl
L
Crose Doos-Alrviii Hrviion
Malti
LTD
12x
Image Enoder
(b)CA
(c)CFI
(dTTAL
attribute
[CLS] Embedding
clusters m
Tolem
white sleeves
long hair
pando
Embedding
Local
VQA
black shorts
image
positive sketch
negative skotch move
PR
Prompt
Encoder
pull
push
Figure 2: Overview of our proposed OLTM network. Our model includes four main parts, i.c., text prompt reasoning (TPR), text-injected coarse-graings alignment Module (TCA), mussensus-guiedd fine-grained interaction module (CFT) and triplet assignment loss (TAL). Specifically, TPR flexibly generates target characteristics through VQA, and combines prompt learning and optimal transport to reason text global embedding and local consensus. TCA and CFI extract modality-specinf representations from image and sketch modalities to achieve hierarchical and multi-gran gaati-granullrity alignment. Finally, TAL is designed to optimize distance measurement between samples and improve the model's capacity to capture local relationships.
the probability simplex, where 1, is the r-dimensional vector of ones. Given two probability simplex vectors I'm and B In and cost matrix C E Rmxn, the objective of OT is seek the optimal transport plan p mapping a to B at the minimum cost:
dc(a, PEU(O,
min
(C,P),
(ar B) = P MX
PIn 1 mm
where ((,, 33 denotes the transport polytope of a and [3, i.e., the solution space of P. The above problem is to find optimal solution P in set of all possible joint probabilities of (X, Y), where X and Y represent random variables with marginal distribution and B.
Eq.8 indicates that OT is linear programming problem which is theoretically solvable in polynomial time, but its complexity becomes prohibitively high the feature dimension increases [43]. To
this end, Sinkhorn algorithm 44] adopts an iterative strategy to obtain the optimal solution P
Diag(u) () K Diag(v) (v) with near-square complexity [45]. and
be solved through alternately
iterating the following two equations: (=))) a a<(Kol (x))))))) and (>() 3/(K" (=)), where K exp(CC/e) is the regularization coefficient and z is the iterations (cf. Appendix). Since this
method integrates the importance of all features when solving the optimal solution, it can analyze the overall data distribution.
The Proposed Method
4.1 Overall Architecture
Fig. 2 provides an overview of the OLTM architecture. The image and text encoders discussed in this paper are based on CLIP (40), and any language-visual model utilizing Transformer architecture may also he employed. Notably, the image encoders utilized for both images and sketches employ shared weights ensure the mapping of features into unified semantic space. For input RGB image, obtain embeddings R (Ras, Ti..TT.... Ti...., rp} R(++1)xd through the image encoder, where p is the number of non-overlapping patches, Ras and R = {ra)e= represent d-dimensional global and local features, respectively. Similarly, the embeddings of a sketch can be represented as S {Sets Stoctt) {Sess . 8p} h (p+1)xd Firstly, to provide reasonable text semantic

---

guidance. Text Prompt Reasoning (TPR) generates attribute descriptions about pedestrians based on the RGB image, and obtains the textual embeddings T {T cos: T Tieat) through prompt learning. Then, TPR extracts the fineggrained consensus X from T toeat through clustering. Subsequently, Text- injected Coarse-grained Alignment (TCA) module embeds global contextual information feo into visual features Rels and Sas- Meanwhile, Consensus-guided Fine-grained Interaction (CFI) module utilizes X to address fine-grained semantic misalignment between Rioco! and Steeat. Additionally, due to the significant differences between sketches and RGB images, Euclidean distance between independent samples may ignore the influence of overall sample distribution. Thus, we introduce more comprehensive distance measurement method and propose triplet assignment loss Liat- During training. all these modules will be jointly optimized through identity loss [46] Lid and Ltat: COLTM Lid + M/Caai, where 1) is scaling factor. During inference, only Rais and Sels are used to match queries for practical application requirements.
4.2 Text Prompt Reasoning
Significant image differences and inherent abstract nature, cause semantic misalignment during knowledge acquisition, severely impacting model's reasoning and generalization capabilities. To address this issue, TPR introduces intermediate modality to guide alignment between modalities without additional costs. Moreover, TPR employs a dynamic consensus acquisition strategy to enhance the discriminative power of local text features.
Text Attribute Generation Sketches, unlike conventional Re-ID tasks, are vulnerable to subjective emotions and drawing skills of artists, leading to lack of detailed information crucial for model learning. The text's objectivity and flcxibility prompt the model to focus more on semantic contextual information during knowledge acquisition. However, directly generating comprehensive textual description of pedestrian images inevitably introduces irrelevant noise, thereby reducing model performance. Therefore, we retain the advanced modeling capabilities of large-scale language-visude models for images as possible. Specifically, for a given RGB image, TPR utilizes a pre-trained visual question answering model to address & specific details (cf. Appendix) and acquire corresponding descriptions for the target: att = {att1, atty,.... att;), ). Importantly, this process introduces textual detail guidance during model training but excludes text-related components during inference.
Learnable Prompt Strategy Inspired by 47 we combine the learnable prompt with original text attributes, without incurring additional expert knowledge compared to the handcrafted prompt. Concretely, TPR initially transform these attributes into tokens through CLIP tokenizer, i.c., a = T Tokenizer(att) Then, / learnable prompts {PI, P2.... pt}
are embedded into these fixed attributes tokens, forming the textual description: q = {p1, a1 P2, a2 Pt, } This integration introduces dynamic knowledge learning mechanism that reduces noise introduction compared to handcrafted prompts, while enhancing the flexibility of modal interaction and transferability of text embeddings. Subsequently, the whole token q is fed into frozen text encoder to generate text embeddings
T {T sos &1, ........ tn. Toos} E (xx+2)xd where sas and Toos denote the [SOS] and [EOS] token, n is the number of d-dimensional word tokens. Based on widely-used token selection, Toos serves the global feature, while Tiocal }i represents sequence of basic local tokens.
Dynamic Consensus Acquisition To more effectively address fine- -in-igraied semantic variations (e.g., hats, shoes) across modalities, we explore methods to filter out non -on-informative features for enhancing the representational capacity of text embedding for detailed information. Therefore, based on local textual feature T Tooat. we employ metric learning to draft a dynamic consensus acquisition strategy for capturing the discrininative prototypical representations X.
To begin, in order to adaptively learn related-task knowledge, local text representations T locat E RR are fed into consensus multi-layer perceptron (ConMLP) blocks achieve feature enhancement. Then, prototypical descriptor is formed by assigning all enhanced features to a set of atoms. The cost matrix C E maxm can be calculated for assignment, where the (i. (th element Ct,, represents the cost of assigning a feature to an atom. In other words, C' evaluates the affinity between the enhanced features and the prototypical atoms. Concretely, to introduce certain priors as guidance (33]. the enhanced features are used to learn the cost matrix through two fully connected layers initialized randomly. In addition, some irrelevant information in textual features, such as those representing association, may disrupt the model's ability to learn the target details. Inspired by the solutions used in graph matching and key-point matching 20 23), we set a learnable "bin" in C. Due to the differences between prior distributions, non-informative features can be assigned to it. Specifically,
5

---

we extend the cost matrix from to C = C, E + [=x((=1) and è wln, where w is learnable parameter and 1, [1, ....... 1] R represents n-dimensional vector of ones. Following Sec. 3.2) we consider that assignment where the enhanced features' mass, a = 1, should be assigned to the atoms or the "bin", , [ n m] is an optimal transport problem:
(a (3 min (C,P), P
U(a (P ( P1 a, 1, 3).
The optimal assignment plan can be solved by Sinkhorn algorithm (cf. Appendix) through iterative strategy. For better descriptors quality, the extra "bin" is discarded obtain the optimal assignment P E 1 R Finally, the augmented comprehensive representation, i.e., consensus X E R is
obtained by aggregating enhanced textual features and optimal transport plan: X = P -PPT Tocal.
4.3 Hierarchical and Multi-granularity Modal Alignment
TPR develops a robust transitional means to facilitate reasonable and efficient modal alignment. In order to leverage effectively the multi-granularity textual information, TCA utilizes global embedding to achieve moare-ggriine modal interaction, while CFI optimizes fine-grained alignment based on enhanced comprchensive representation.
Text-injected Coarse-grained Alignment Module TCA utilizes transformer architecture, known for its efficiency in modeling long-distance dependencies, to capture global information, as shown Fig. Furthermore, we implement cross- mechanism using global text embedding to emphasize contextual information injection at the beginning. For an input pair (R. S), cross-attention mechanism computes the query using text global representation T cos and derives the key and value from mondality-specifie global features Rits and Sels
Q = Toos 2º, K R/S = (R/S)ets w V a/S = (R/S) w
CA(R/S, cos) = Attention R/S: K R/S, V R/s);
where R/S signifies identical operations across both modalities; w w and v denote shared learnable parameters, while Qn/s- K R/S and V R/S represent query, key and value for either the RGB or sketch modality, respectively. After obtaining the fusing features that contain textual knowledge, TCA introduces the standard transformer blocks to refine modality-specifie features. Finally, we can acquire the final global concept representations Rd and S Sde for sketch Re-ID.
Consensus-guided Fine-grained Interaction Module Due to the inherent complexity of sketches and RGB images and the potential for semantic misalignment during learning, capturing detail variations in modalities is crucial. Fortunately, the fine-grained consensus X provided by TPR, which contains rich detail information, offers key solution to this problem. CFI adopts transformer structure based on multi-head cross attention, and it converts the original local feature R and S locat more discriminative representations through for robust Re-ID:
Q QRIS = X u K R/S (R/S) u V (R/S) w."
Head R/S
Allention QRIS, K KRIS, V.s.V.n R/s),
(5)
MH(R/S, X) Concatí Head R/S Head R/S
where & " and w" denote the parameters of project layers of h-th head for both modalities. As result, the text detail information about pedestrian characteristies can assist our model address the problem caused by the diversity and uncertainty of sketches and RGB images. Eventually, the local concept representations R Rioeat and S Steeat can be obtained.
4.4 Triplet Assignment Loss
The triple loss, a commonly-used matching loss cross-modal learning, performs well performance through adjusting the distance of hardest negatives in scenarios like image-text matching [12 48 and video-text retrieval [49]. However, this strategy independently trains each semantic part with equal contribution, disregarding the overall data distribution impact when multiple samples from

---

Table I: Comparison with state-of-the-ar methods on Market-Sketche dataset. Both training
and testing set uses all sketches. 'S' and 'M' represent single-query and multi-query respectively.
"Backbone" refers to network structure used by each method, mainly including: ResNet50 [0 and CLIP [40]. Bold values represent the optimal results.
Methods
Query
Backbone
Reference
mAP
Rank @
Rank @5
Rank
DDAG 5
S
ResNet50
ECCV'2020
12.13
11.22
25.40
35.02
CM-NAS 52
S
ResNet50
ICCV'2021
0.82
0,70
2.00
3.90
CAJ 53
S
ResNet50
ICCV
2.38
1,48
3.97
7.34
MMN [54
S
ResNet50
MM'2021
10.41
9.32
21.98
29.58
DART 55
ResNet50
CVPR'2022
7.77
6,58
16.75
23.42
DCLNet
56
ResNet50
MM'2022
13.45
12.24
29.20
39.5
DSCNet
57
S
ResNet50
TIFS
14.73
13.84
30.55
40,34
DEEN
58
S
ResNet50
CVPR 20223
12.62
12.11
25.44
30.94
S
19.61
18.10
38.95
50.75
BDG
ResNet50
MM'2023
M
24.45
24.70
50.40
63.45
S
34.97
31,52
57.17
70.46
UNIReID
CLIP
CVPR 2203
M
55.18
56.63
82.33
91.97
S
38.35
36.75
63.88
74.05
OLTM (Ours)
CLIP
M
62.55
69.48
90.36
95.18
different modalities exhibit slight differences. This oversight may lead to inaccurate estimation of sample distances and potentially result in sub-optimal local minima [27]. To this end, we propose a new triplet assignment loss (TAL) to establish more rational measure for evaluating the proximity of local features.
For an input positive pair RR; S:) in mini-batch the feature representations obtained through model inference R, and S,. we treat the feature sets of all samples for each of two modalities in as two discrete distributions, their alignment can be considered an optimal transport problem. The cost matrix Č is derived from pairwise feature similarities: Č₁Z [(R.) ) 5J]+. We aim to acquire the optimal transport matrix P with the least amount of cost, where P represents the assignment weight of (R; Sj) obtained after balancing the overall distribution. TAL be represented based on triplet loss as the weighted sum of the original distance and the optimal assignment distance, dynamically updated certain rate ry:
RR;, Si) [m D(R,, S.) + D(R,, S,). + Im D(R, S₁) D D(R Si)l+, DRR, S:) 7E,R, S:) + (1->)(1 (R,,)
(6)
where [x]+ max x. 0). R. argmaar n, DRR, and $, ar argmazz D(R; Sj) the most similar negatives in for (R,, S,), and E(R,, S,) R S/ll2 denotes the Euclidean distance between feature representations.
5
Experiment
5.1 Experiment Setup
Datasets Two publicly available benchmark datasets, namely PKU-Sketch [4] and Market-Sketchh IK (6). are utilized for performance evaluation. Both of them are sketched and annotated by professional artists. PKU-Sketch is the first publicly Sketch Re-ID dataset, containing data for 200 pedestrians, with each individual being represented through one sketch and two photos. In accordance with the setting of [4, we randomly select 150 identities for training and 50 for testing, and final results are derived from the average of 10 experimental runs. Market--Seethh-IK is large-scale dataset derived from the Market- 1501 [1, which created by six artists based on descriptions, featuring multiple perspectives and artistic styles. The training set consists of 2,332 sketches and 12,936 photos, while the testing set comprises 2,375 sketches and 19,732 photos. Following the experimental setup in ( method will be evaluated in three settings: singlc-query, multi-query, and cross-style retrieval.
7

---

Table 2: Comparison with state-of-the-an methods on PKU-Sketch dataset. 'Backbone' includes
GoogleNet [62]. VGG-16 [3), ResNet50, ViT [64] and CLIP. denotes the unavailable results. *f' indicates that we reproduce UNIRelD results following our training configuration.
Methods
Backbone
Reference
mAP
Rank
Rank @5
Rank @ I0
TripleSN
65
2016
9.0
26.8
42.2
GNSiamese
66
GoogleNet
T06'2016
28.9
54.0
62.4
AFLNet
GoogleNet
MM1
34.0
56.3
72.5
LMDI S
VGG-16
2020 2020
49.0
70.4
80.2
SKetchTrans
10
VIT
MM'202
84.6
94.8
98.2
CCSC 0
VIT
MM'
83.7
86.0
98.0
100.0
SKetchTrans
5
ViT
PAM1'2023
85.8
96.0
99.0
UNIReID
CLIP
CVPR '2023
88.7
92.4
98.0
99.6
DALNet
ResNet50
2024
86.2
90.0
98.6
100.0
OLTM (Ours)
CLIP
91.4
94.0
99.4
100.0
Market-Sletth-II (Singl-Query)
Marcet (Malti-Qury))
(0 PKU-Skuthh
Figure 3: The Rank-5 retrieval results on two datasets. For the Market-Sketch-I dataset, both single-query and multi-query scenarios presented. Green border indicates correctly retrieved target pedestrians, while yellow border indicates incorrectly matched pedestrians.
Evaluation Metrics In line with 4 9 6 59 we use Rank-k metrics (k = 1,5, 5, 10) and mean Average Precision (mAP) as evaluation metries. The higher values of the above three metries, the better performance.
image encoder, and extracts text features with pre-trained CLIP Text Transformer. Fine-grained text attributes are derived from ViLT-based (60) VQA model. Importantly, our VQA model is replaceable. To avoid the cost overhead from multiple calls to VQA model during training and inference, text attribute generation is performed during the data processing stage. For multi-query scenarios, weighted sum of multiple sketches. Input images are resized to 288 x 144, and augmented with random horizontal flipping and style augmentation [6]. More experimental configuration details available in the supplementary materials.
Implementation Details OLTM uses pre-trained VTTB-V1- 40
input
5.2 Comparison with State-of-the-Art Methods
Performance on Market-Sketch-IIK The experimental results for Market-Sketch-IE M are shown in Tab. OLTM significantly outperforms all state-of-the-ar methods in both single-query and multi-query settings. In the single-quen scenario, OLTM achieves an mAP of 38.35% and Rank- of 66,75% surpassing ofate-of-the-ee method by 3.38% and 5.23%, respectively. scenario, OLTM achieves mAp of 62.55% and Rank- 1 of 69.48%, exceeding state-of- of the-rhh methods by 7.37% and 12.85% respectively. This result is primarily due to the introduction of textual semantic information in the training process of OLTM, which implicitly guides images and sketches focus on modality-invarianl features. Through hierarchical and granuliranullrrtty alignment, the model is able to uncover discriminative fine- iine-grained information, leading to more
the multi-query

---

Table 3: Ablation studies on Market-Sketch- IK dataset. Training and testing are under the multi-query setting. "Handcrafted" and "VQA" denote manually annotated and VQA generated text attributes, respectively. "Template" represents the sentence template defined by experts. "Prompt" denotes the learnable text prompts. The 'Baseline" uses an image encoder to process both modalities and employs simple cross-attention to integrate the global features. 'Lmil [67
represents the hard triplet loss. Bold
values represent the optimal results.
Prompt setting
Module
Loss
Metries
Handerafted
VQA
Template
Prompt
Baseline
TCA
CFI
Ln
Lui
Lua
mAP
Rank @ I
55,47
60.04
61.46
68.07
V
V
61.81
67.47
/
61.76
65.46
V
57.74
60.84
/
d
V
V
61.10
65.66
54.93
57.83
d
v
61.63
66.06
62.55
69.48
accurate queries. The visualization results on Market-Sketch. IK are shown in Fig. 3 When sketch possesses clearly distinguishable features, query results are satisfactory. In contrast, when sketch closely resembles multiple images, making its features challenging to distinguish with the naked eye, the model encounters errors that are justifiable. addition, to test the generalization capability of OLTM on unknown styles, cross-style retrieval evaluation is performed on Market-Sketch-IIK and detailed results are in supplementary material.
Performance on PKU-Sketch. Tab. presents the model's performance on PKU-SKetch dataset. The results indicate that our OLTM outperforms all competitors by a significant margin. For example, mAP and Rank- 1 of OLTM are remarkably high at 91.4 and 94.0. surpassing wate-of-the-aa method by 5.2% and 4.0%, respectively. Because the sketches on PKU-Sketch contain more detailed information, they can assist grati-gra interaction in acquiring more fine- fine-graines grained knowledge. Fig. 3 iillustrates the top-5 visualization results of OLTM on PKU-Sketch. Our method can accurately identify the target pedestrians despite challenges such variations in posture, viewpoint, occlusion, sketch abstraction, and different painting styles.
5.3 Ablation Study
In this section, ablation experiments are conducted on Market-Sketch-II IK evaluate the effectiveness of each component within the OLTM framework.
The Effectiveness of Text Prompt Reasoning. To evaluate the contribution of different prompt setting, we train the model with different combination of each setting. As shown in Tab. 3 the combination of VQA and Prompt brings significant contribution. Compared to the strategy of directly aligning different modalities, leveraging TPR to implicitly guide the alignment results in improvements of 7.08% in mAP and 9.44% rank- 1. This significant improvement is primarily attributed to introducing textual information during the training phase, which enables the model to effectively capture semantic correlations between images and sketches during inference. Furthermore, compared to manually annotated fixed attributes, those generated by VQA enhance the model's performance by adaptively adjusting the level of detail on which it focuses. Moreover, the introduction of learnable prompts improved the mAP and rank- 1 by 0.79% and 4.02%, respectively, compared to fixed templates. Prompt learning can enhance the network's 's learning and reasoning capabilities, allowing it to more flexibly adapt to diverse modalities and enhance its sensitivity to subtle distinctions.
The Effectiveness of Our Designed Modules. To validate the effectiveness of the TCA and CFI module, we progressively integrate them into the baseline and evaluate performance. The results in Tab. indicate that both modules significantly enhance the alignment and interaction capabilities of model across modalities. Specifically, TCA improves mAp and rank-1 by 3.36% and 4.82% compared to baseline, respectively. The introduction of textual information in TCA effectively provides semantic guidance for coarse-grained alignment between modalities, enabling the model

---

to focus more on similar semantic relationships. Furthermore, the integration of CFI has increased mAP and rank-1 by 1.45% and 3.82%, respectively. CFI selectively focuses on key regions in visual concept representations through semantic consensus. This process optimizes feature interaction and ensures capturing more detailed information a fine-grained level.
The Effectiveness of Triplet Assignment Loss. The experimental results in Tab. demonstratt that the combination of proposed TAL Lia and identity loss LID achieves optimal performance. The identity loss ensures that the model correctly identifies different individual identities, while TAL further optimizes feature space by pulling posi- tive samples closer and pushing negative samples apart. Additionally, to verify the generalization of TAL. as shown in Tab, 4 we achieve superior performance across various network frameworks by substi- tuting TAL for the weighted regularization triplet loss (WRT) [59] and hard triplet loss (HTL.). Balancing between Euclidean distance and optimal transport distance can significantly enhance model per- formance. Please refer to the supplementary materials for more verification experiments on the key role of overall data distribution in enhancing sample feature distance.
Table 4: Performance of TAL Lua with various baselines. '+' represents WRT; ** repre- sents HTI Lan-
Methods
MAP
R@1
BDG BDG + TAI.
24.45
24.70
27.79
27.71
baseline*
57.74
60.84
baseline + TAL
58.41
61.04
OLTM
61.66 31.63
66.06
OLTM + TAL
62.55
69.48
Conclusion
In this paper, we present optimal transport-based labor-free text prompt modeling (OLTM) frame- work for sketch re-identification. OLTM embeds text prompt reasoning module and distance measure- ment into transformer for achieving hierarchical multi-granulant alignment through the guidance of text semanties, leveraging the advantages of prompt learning and optimal transport. In addition, to address the limitations of Euclidean distance in measuring sample similarity, we propose a triplet assignment loss that guarantees more effective standard based on the overall data distribution. Extensive experiments conducted on two public datasets indicate outstanding performance compared to other sate-of-the-at methods for sketch Re-II>.
In this work, we employ text injection and sample distance optimization to direct the model's attention toward key details, thereby minimizing performance losses due modal gap and sample abstraction. However, our experiments revealed that when confronted with extremely vague sketch samples (i.e., those that human cannot discern features match), the model's identification process deteriorates into random selection among multiple potential outcomes. Therefore, sketch Re-ID heavily depend on the quality of sketches. Enhancing the discriminability of sketches without incurring additional labor costs is a topic worthy of further exploration in future research.
Discussion:
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (62272133) in part by the Shenzhen Science and Technology Program 11ZZD202309231146000000 in part by the Shenzhen Colleges and Universities Stable Support Program (GXWD20220811170100 part by the Key Laboratory of Industrial Equipment Quality Big Data (2024-F:BBD-00), in part by Guangdong Basic and Applied Basic Research Foundation (2024A1515030213
in
References
[1] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pages 11 1116-11 1124, 2015.
[2] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[3] Jing Xu, Rui Zhao, Feng Zhu, Huaming Wang, and Wanli Ouyang. Attention-aware composi- tional network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2119-2128. 2018.
10

---

[4] Lu Pang, Yaowei Wang, Yi-Zhe Song, Tiejun Huang, and Yonghong Tian. Cross-domain adversarial feature learning for sketch re-identification. Proceedings of the 26th ACM international conference Multimedia, pages 609-617, 2018.
[5] Cuiqun Chen, Mang Ye, Meibin Qi, and Bo Du Sketchtrans: Disentangled prototype learning with transformer for sketch-photo recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[6] Kejun Lin, Zhixiang Wang, Zheng Wang, Yinqiang Zheng, and Shin'ichi Satoh. Beyond domain gap: Exploiting subjectivity in sketch-based person retrieval. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2078-2089, 2023.
[7] Cuiqun Chen, Mang Ye, and Ding Jiang. Towards modality-agnostic person re-identification with descriptive query. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15128-1511 11128-1513,,0 2023.
[8] Shaojun Gui, Yu Zhu, Xiangxiang Qin, and Xiaofeng Ling. Learning multi-level domain invariant features for sketch re-identification. Neurocomputing, 403:294-303, 2020.
[9] Yafei Zhang, Yongzeng Wang, Huafeng Li, and Shuang Li. Cross-compatible embedding and semantic consistent feature construction for sketch re-identification. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3347-3355, 2022.
[10] Cuiqun Chen, Mang Ye, Mcibin Qi, and Bo Du Sketch transformer: Asymmetrical disen- tanglement learning from dynamic synthesis. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4012-4020, 2022.
[11] Xingyu Liu, Xu Cheng, Haoyu Chen, Hao Yu, and Guoying Zhao. Differentiable auxiliary learning for sketch re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence volume 38, pages 3747-3755, 2024.
[12] Yajing Zhai, Yawen Zeng, Da Cao, and Shaofei Lu. Trireid: Towards multi-modal person re- identification via descriptive fusion model. In Proceedings of the 2022 International Conference on Multimedia Retrieval, pages 63-71, 2022.
[13] Enhao Ning, Changshuo Wang. Huang Zhang, Xin Ning, and Prayag Tiwari. Occluded person re-identification with deep learning: a survey and perspectives. Expert Systems with Applications, page 122419, 2023.
[14] Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences, 133(4)11381-1382 2006.
[15] Wanxing Chang. Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for universal domain adaptation. Advances in Neural Information Processing Systems, 35:29512- 29524, 2022.
[16] Bharath Bhushan Damodaran, Benjamin Kellenherger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV), pages 447-463, 2018.
[17] Kilian Fatras, Thibault Séjourné, Rémi Flamary, and Nicolas Courty. Unbalanced minibatch optimal transport: applications to domain adaptation. In International Conference on Machine Learning, pages 3196-3197. PMLR. 2021.
[18] Wanxing Chang, Ye Shi, and Jingya Wang. Csot: Curriculum and structure-aware optimal transport for learning with noisy labels. Advances in Neural Information Processing Systems, 36:8528-8544 2023.
[19] Chuanwen Feng, Yilong Ren, and Xike Xie. Ot-filter: An optimal transport filter for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16164- 16164-16174, 2023.
[20] Paul-Fdouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 4938-4944 2020.
11

---

[21] Jiaming Sun, Zehong Shen. Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector- free local feature matching with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 19922-9931, 2021.
[22] Chao Zhang, Stephan Liwicki, and Roberto Cipolla. Beyond the cls token: Image reranking using pretrained vision transformers. In BMVC, page 80, 2022.
[23] Sergio Izquierdo and Javier Civera. Optimal transport aggregation for visual place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 16164-16174, 2023.
[24] Jiangming Shi, Yachao Zhang, Xiangbo Yin, Yuan Xie, Zhizhong Zhang, Jianping Fan, Zhongchao Shi, and Yanyun Qu. Dual pseudo-labels interactive self-training for semi-supervised visible-infrared person re-identification. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 11218-11228, 2023.
[25] Jiangming Wang, Zhizhong Zhang, Mingang Chen, Yi Zhang, Cong Wang, Bin Sheng, Yanyun Qu, and Yuan Xie. Optimal transport for label-efficient visible-infrared person re-identification. In European Conference Computer Vision, pages 93-109. Springer, 2022.
[26] Yongguo Ling, Zhun Zhong, Zhiming Luo, Fengxiang Yang, Donglin Cao, Yaojin Lin, Shaozi Li, and Nicu Sebe. Cross-modality earth mover's distance for visible thermal person re- identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1631-1639, 2023.
[27] Zhizhong Zhang, Yuan Xie, Ding Li, Wensheng Zhang, and Qi Tian. Learning to align via wasserstein for person re-identification. IEEE Transactions on Image Processing, 29:7104-7116, 2020.
[28] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint (0Xiv::204408699) 2021.
[29] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101. 100.00990, 2021.
[30] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint arXiv:21 2021.
[31] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for languuge models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6816-16825, 2022.
[32] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9))2337-2348, 2022.
[33] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659-15669, 2023.
[34] Menglin Jia, Luming Tang, Bor- Cor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[35] Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631-648. Springer, 2022.
[36] Zifeng Wang, Zizhao Zhang, Chen- Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
12

---

[37] Bingzhi Chen. Zhongqi Wu, Yishu Liu, Biqing Zeng, Guangming Lu, and Zheng Zhang, En- hancing cross-modal retrieval via visual-textual prompt hashing. In Proceedings of international joint conference artificial intelligence (IJCAI), pages 1-10, 2024.
[38] Rajshekhar Das, Yonatan Dukler, Avinash Ravichandran, and Ashwin Swaminathan. Learning expressive prompting with residuals for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3366-3377, 2023.
[39] Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch-based image retrieval, fine- ine-grained or not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2765-2775, 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[41] Yajing Zhai, Yawen Zeng, Zhiyong Huang, Zheng Qin, Xin Jin, and Da Cao. learning with cross-modal alignment for attribute-based person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6979-6987, 2024.
[42] He Li, Mang Ye, Ming Zhang, and Bo Du. All in one framework for multimodal re-identification in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17459-17469, 2024.
[43] Daeha Kim and Byung Cheol Song. Optimal transport-based identity matching for identity- invariant facial expression recognition. Advances in Neural Information Processing Systems, 35:18749-18762, 2022.
[44] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[45] Jason Altschuler, Jonathan Weed, and Philippe Rigollet. Near-linear time approxima- tion algorithms for optimal transport via sinkhorn iteration. Advances in neural information processing systems, 30, 2017.
[46] Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, and Yi-Dong Shen. Dual-path convolutional image-text embeddings with instance loss. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 16(2):1-23, 2020.
[47] Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, and Yanning Zhang. Vadclip: Adapting vision-languas models for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 2074-6082, 2024.
[48] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 2218-1226. 2021.
[49] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang. Dual encoding for video retrieval by text. IEEE Transactions Patlern Analysis and Machine Intelligence, 44(8):4065-4080 2021.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[51] Mang Ye, Jianbing Shen, David Crandall, Ling Shao, and Jiebo Luo. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 229-247. Springer, 2020.
13

---

[52] Chaoyou Fu, Yibo Hu, Xiang Wu, Hailin Shi. Tao Mei, and Ran He. Cm-nas: Cross-modalls neural architecture search for visible-infrared person re-identification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1 11823-1188 11832, 2021.
[53] Mang Ye, Weijian Ruan, Bo Du, and Mike Zheng Shou. Channel augmented joint learning for visible-infrared recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13567-13576, 2021.
[54] Yukang Zhang, Yan Yan, Yang Lu, and Hanzi Wang. Towards a unified middle modality learning for visible-infrared person re-identification. In Proceedings of the 29th ACM international conference on multimedia, pages 788-796, 2021.
[55] Mouxing Yang, Zhenyu Huang, Peng Hu, Taihao Li, Jiancheng Lv, and Xi Peng. Learning with twin noisy labels for visible-infrared person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14308-14317, 2022.
[56] Hanzhe Sun, Jun Liu, Zhizhong Zhang, Chengjie Wang, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Not all pixels are matched: Dense contrastive learning for cross-modality person re- identification. In Proceedings of the 30th ACM international conference multimedia, pages 5333-5341, 2022.
[57] Yiyuan Zhang, Yuhao Kang, Sanyuan Zhao, and Jianbing Shen. Dual-semantic consistency learning for visible-infrared person re-identification. IEEE Transactions on Information Foren- sics and Security, 18: 18:1554-1565, 2022.
[58] Yukang Zhang and Hanzi Wang. Diverse embedding expansion network and low-light cross- modality benchmark for visible-infrared person re-identification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2153-2162, 2023.
[59] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re- re-identification: A survey and outlook. IEEE transactions on pattern analysis and machine intelligence, 44(6):2872-288) 2021.
[60] Wonjae Kim, Bokyung Son, and Iddoo Kim. Vilt: Vision-and- languuuggg transformer without convolution region supervision. In International conference machine learning, pages 1894. PMLR, 2021.
[61] Chuhao Zhou, Jinxing Li, Huafeng Li, Guangming Lu, Yong Xu, and Min Zhang. Video-based infible-nfrarre person re- re-ideniification via style disturbance defense and dual interaction. In Proceedings of the st ACM International Conference on Multimedia, pages 46-55, 2023.
[62] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[63] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409 1556, 2014.
[64] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint urXiv:2010.11929 2020.
[65] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Chen Chen-Change Loy. Sketch me that shoe. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 799-807, 2016.
[66] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG), 35 2016.
[67] Alexander Hermans, Lucas Beyer, and Bastian Leibe. defense of the triplet loss for person re-identification. arXiv preprint trXiv:1103.077335 2017.
14

---

[68] Brandon Amos and J Zico Kolter. Opinet: Differentiable optimization as a layer in neural networks. International Conference Machine Learning, pages 136-145. PMLR, 2017.
[69] Yang Qin, Yingke Chen, Dezhong Peng, Xi Peng, Joey Tianyi Zhou, and Peng Hu. Noisy- correspondence learning for text-to-image person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-18, 2024.
[70] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900, PMLR, 2022.
[71] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022.
15

---

A Details about Optimal Transport
This section mainly introduces optimal transport and its corresponding algorithm, Sinkhorn-Knoph Let r {ar E 1
1) represents the probability simplex, where 1, is the dimensional vector ones. Given two probability simplex vectors a E m and 8 E nn and cost matrix C' R=x", the objective of OT is to seek the optimal transport plan P mapping a to B at the minimum cost:
44
dc(a, min (), PEU(a. U(a 3)= (P ER pl, a, P P Im 3},
where U(a, (3) denotes the transport polytope of a and B. i.e., the solution space of P. The above problem is to find optimal solution P in set of all possible joint probabilities of (X, Y), where X and Y represent random variables with marginal distribution a and [.
Generally, optimal transport (OT) problem is a linear programming problem that can theoretically be solved in polynomial time. However, the actual solving process. involves the square of anchor feature dimensions at all scales, requiring near-cubic complexity [68]. Therefore, consider optimizing the problem through iteration. This method optimizes the solving process by adding an entropy constraint to the OT problem, transforming Eq. 7 into a non-linear convex form with regularization term:
8) min PEU(a.
(C,P) (.P)+8E11 -EE(P)
(8)
( U(a, 3) = {PE ERRTT Rmxx Pl, = = PT, 1m Pm 33.
where hyper-parameter and E(P) Plog P regularization term is equivalent to introducing a prior knowledge: without considering the cost matrix C, the distribution of assignment matrix / is expected be as uniform as possible. Eq. is an entropy-rege regularrgu OT (EOT) problem and can be solved by Sinkhorn- Knopp algorithm 44 based on iterative updates of vectors. According the Lagrange Multiplier Method, the conditional extremum problem (i.e., Eq. g can be transformed into an unconditional extremum problem:
is the entropy of P. Introducing
da B) = min P\ ) + SE(P) + μ(PI, - a) + p(P 1m -3)
(9)
p
where, μ and p the Lagrange multipliers. If take its derivative and set it to O, find P*:
C 1 () eeppl Jexpl
p exp( -ompl-1
(10)
Let cx ezp(- () and K exp(- ) two constraint of Eq. needs to be met simultancously. Thus, one possible solution is to iterate enough times according to the following iteration formula:
(11)
Ky=-1
KT T²
Eq. 11 is called as Sinkhom-Knopp iteration. After z iterations, p can be obtained by the following equation:
P Diag(u) Diag(v)
(12)
It is worth noting that the process of updating and v (i.c., Eq. I1 alternately can be simplified to
single step: u² a/ ( K === - Furthermore, when all elements of a are positive, this single step can be further simplified as: u² < 1./K(3/K' u--1) where K = -iia((./a)) The overall flow of Sinkhorn-Knoo is described by Algorithm
B Details about Triplet Assignment Loss
B.1 Derivation for Gradient
This appendix provides some details gradient derivation. To simplify representation and analysis, we focus on single direction, following the approach [69), considering that sketch-to-image retrieval and image-to-sketch retrieval are symmetrical, Moreover, assume that there is only one
16

---

Algorithm The OT problem via Sinkhorn-Knopp
Require: Cost matrix C, weights a, 8 Require: Hyper-parameter 8, iterations
max iteration 2
Initialize K e cc/6 K Diag(1./a)) K, uo 1
2: while u changes or i is less than equal do
3: v² (3/ K u 1111)
4: i i-i+1
5: end while
6: Get optimal value u = u
7: 8" = 3//K" "K''" "t*
8: return optimal flow matrix P = Diag(u" K Diag(o")
paired image for each sketch in the mini-batch. Consequently, we can simplify TRL and TAL as shown below:
(i,, = [m - r 8 + f. ssl+
Lest S.)= = [mm d(r..s)) + d(fr, s:))l+, d(r,s) = "rr - 211-))(1-))) (1-3))1-epp(-55 exp((--)p(-
(13)
where d is the hyper-parameter of OT problem, f; and are the hardest negative sample and positive sample of the anchor sample 8;, respectively. These Iz-normalized features are embedded by the modality-specific models, i.e., fo,(-) and (-) Due to the truncation operation [x]]+. we only discuss the case of L > that could generate gradients. For TRL, the gradients to the parameters 0, and 0g are:
ahrt 0hmt 0s; 00,
0h 0hrtt dr. 0hri Of: Os, 00, 00, Dr, 00, Of, 00,-
(14)
Since the learning of normalized features can be viewed as the movement process of points on a unit hyperplane, we only consider the loss gradients with respect to ri. f. and s; are:
dCrl Or,
OCtrl = p. rit Os:
0Ctri Or.
8;+
84-
(15)
For our TAL, the gradients to the parameters 0, and 0, are:
L Ohra ds;
HLiat Oi: a0, Os 00," ae, Or, 00, Or, a0,
0hral 0Lut Ôr;
(16)
Since the result of crp(-- = is obtained through several iterations of LinkhornKnoop algorithm,
this part does not conduct gradients and can be simplified as a coefficient å exp( - E [0. 1]. Thus, the gradients for Tı, f; and are:
0Lul Ör;
dLual b4 (1 -7)8]8+ Oi,
- + (1 7)888
(17)
Thral Øs;
7(1 - ri) + (1 7))((F - -r))-
B.2
The Effectiveness of Triplet Assignment Loss
We have included visual analysis in Fig. which illustrates the convergence curve and sample distances during training. Figure (a) shows that conventional triplet loss converges prematurely. In contrast, our proposed triplet assignment loss exhibits higher volatility, reducing the risk of suboptimal local minima. Additionally, Figure 1(b) shows that a specific sketch sample (red box in the top left image) may have similar Euclidean distances to multiple RGB samples. The triplet assignment loss comprehensively considers the distribution of samples (red box the lower right image), offering broader possibilities for selecting the most relevant ones.
17

---

Buclabea dadamar
Optinad tnaagard (P")
Loss
Optmall
ICL 1")
A.qustrustt
Froch
(a) The convergence curves of different losses
(b) The distance distribution across modalities
Figure 4: The effectiveness analysis of Triplet Assignment Loss. Figure (b), the vertical axis represents RGB images, and the horizontal axis represents sketches. For each ID, 4 training examples are sampled, so the 4x4 cells on the diagonal represent positive sample pairs.
C Computational Complexity
Our OLTM achieves the trade-off between performance enhancement and computational complexity. To this end, we select several methods for comparing parameters, floating-poing operations (FLOPs), and frames per second (FPS), shown in the Tab. [5] below. The results show that OLTM gets remarkable performance while maintaining reasonable computational costs. The reason is that: I) only TCA module is required for inference; the Visual Question Answering(VQA) model is used during data processing.
Table 5: The number of parameters, FLOPs, and FPS of different methods, where bold indicates the best performance in this field and underline indicates the second-best performance. VI denotes visible-infrared person re- re-identificatio..
Methods
Field
Backbone
Reference
Paras(M)
FLOPs(G)
FPS
DDAG[53]
VI
ResNet50
ECCV*2020
95.6
5.2
13.2
CM-NAS[54]
ResNet50
ICCV"2021
24.5
5.2
14.8
CAJ[55]
VI
ResNet50
ICCV*2021
71.6
5.2
13.1
DEEN[56]
VI
ResNet50
CVPR 2023
89.3
13.8
7.7
CCSC[9]
Sketch
VIT
MM'2022
203.9
383.8
BDG[6]
Sketch
ResNet50
MM'2023
222.9
5.2
14.9
UNIReID/71
Unified
CLIP
149.6
9.3
8.2
OLTM (Ours)
Sketch
CLIP
181.9
6.2
11.7
D Additional Experimental Results
D.1 Experimental Setup
Implementation Details The dimensions of image and text features are set to 512. Within batch, we randomly select 8 identities, each comprising 4 images and sketches. Each image is associated with 9 fine-gra grained textual attributes. To ensure more reliable comparisons, the random seeds are all set 0. the Text Prompt Reasoning module, ConMLP consists of a stack of N identical MLPs, where mmp represents the trainable parameters. The iteration number of the Optimal Transport algorithm is In the Triplet Assignment Loss, the iteration number of the Optimal Transport algorithm is 50. The model is trained with the Adam optimizer, starting with a learning rate of le-5, decaying with a cosine scheduler. The model is implemented in PyTor on the RTX 4090 24GB GPU.
18

---

D.2 Experiment Result
Cross-style Retrieval Given the extensive Market-Sketch- IK dataset, where each person sketched by six different artists, notable variations exist across these sketches. Consequently, we devise this experiment to assess our model's resilience diverse artistic styles. Experimental setups involve sketches labeled as S1 to S6, each originating from different artists. Models are trained on sketches by specific artists and tested on sketches by others. And, "single query" denotes separate queries for sketches by different artists of the same individual, while "multi query" indicates queries combining multiple sketches of the same person. The specific details of these experiments are presented in Tab.
Table 6: Testing unseen styles. We report the mAP score.
a) single-query train and single-query test
b) multi-query train and multi-query
test
Sa
Ss.s
S
Ss...
Sa..
So
S5.6
Si...6
Ss..
S2...
train
train
S1
25.83
22.41
22.71
23.47
22.96
Si
S1.2
27.10
22.72
23.18
26.47
S1.2
31.82
33.68
39.49
Si...s
25.75
26.60
26.83
Si...s
34.30
36.49
S1...
32.60
30.07
S1...4
39.42
S
33.87
S1...6
<) single-query train and multi-query test
multi-query train and single-query lest
test
Sg
S5,6
S4...
Ss...
S2...
Se
S5.6
S
Ss...
S2..
train
train
51
29.17
30.35
34.65
34.49
S1.2
27.10
22.68
23.35
24.81
$1,2
31.23
34.34
41.05
S
26.97
25.13
31.96
S
34.52
37.65
S
29.99
27.79
$1..4
40.10
St...
31.96
Multi-query Setting Similar to 6 "multiiquery" involves combining multiple sketches of the same ID during both training and inference. Our paper employs straightforward fusion method by averaging the image features from multiple sketches. Tab. below provides a comparative analysis of various fusion strategies. The results demonstrate that the basic and simple fusion method achieves the best experimental performance.
Table 7: Performance comparison of different multi-query experimental methods.
Methods
mAP
Rank@1
Rank @5
Rank @ 10
Simple Fusion
62.55
69.48
90.36
95.18
Average Pooling
60.95
66.27
88.15
94.38
Non-local Attention
60.98
65.66
90.16
94.98
Overfitting Analysis To mitigate overfitting, we apply various data augmentation techniques, including random cropping, rotation, and style augmentation. Furthermore, validate the model's robustness and generalization, we conduct supplementary evaluation experiments on two large-scale datasets (SYSU-MMO1 and RegDB) for visible-infrared person re-ID, as shown in Tab. g The results demonstrate that our OLTM achieves comparable performance in the visible-infrared domain.
Parameter Analysis For the Triplet Assignment Loss proposed in work, we compute the sample distance using Eq. 18 Fig. 5 illustrates an analysis of the hyper-parameter y. can be observed that setting 7
0.3 yields the best performance in Rank-1 and mAP.
D DRR, St) (R, S.) (1 71 -., E(,,)
(18)
where F(R, = |R - S| denotes the Fuclidean distance between feature representations.
In Dynamic Consensus Acquisition, the cost matrix E IR* can be calculated for assignment, where the (i,j)-th element Ci,j represents the cost of assigning a feature to an atom. Concretely, to introduce certain priors as guidance (23], the enhanced features are used to learn the cost matrix through two fully connected layers initialized randomly. m is hyper-parameter that needs to be set. Fig. [6 analyzes the values of m, showing that the best performance in terms of Rank-1 and mAP achieved when 32.
19

---

Table Comparison results of our method on issble-infrared datasets, namely SYSU-MM01 and
RegDB. Market Sketch- respective task domains.
Market-Sketch- IK is a sketch dataset used for reference. "VI" and "Sketch" represent their
SYSU-MMO1
RegDB
Market-Sketch- IK
Methods
Domain
All Search
Indoor
Search
VIS to IR
TR to
VIS
Sketch to VIS
R-1
mAP
R-1
mAP
R-1
MAP
R-1
mAP
R-1
mAP
DDAG[53]
VI
54.8
53.0
61.0
68.0
69.3
63.5
68.1
61.8
11.2
12.1
CM-NAS[54]
VI
60.8
58.9
68.0
52.4
82.8
79.3
81.7
77.6
0.7
0.8
CAJ|55]
69.9
66.9
76.3
80.4
85.0
79.1
84.8
77.8
1.5
2.4
DART157]
68.7
66.3
72.5
78.2
83.6
75.7
82.0
73.8
7.8
5CLN
70.8
65.3
73.5
76.8
81.2
74.3
78.0
70.6
12.2
13.5
OLTM(Ours)
Sketch
70.6
68.6
76.2
80.4
84.8
77.2
83.9
75.5
36.8
38.4
Figure 5: Analysis of the hyperparameter
Figure 6: Analysis of the hyperparameter m
Qualitative Analysis Fig. 7 shows the Rank-5 results of OLTM and the baseline on the Market- Sketch- IK dataset. The left and right parts show the retrieval results of the baseline and OLTM, respectively. We can observe that OLTM can focus on more fine-grained discriminative information. such as bag and hat. In contrast, the baseline only considers global information matching, which leads to performance degradation.
Figure 7: The Rank-5 retrieval results under the multi-query setting on the Market-Sketch- 1K dataset are presented. On the left side the retrieval results of the baseline, and on the right side the retrieval results of OLTM. Green borders indicate successful retrieval of the target pedestrian, while red borders indicate incorrect results. Yellow boxes represent fine- fine-grained information.
20

---

E Analysis of VQA Models
E.1 The Setup the VQA Problems
Through extensive statistical analysis, we have formulated specific questions to obtain the corre- sponding fine-grained textual attributes for a given image. These nine questions are as follows:
1. What is the gender of this person?
2. Is this person with long short hair?
3. What the color of this person's shirt?
4. Is this person wearing long sleeves or short sleeves?
5. Is this person wearing pants a dress underneath?
6. What is the color of this person's 's lower garment?
7. Is this person carrying a backpack?
8. Is this person wearing hat?
9. Is this person wearing glasses?
E.2 Replaceability of VQA Model
The VQA model inherently substitutable. Essentially, any visual-language model which is capable of generating target attribute information from images serve alternative. We also use other VQA models to demonstrate their substitutability, as shown in the Tab. below.
Table Performance comparison of different VQA models.
Methods
mAP
R@1
R@5
R@10
BLIP 70
62.63
67.87
91.37
96.79
GIT 71
62.33
68.47
90.76
96.59
VILT (ours)
62.55
69.48
90.36
95.18
E.3
The Fine-grained Recognition Ability of VQA Model
The VQA model has the capability to describe fine-graines recognition information for the following reasons: 1) The VQA model generates detailed attributes about various aspects of the pedestrian target (e.g., hair, backpack, hat), rather than relying on complete descriptive sentences. 2) We provide visualization comparison, as shown in Fig. S This comparison demonstrates that using text attributes can guide effectively the attention of model.
Sketch Modality
Visihle Modality
Figure 8: Visualization of attention maps. The Green and Red box indicate presence and absence of text attribute guidance, respectively. The Orange box represents partial text attribute guidance (excluding head-related information).
21

---

E.4 Background Information
We conduct additional background evaluation on Market-Sketchh dataset based on our initial studies. Specifically, we formulate the question 'What is the background of this image?' to extract textual attributes about background. The extracted background details are illustrated in Fig. 9 However, the introduction of background information result in decrease of 2.81 1 in Rank-1 and 1.62 in mAP. This decline can be attributed the absence of corresponding background information in the sketches, which potentially interferes with the model's ' learning process.
male short hair yellow sleeves short sleeves shorts gray shorts with bag without hat with glasses bike
male short hair green sleeves short sleeves pants blue pants
with bag
without hat with glasses tree
Figure 9: The attributes generated by VQA model on RGB images. Red indicates background
information obtained from the question: "What is the background of this image?".
E.5 The Availability of Text Attributes
To verify the effectiveness of different text attributes, we have provided additional ablation experi- ments in Tab. bolloo.. The results show significant decrease in model performance after discarding several diddiissiin attributes (e.g.. color and gender) in sketches. As shown in Fig. I0 sketches convey gender-related information through factors like body shape, and the contrast between light and dark areas effectively highlights specific color details. The TPR module injects detailed information into modal interactions during training. This enables the model to focus these nuances autonomously, even without TPR during inference.
Table 10: Building on the original experimental setup, this comparison evaluates performance by removing the fine- mnaigrai textual attributes of gender, up color, and down color.
gender
up color
down color
mAP
Rank@1
Rank @5
Rank @ 10
62.55
69.48
90.36
95.18
v
v
62.28
68.07
89.96
95.18
/
61.64
67.67
89.76
94.98
61.35
67.07
89.16
94.78
Figure 10: Comparative analysis of RGB images and corresponding sketches. Gender-related factors sketches include body shape and hair, and the contrast highlights color details.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The abstract and introduction provide a concise overview of the research area, the novel approaches introduced, and the key contributions, ensuring that readers have clear understanding of what the paper aims to achieve and the significance of its findings.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the "Discussion" at the end of the paper, we discuss in detail the limitations of this work. However, our experiments revealed that when confronted with extremely vague sketch samples (i.e., those that human cannot discern features or match), the model's identification process deteriorates into random selection among multiple potential out- comes. Therefore, sketch Re-ID heavily depend on the quality of sketches. Enhancing the discriminability of sketches without incurring additional labor costs is topic worthy of further exploration future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
23

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All formulas and proofs in the paper are numbered and cross-referenced. The paper provides complete proofs of the formulas, and for those appearing the supplementary material, brief proof sketch is provided in the main text. The proofs in the paper are rigorously reasoned, adhere to accepted mathematical principles, and do not omit any critical steps. The theorems and lemmas relied upon in the proof's are appropriately referenced and cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results. The experimental section of the paper provides detailed descriptions of experimental setups, configurations, and other relevant details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
24

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b)
If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification: This paper does not currently provide open-access code, but it is planned to be made public in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc.
Beaisc//uuddes//oeeSubbee hwt.:suubbiiiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. cwcpppblcc/guieeesseecccpppseepe https//oooouummmsiioiisc
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies all the training and test details necessary to understand the results. It includes information such as data splits, hyperparameters, how they were chosen, the type of optimizer used, and any other relevant details regarding the training and testing procedures.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
25

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars suitably and correctly defined, nor does provide other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, the paper provides sufficient information the computer resources needed to reproduce the experiments (type of compute workers, memory).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wuurrp..ccppppac/aaaaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ggg//ppppp....oo////////a
26

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the potential positive societal impacts of the work. See
introduction 1
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
27

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets (e.g., code, data, models), used the paper, properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/coddemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
28

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "d2GUquulhI", "reviewer": "Reviewer_nGMK", "summary": "This article proposes a framework for pedestrian re identification in sketch images based on optimal transportation theory, which utilizes visual question answering pre training models to address the current issue of high text annotation costs and only focuses on global feature representation. This framework utilizes a visual question answering model to automatically generate text prompts, and extracts coarse-grained features through interaction between the text encoder and the visual encoder. By combining clustering methods based on optimal transportation theory with multi head attention mechanism, fine-grained features are extracted, achieving hierarchical and multi granularity alignment, and obtaining a relatively complete feature representation. In addition, a triplet loss function was designed that prioritizes regions with maximum local feature similarity, providing a more reasonable representation for measuring local feature distance. Through experiments on two public datasets, it can be proven that the pedestrian re identification and generalization ability of this model framework have been improved compared to previous model methods.", "questions": "(1) How is the number of learnable text prompt units added to a fixed text prompt vector determined?\n(2) Does the visual question answering model have the ability to describe feature information that truly possesses fine-grained recognition ability?\n(3) Is there a comparison of models with artificial text prompts in the experimental section? Will this method still have advantages in terms of re identification performance compared to models with manual text prompts that have been described with fine-grained features?\n(4) Regarding the use of optimal transportation theory to remove irrelevant information, is it possible to increase recognition accuracy if the text description can involve background information?", "limitations": "(1) The performance of visual question answering models on sketch images will limit their ability to extract features, so it is necessary to evaluate whether the VQA model can provide fine-grained local feature descriptions on sketch images;\n(2) There may still be misjudgments when performing re identification on two individuals with similar appearances but different identities, so text prompts may need to refer to more information, such as extracting content that is beneficial for judgment from discarded background information.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) Reduced the cost of text annotation. Due to the significant modal differences between sketched images and real images, it is almost inevitable to add an intermediate modality to assist information exchange, while text information is a very practical intermediate modality information. Due to the high cost of manual annotation, the introduction of visual question answering models to supplement intermediate modal information is highly effective.\n(2) Layered and multi granularity alignment helps to mine richer sample information. Traditional text prompts mainly focus on global features, while this model uses clustering methods based on optimal theory and multi head attention mechanisms to process fine-grained information after extracting coarse-grained features, and obtains valuable local information from it.\n(3) The triplet loss function takes into account the similarity differences between introduced local features. The traditional triplet loss function assigns the same weight to the differences between positive and negative samples, which may lead to inaccurate estimation of the local feature distance of the samples. The model framework proposed in this paper takes into account the distance difference between regions corresponding to fine-grained features, providing a more reasonable measurement method.", "weaknesses": "(1) The calculation cost is relatively high. Due to the introduction of a new visual question answering model and the addition of multiple attention mechanism structures, the computational resources required by the model are significantly increased compared to previous models.\n(2) Overfitting is prone to occur. During the experiment, it was found that due to the small sample size of the sketch dataset and the rich granularity and depth of the model's feature extraction, overfitting is prone to occur, which can also affect the model's generalization ability.\n(3) The accuracy of text prompts is greatly affected by image quality. The text prompts designed in this article have some fixed prompts and some learnable units. Due to the accuracy of text information directly affecting the quality of feature extraction, the performance of visual question answering pre training models on sketch image modalities is easily affected by different styles of images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Computational complexity.**\n\n**A1:** Please refer to **[Response to nGMK:W1, edWJ:W6, beT2:W1]** in **Common Response.**\n\n**W2: Overfitting.**\n\n**A2:** We sincerely appreciate your valuable comments regarding overfitting, as this is also a concern for us. Consistent with the approaches in [1] and [2], we validate our model on two publicly available benchmark datasets, PKU-Sketch and Market-Sketch-1K, to ensure fair comparisons. To mitigate overfitting, we apply various data augmentation techniques, including random cropping, rotation, and style augmentation. Furthermore, to validate the model's robustness and generalization, we conduct supplementary evaluation experiments on two large-scale datasets (SYSU-MM01 and RegDB) for visible-infrared person re-ID, as shown in **Table 1 of PDF**. The results demonstrate that our OLTM achieves comparable performance in the visible-infrared domain.\n\n**W3: Image quality.**\n\n**A3:** Thank you for your insightful comments.\n1) It should be clarified that the VQA model is only applied to RGB images, not sketch. \n2) In fact, image quality affects not only the attributes generated by VQA, but also the performance of all visual tasks. Therefore, in addition to several data augmentation operations (e.g., random rotation, flipping, and style augmentation), we have also implemented the following strategies: 1）In contrast to the conventional method of directly generating a complete descriptive sentence，we use a \"divide-and-conquer\" strategy. By employing multiple questions to capture specific details (e.g., hair and shirt color), we effectively reduce potential noise in the text prompts. 2) Fixed attribute tokens are dynamically embedded into learnable prompts. In addition, applying various loss functions guides the model to prioritize reliable and valuable information. 3) To further refine fine-grained text embeddings, a \"bin\" mechanism is added to categorize potentially ambiguous content during Dynamic Consensus Acquisition (as analyzed in Sec. 4.2).\n\n**Q1: The number of learnable text prompts.**\n\n**A1**: Thank you for your question. As analyzed in Section 4.2, we extract $k$ text attributes $\\\\{att_1, att_2,..., att_k \\\\}$ using CLIP tokenizer to derive text tokens $a_i=Tokenizer(att_i)$, where each $a_i$ contains $n$ fixed numbers of tokens. Subsequently, $a_i$ preserves valid tokens, and is fed into CLIP token encoder to obtain the fixed attribute tokens $a_i \\in{\\mathcal{R}^{m\\times{c}}}$. Here each word can be represented by $m$ valid tokens of dimension $c$ [3]. Those fixed parts are uniformly embedded into $l$ learnable prompts of dimension $c$, and finally we obtain final text description $q$. In our implementation details, the value of $k$ is $9$, and we set $n=77$ based on CLIP's fixed configuration. Thus the number of learnable text prompt is $n-k\\times{m}$.\n\n**Q2: The fine-grained recognition ability of VQA model.**\n\n**A2:** Thank you for your valuable comment. The visual question answering (VQA) model has the capability to describe fine-grained recognition information for the following reasons: 1) The VQA model generates detailed attributes about various aspects of the pedestrian target (e.g., hair, backpack, hat), rather than relying on complete descriptive sentences. 2) We provide a visualization comparison, as shown in **Figure 4 of PDF**. This comparison demonstrates that using text attributes can guide effectively the attention of model.\n\n**Q3: The performance comparison with the handcrafted prompts.**\n\n**A3:** Thank you for pointing out the incompleteness in our ablation study. Our method remains competitive compared to handcrafted prompts mechanism[4], as shown in Table 4 below. The performance improvements can be attributed to the following reasons: 1) We introduce a dynamic learnable prompt mechanism without requiring additional expert knowledge. This enhances the model's adaptability and robustness. 2) VQA is more flexible compared to handcrafted prompts. It can generate more detailed information (e.g., glasses, hats) than what is included in handcrafted prompts.\n\n*Table 4: Comparison of prompt setting methods. \"Handcrafted\" and \"VQA\" denote manually annotated and VQA generated text attributes, respectively. \"Template\" represents the sentence template defined by experts. \"Prompt\" denotes the learnable text prompts.*\n| Handcrafted | VQA | Template | Prompt | mAP   | R@1   | R@5   | R@10  |\n|-|-|-|-|-|-|-|-|\n| ✓ | - | ✓ | - | 61.46 | 68.07 | 89.96 | **96.79** |\n| ✓ | - | - | ✓ | 61.81 | 67.47 | **90.56** | 95.78 |\n| - | ✓ | ✓ | - | 61.76 | 65.46 | 90.16 | 96.18 |\n| - | ✓ | - | ✓ | **62.55** | **69.48** | 90.36 | 95.18 |\n\n**Q4: Background information.**\n\n**A4:** Thanks for your constructive advice. This motivates us to investigate the feasibility of incorporating background information into text descriptions. Accordingly, we conduct additional background evaluation on Market-Sketch-1k dataset based on our initial studies. Specifically, we formulate the question **'What is the background of this image?'** to extract textual attributes about background. The extracted background details are illustrated in **Figure 2 of PDF**. However, the introduction of background information result in **a decrease of 2.81 in Rank-1 and 1.62 in mAP**. This decline can be attributed to the absence of corresponding background information in the sketches, which potentially interferes with the model's learning process.\n\n[1]Pang L, et al. \"Cross-domain adversarial feature learning for sketch re-identification\" *ACMMM* 2018.\n\n[2]Lin K, et al. \"Beyond Domain Gap: Exploiting Subjectivity in Sketch-Based Person Retrieval\" *ACMMM* 2023.\n\n[3]Radford A, et al. \"Learning transferable visual models from natural language supervision\" *ICML* 2021.\n\n[4]Lin Y, et al. \"Improving person re-identification by attribute and identity learning. *Pattern recognition* 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hmeUv9HyYr", "reviewer": "Reviewer_edWJ", "summary": "This paper proposes an optimal transport-based labor-free text prompt network for sketch re-identification. The authors address two primary challenges: the expense of text annotation and cross-modal interaction, leveraging generated text attributes for multi-granularity modal alignment. The experimental results on two datasets achieve significant improvements.", "questions": "Major concerns: see Weakness.\nMinor concerns:\na)\tIn 4.1, the inference stage uses Rcls and Scls instead of r and s; the bolding of the formula needs to be checked.\nb)\tAdd detailed descriptions of the cross-style retrieval experiment in the supplementary materials.", "limitations": "The author mentions that their proposed framework effectively tackles challenges like occlusion and perspective but still encounters misjudgments with highly similar but distinct individuals. I consider this a critical issue for collective resolution in advancing re-identification tasks. My positive evaluation of the research remains unchanged. I will, however, follow the authors' work and look forward to potential solutions for these challenges.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper solves a practical problem and proposes a framework setting corresponding to the sketch re-identification task. The experimental results have effectively verified the effectiveness of this setting. If all the data and protocols are available, I believe it will be valuable for the research community.", "weaknesses": "The paper is interesting and the topic authors discussed is very promising, but I feel there are some doubts:\n\n1. The author suggests that the classic triplet loss may lead to inaccurate estimates of sample distances and potentially result in suboptimal local minima. The proposed Triplet Assignment Loss aims to address this issue. How can this viewpoint be substantiated from experiments?\n\n2. Comparisons with the most related method [1] are missing in TABLE 1 and TABLE 2.\n\n3. Do the transformer blocks in feature enhancement module for different modalities share weights?\n\n4. The description of pedestrians comes from VQA's answers to 9 questions. These questions typically cover details like color and gender. However, sketches used in testing often lack such explicit information. How does the model then prioritize these aspects?\n\n5. The experimental results in Table 1 include \"multiple-query\". However, the paper does not provide a description of how this is achieved.\n\n6. The paper compares the structural and performance differences between OLTM and other methods. However, it does not discuss whether the additional structural design increases the computational burden. Including relevant cost comparison experiments would greatly enhance the study.\n\n7. Is the VQA model irreplaceable? I did not come across a statement like \"VQA model is optional\" in the paper.\n\n[1] Li H, Ye M, Zhang M, et al. All in One Framework for Multimodal Re-identification in the Wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 17459-17469.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Triplet Assignment Loss.**\n\n**A1:** Thank you for your inquiry. We have included visual analysis in **Figure 1 of PDF**, which illustrates the convergence curve and sample distances during training. **Figure 1(a)** shows that conventional triplet loss converges prematurely. In contrast, our proposed triplet assignment loss exhibits higher volatility, reducing the risk of suboptimal local minima. Additionally, **Figure 1(b)** shows that a specific sketch sample (red box in the top left image) may have similar Euclidean distances to multiple RGB samples. The triplet assignment loss comprehensively considers the distribution of all samples (red box in the lower right image), offering broader possibilities for selecting the most relevant ones.\n\n**W2: Reference.**\n\n**A2:** We appreciate the reviewer’s recommendation. Inspired by this valuable work, we have cited this reference into Sec. 2 of the manuscript: \"... fine-grained interaction. **Furthermore, Li et al. [1] first attempt to conduct in-depth research on zero-shot multimodal ReID through a large foundational model.** In this paper, we ...\". However, since the source code of this reference is not available and it mainly focuses on zero-shot learning, we cannot make comparsion with it in experiments.\n\n**W3: Weights sharing.**\n\n**A3:** Thank you for your question. As stated in Sec. 4.3, Transformer blocks for different modalities do not share weights in feature enhancement module. The cross-modal interaction component in front of this module shared weights, facilitating the extraction of common features between modalities. Consequently, this module enables each modality to autonomously refine its representation. This strategy ensures precise adjustments according to specific needs and enhances the overall performance.\n\n**W4: The availability of text attributes.**\n\n**A4:** Thank you for pointing out this comment. To verify the effectiveness of different text attributes, we have provided additional ablation experiments in Table 5 below. The results show a significant **decrease in model performance** after discarding several hard-distinguished attributes (e.g., color and gender) in sketches. As shown in **Figure 3 of PDF**, sketches convey gender-related information through factors like body shape, and the contrast between light and dark areas effectively highlights specific color details. The TPR module injects detailed information into modal interactions during training. This enables the model to focus on these nuances autonomously, even without TPR during inference.\n\n*Table 5: The experiment results on various text attributes. \"Gender\", \"Shirt\" and \"Pants\" indicate whether the attribute includes gender, shirt color and pants color.*\n| Gender | Shirt  | Pants  | mAP | Rank@1 | Rank@5 | Rank@10 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| ✓ | ✓ | ✓ | 62.55 | 69.48 | 90.36 | 95.18 |\n| - | ✓ | ✓ | 62.28 | 68.07 | 89.96 | 95.18 |\n| - | - | ✓ | 61.64 | 67.67 | 89.76 | 94.98 |\n| - | - | - | 61.35 | 67.07 | 89.16 | 94.78 |\n\n**W5: Multi-query.**\n\n**A5:** Thank you for your question regarding the experimental setup. Similar to [2], \"multi-query\" involves combining multiple sketches of the same ID during both training and inference. Our paper employs a straightforward fusion method by averaging the image features from multiple sketches. Table 6 below provides a comparative analysis of various fusion strategies. The results demonstrate that the basic and simple fusion method achieves the best experimental performance.\n\n*Table 6: Performance comparison of different multi-query experimental methods.*\n| Methods | mAP | Rank@1 | Rank@5 | Rank@10 |\n| ---- | ---- | ---- | ---- | ---- |\n| Simple Fusion | 62.55 | 69.48 | 90.36 | 95.18 |\n| Average Pooling | 60.95 | 66.27 | 88.15 | 94.38 |\n| Non-local Attention | 60.98 | 65.66 | 90.16 | 94.98 |\n\n**W6: Computational complexity.**\n\n**A6:** Please refer to **[Response to nGMK:W1, edWJ:W6, beT2:W1]** in **Common Response**.\n\n**W7: Replaceability of VQA model.**\n\n**A7:** Please refer to **[Response to edWJ:W7, beT2:Q1]** in **Common Response**.\n\n**Q1: Minor concerns.**\n\n**A1:** We apologize for this typing mistake. We have revised it in the manuscript and added the following note to the supplementary material: *\"This section provides a detailed explanation of the cross-style retrieval. Given the extensive Market-Sketch-1K dataset, where each person is sketched by six different artists, notable variations exist across these sketches. Consequently, we devise this experiment to assess our model's resilience to diverse artistic styles. Experimental setups involve sketches labeled as S1 to S6, each originating from different artists. Models are trained on sketches by specific artists and tested on sketches by others. And, \"single query\" denotes separate queries for sketches by different artists of the same individual, while \"multi query\" indicates queries combining multiple sketches of the same person.\"*\n\n[1]Li H, Ye M, Zhang M, et al. \"All in One Framework for Multimodal Re-identification in the Wild\" *CVPR* 2024.\n\n[2]Lin K, Wang Z, Wang Z, et al. \"Beyond Domain Gap: Exploiting Subjectivity in Sketch-Based Person Retrieval\" *ACMMM* 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzsWZd6kZK", "reviewer": "Reviewer_Bynq", "summary": "This paper focuses on sketch based person ReID. Especifically, a labor free text method with OT is proposed, which achieve a large performance improvement on two public databases. Overall, I think the idea of this paper is interesting and the experiments show its superiority.", "questions": "1. In the related works, there is a subsection on optimal transport. What is the relationship of [26]/[27] and OT.\n2. On Line 143 “represents” should be modified to “represent”. On page 6, P* should be bold, be consistent to above.\n3. On line 185, authors argue that the proposed strategy would not introduce any noise. So, how to ensure it?\n4. In eq.(7), it is better to give more description about how to calculate the P*.", "limitations": "The author has addressed the limitation.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is interesting. For instance, authors introduce the text without any labor cost to handle the modal-gap removal. Also, the OT is used to get interested local text representation.\n2. The performance on two datasets achieves remarkable improvement.", "weaknesses": "1. Some grammar errors should be corrected.\n2. More detailed analysis on OT and TPR should be provided.\n3. From the paper, I don't quite understand the settings of the parameters. In the proposed method, what is the mean of $\\alpha$ and $\\beta$? What is the relationship between them and cost matric $C$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1, Q2: Minor issues.**\n\n**A:** Thanks for your nice comment. We have identified all errors and corrected them in the manuscript.\n\n**W2, W3, Q4: Optimal transport.**\n\n**A:** Thanks for your constructive comment. Due to space constraints, a detailed description of optimal transport (OT) and matrix $\\boldsymbol{P^\\ast}$ can be found in the supplementary materials. Essentially, the OT problem resembles a \"supplier-demander\" issue: $\\boldsymbol{\\alpha}$ denotes the quantity of goods each supplier can provide, $\\boldsymbol{\\beta}$ denotes the quantity of goods each demander requires, and the cost matrix $\\boldsymbol{C}$ indicates the expense incurred when each supplier supplies one unit of goods to each demander. The objective is to find the assignment solution that minimizes costs, yielding the optimal transport matrix $\\boldsymbol{P^\\ast}$. We add the description about how to calculate the $\\boldsymbol{P^\\ast}$:\n\n*For an input positive pair $(R\\_i, S\\_i)$ in a mini-batch $x$, the feature representations obtained through model inference are denoted as $f(R\\_i)$ and $f(S\\_i)$. If the sets of all sample features from different modalities in $x$ are treated as two discrete distributions, their alignment can be considered an optimal transport problem. The cost matrix $\\boldsymbol{\\hat{C}}$ is derived from pairwise feature similarities: $\\boldsymbol{\\hat{C}}\\_{i,j}=[f(R\\_i)^{\\top}f(S\\_j)]\\_+$. We aim to acquire the optimal transport matrix $\\boldsymbol{P}^{\\ast}$ with the least amount of cost, where $\\boldsymbol{P}^{\\ast}\\_{i,j}$ represents the assignment weight of $(R\\_i, S_j)$ obtained after balancing the overall distribution. TAL can be represented based on triplet loss as the weighted sum of the original distance and the optimal assignment distance, dynamically updated at a certain rate $\\gamma$:*\n\n\\begin{array}{c}\n    \\mathcal{L}\\_{tal}(R\\_i, S\\_i) = [m-D(R\\_i, S\\_i)+D(R\\_i, \\hat{S}\\_h)]\\_{+} + [m-D(R\\_i, S\\_i)+D(\\hat{R}\\_h, S\\_i)]\\_{+},    \\\\\\\\\n    D(R\\_i, S\\_i) = \\gamma E(R\\_i, S\\_i) + (1-\\gamma)(1-\\boldsymbol{P}^{\\ast}\\_{i,i})E(R\\_i, S\\_i)\n\\end{array}\n\n*where  $\\hat{R}\\_h=argmax\\_{R\\_j\\neq{R\\_i}}D(R\\_j,S\\_i)$  and  $\\hat{S}\\_h=argmax\\_{S\\_j\\neq{S\\_i}}D(R\\_i,S\\_j)$ are the most similar negatives in $x$ for $(R\\_i, S\\_i)$, $[x]\\_{+}=\\max(x,0)$,  and $E(R\\_i, S\\_i)=\\Vert{f(R\\_i)-f(S\\_i)}\\Vert\\_2$ denotes the Euclidean distance between feature representations.*\n\n**Q1: The relationship between Reference [26]/[27].**\n\n**A:** Both papers [26-27] utilize optimal transport to address inter-modal distribution alignment in re-ID tasks. Ling et al. [26] select an optimal transport strategy and assign high weights to pairs with smaller intra-identity variation. Zhang et al. [27] first attempt to find an optimal transport matrix to re-weight the distance of different local parts in re-ID. These two papers inspire us to apply optimal transport theory for exploring more discriminative feature representations and establishing more reasonable standards for measuring sample distances in re-ID.\n\n**Q3: Noise.**\n\n**A:** Thank you for pointing out the ambiguity in this sentence. Compared to fixed handcrafted prompts, our dynamic knowledge learning mechanism does not rely on expert knowledge. This strategy can mitigate **additional noise caused by inaccurate sentence templates**, since our prompt include adaptable learnable components constrained by various loss constraints. We have revised it in the manuscript: *\"This integration introduces a dynamic knowledge learning mechanism that reduces noise introduction compared to handcrafted prompts, while enhancing flexible interaction across modalities and boosting the transferability of text embeddings.\"*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noZRS16NY5", "reviewer": "Reviewer_beT2", "summary": "This article extends image-based person and vehicle reid task to sketch field. First, authors introduce a novel sketch-based reid framework named OLTM, which utilizes text information to achieve modal alignment. Additionally, in sketch-based reid, authors apply the VQA model to generate textual attributes of persons, thus avoiding costly annotation efforts. Furthermore, authors introduce a novel triplet alignment loss to enhance distance calculation.", "questions": "1. Why is the ViLT-based VQA model used instead of others, like Bert-based model?\n\n2. Is there an error in Eq. 7? After checking the supplementary materials, my understanding is that the optimal transport matrix P* is a weight matrix, not a loss value.\n\n3. Some details are not clear to me. It is unclear whether the process of obtaining text attributes occurs during model training or after pre-processing. According to the framework Figure 2, this process seems to be executed during the model training process, which can result in complex model calls.\n\n4. The experiment doesn’t mention whether it is compared with existing image-based re-identification methods.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors use the VQA model to generate attribute descriptions about pedestrians, effectively reducing the cost of manual labeling.\n\n2. The method utilizes optimal transport theory and text prompt to improve model performance and designs a new triplet alignment loss.\n\n3. The experimental results are convincing compared to the state-of-the-arts.", "weaknesses": "1. Compared to CNN-based methods, using CLIP as a Backbone may suffer from high computation complexity.\n\n2. Some minor issues: the symbols for K, Q, and V in Eq. 4 and 5 should be bolded.\n\n3. The names in the paper and the supplementary material are inconsistent, i.e., HTL/TRL.\n\n4. Figure 2 in the supplementary material is rather blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Computational complexity.**\n\n**A:** Please refer to **[Respones to nGMK:W1, edWJ:W6, beT2:W1]** in **Common Response**.\n\n**W2, W3, W4: Minor issues.**\n\n**A:** Thank you for your thorough review and for highlighting these minor issues. We have revised the manuscript to address the mentioned issues.\n\n**Q1: Replaceability of VQA model.**\n\n**A:** Please refer to **[Response to edWJ:W7, beT2:Q1]** in **Common Response**.\n\n**Q2: $\\boldsymbol{P}^\\ast$.**\n\n**A:** Thank you for bringing up this question. Indeed, $P^\\ast$ represents a weight matrix. Here is the corrected formula:\n\n\\begin{array}{c}\n    \\mathcal{L}\\_{tal}(R\\_i, S\\_i) = [m - D(R\\_i, S\\_i) + D(R\\_i, \\hat{S}\\_h)]\\_{+} + [m - D(R\\_i, S\\_i) + D(\\hat{R}\\_h, S\\_i)]\\_{+}, \\\\\\\\\n    D(R\\_i, S\\_i) = \\gamma E(R\\_i, S\\_i) + (1 - \\gamma)(1 - \\boldsymbol{P}^{\\ast}\\_{i,i})E(R\\_i, S\\_i)\n\\end{array}\n\n*where, $\\boldsymbol{P}^{\\ast}$ is the optimal transport matrix with the least amount of cost, and $\\boldsymbol{P}^{\\ast}\\_{i,i}$ represents the assignment weight of $(R\\_i, S\\_i)$ obtained after balancing the overall distribution.*\n\n**Q3: Details of text attribute acquisition.**\n\n**A:** Thank you for raising this question. The text attributes acquisition occurs **during the data processing stage**, and does not lead to complex model calls during training. We have enhanced clarity in our manuscript to prevent such ambiguities.\n\n**Q4: Experimental comparisons.**\n\n**A:** Thank you for your constructive feedback. Based on task relevance, our adopted methods are all image-based re-identification methods, categorized into visible-infrared and sketch-based methods. The former include DDAG [54], CM-NAS [55], CAJ [56], MMN [57], DART [58], DCLNet [59], DSCNet [60], and DEEN [61]. The latter comprises BDG [6] and UNIReID [7]. We have already given the detailed descriptions of these experimental methods in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' valuable time in providing constructive feedback. We have thoroughly reviewed the comments and made the necessary responses and corrections. If our responses do not fully address the reviewers' questions, we are open to further discussion.

We are honored that the reviewers have recognized the practicality and value of our task **(edWJ, Bynq)**, as well as the key contributions of this paper:

- Utilization of a visual question answering model to acquire text attribute, thereby reducing manual annotation costs **(nGMK, Bynq, beT2)**;
- Implementation of hierarchical and multi-granularity alignment **(nGMK, edWJ)**;
- Incorporation of optimal transport theory and text prompts **(nGMK, Bynq, beT2)**;
- Introduction of a novel triplet alignment loss **(nGMK, beT2)**;
- Effective validation of experimental results **(Bynq, edWJ, beT2)**.

Considering that several reviewers have expressed similar concerns about our method, we provide a comprehensive answer for all reviewers here.

**[Response to nGMK:W1, edWJ:W6, beT2:W1]: Computational complexity.**
Thanks for your valuable comment. Our OLTM achieves the trade-off between performance enhancement and computational complexity. To this end, we select several methods for comparing parameters, floating-point operations (FLOPs), and frames per second (FPS), as shown in the Table 2 below. The results show that OLTM gets remarkable performance while maintaining reasonable computational costs. The reason is that: 1) only TCA module is required for inference; 2) the VQA model is used during data processing.

*Table 2: The number of parameters, FLOPs, and FPS of different methods. VI denotes visible-infrared person re-identification.*
| **Methods** | **Field** | **Backbone** | **Reference** | **Paras(M)** | **FLOPs(G)** | **FPS** |
|-------------|-----------|--------------|---------------|--------------|--------------|---------|
| DDAG[53]    | VI        | ResNet50     | ECCV'2020     | 95.6         | 5.2      | 13.2    |
| CM-NAS[54]  | VI        | ResNet50     | ICCV'2021     | 24.5         | 5.2      | 14.8|
| CAJ[55]     | VI        | ResNet50     | ICCV'2021     | 71.6         | 5.2      | 13.1    |
| DEEN[56]    | VI        | ResNet50     | CVPR'2023     | 89.3         | 13.8         | 7.7     |
| CCSC[9]     | Sketch    | ViT          | MM'2022       | 203.9        | 383.8        | -       |
| BDG[6]      | Sketch    | ResNet50     | MM'2023       | 222.9        | 5.2      | 14.9    |
| UNIReID[7]  | Unified   | CLIP         | CVPR'2023     | 149.6        | 9.3          | 8.2     |
| OLTM (Ours) | Sketch    | CLIP         | -             | 181.9        | 6.2          |  11.7   |

**[Response to edWJ:W7, beT2:Q1]: Replaceability of VQA model.**
Thank you for bringing this issue to our attention. The VQA model is inherently substitutable. Essentially, any visual-language model which is capable of generating target attribute information from images can serve as an alternative. We also use other VQA models to demonstrate their substitutability, as shown in the Table 3 below.

*Table 3: Performance comparison of different VQA models.*
| **Methods** | **mAP** | **Rank@1** | **Rank@5** | **Rank@10** |
|-------------|---------|---------|---------|----------|
| Vilt        | 62.55   | 69.48   | 90.36   | 95.18    |
| Blip[1]     | 62.63   | 67.87   | 91.37   | 96.79    |
| GIT[2]      | 62.33   | 68.47   | 90.76   | 96.59    |


**Note: A PDF file** is attached in the common response. It contains **all the new figures** we used in the rebuttal phase.

[1]Li J, Li D, Xiong C, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation" *ICML* 2022.

[2]Wang J, Yang Z, Hu X, et al. "Git: A generative image-to-text transformer for vision and language" *arXiv preprint* 2022.

Meta-review: This paper introduces a novel framework for  Sketch Re-ID. After the author-reviewer discussion period, this paper received an average score of 5.75, with one negative score of 4. Since the reviewer with such a negative score did not engage in the discussion, I have checked the author's rebuttal carefully. I agree that the author has addressed the concerns.

Since the AC recognizes the overall quality of this paper, it is recommended to accept it to the NeruIPS'24 conference.
Decision: Accept (poster)