Optimal Transport-based Labor-free Text Prompt Modeling for Sketch Re-identification

Authors: rui  li
tingting  ren
jie  wen
jinxing  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:11:31

Abstract: Sketch Re-identification (Sketch Re-ID), 
which aims to retrieve target person from an image gallery based on a sketch query, is crucial for criminal investigation, law enforcement, and missing person searches. 
Existing methods aim to alleviate the modality gap by employing semantic metrics constraints or auxiliary modal guidance. However, they incur expensive labor costs and inevitably omit fine-grained modality-consistent information due to the abstraction of sketches.
To address this issue, this paper proposes a novel $\textit{Optimal Transport-based Labor-free Text Prompt Modeling}$ (OLTM) network, which hierarchically extracts coarse- and fine-grained similarity representations guided by textual semantic information without any additional annotations. 
Specifically, multiple target attributes are flexibly obtained by a pre-trained visual question answering (VQA) model. Subsequently, a text prompt reasoning module employs learnable prompt strategy and optimal transport algorithm to extract discriminative global and local text representations, which serve as a bridge for hierarchical and multi-granularity modal alignment between sketch and image modalities.
Additionally, instead of measuring the similarity of two samples by only computing their distance, a novel triplet assignment loss is further proposed, in which the whole data distribution also contributes to optimizing the inter/intra-class distances. Extensive experiments conducted on two public benchmarks consistently demonstrate the robustness and superiority of our OLTM over state-of-the-art methods.

Main paper: 
 limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In the "Discussion" at the end of the paper, we discuss in detail the limitations of this work. However, our experiments revealed that when confronted with extremely vague sketch samples (i.e., those that human cannot discern features or match), the model's identification process deteriorates into a random selection among multiple potential outcomes. Therefore, sketch Re-ID heavily depend on the quality of sketches. Enhancing the discriminability of sketches without incurring additional labor costs is a topic worthy of further exploration in future research.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem statement: 
 To ensure clarity, we represent the gallery containing m images I as G = {I i , y i } m i=1 and the query set containing n sketches S as S = {S j , y j } n j=1 , where y ∈ {1, . . . , C} are the identity labels for C distinct pedestrian entities. Notably, each entity may include multiple images and sketches. The goal of Sketch Re-ID is to retrieve pedestrian images from the gallery G that match one or multiple given sketch. Like [6] , there exist two types of query methods: single sketch query and multiple sketches query. This section will use single sketch query as an example, and the same applies to multiple sketches query. Formally, we define a matching function M : S × G → R n×m that assigns a similarity score to each pair (I i , S j ). The objective is to learn a function M such that for any sketch S j and image I i , M(I i , S j ) > M(I k , S j ) if y i = y j and y k ̸ = y j , (1) where I i and I k are images from the gallery set, and y i and y k are their respective identity labels.Optimal Transport is a mathematical theory that focuses on finding an efficient solution between two probability distributions, minimizing the cost of transporting one distribution into another. We briefly review the theoretical derivation of optimal transport. Let Γ r : the probability simplex, where 1 r is the r-dimensional vector of ones. Given two probability simplex vectors α ∈ Γ m and β ∈ Γ n and a cost matrix C ∈ R m×n , the objective of OT is to seek the optimal transport plan P * mapping α to β at the minimum cost:= {x ∈ R r + |x ⊤ 1 r = 1} representsEQUATIONwhere U (α, β) denotes the transport polytope of α and β, i.e., the solution space of P . The above problem is to find optimal solution P * in a set of all possible joint probabilities of (X, Y ), where X and Y represent random variables with marginal distribution α and β.Eq. 8 indicates that OT is a linear programming problem which is theoretically solvable in polynomial time, but its complexity becomes prohibitively high as the feature dimension increases [43] . To this end, Sinkhorn algorithm [44] adopts an iterative strategy to obtain the optimal solution P * = Diag(u)KDiag(v) with near-square complexity [45] . u and v can be solved through alternately iterating the following two equations: u (z) = α/(Kv (z-1) ) and v (z) = β/(K ⊤ u (z) ), where K = exp(C/ϵ), ϵ is the regularization coefficient and z is the iterations (cf. Appendix). Since this method integrates the importance of all features when solving the optimal solution, it can analyze the overall data distribution.4 The Proposed Method 4.1 Overall Architecture Fig. 2 provides an overview of the OLTM architecture. The image and text encoders discussed in this paper are based on CLIP [40] , and any language-visual model utilizing a Transformer architecture may also be employed. Notably, the image encoders utilized for both images and sketches employ shared weights to ensure the mapping of features into a unified semantic space. For an input RGB image, we obtain embeddings R = {R cls , r 1 , . . . , r p } ∈ R (p+1)×d through the image encoder, where p is the number of non-overlapping patches, R cls and R local = {r i } p i=1 represent d-dimensional global and local features, respectively. Similarly, the embeddings of a sketch can be represented as S = {S cls , S local } = {S cls , s 1 , . . . , s p } ∈ R (p+1)×d . Firstly, to provide reasonable text semantic guidance, Text Prompt Reasoning (TPR) generates attribute descriptions about pedestrians based on the RGB image, and obtains the textual embeddings T = {T eos , T local } through prompt learning. Then, TPR extracts the fine-grained consensus X from T local through clustering. Subsequently, Textinjected Coarse-grained Alignment (TCA) module embeds global contextual information T eos into visual features R cls and S cls . Meanwhile, Consensus-guided Fine-grained Interaction (CFI) module utilizes X to address fine-grained semantic misalignment between R local and S local . Additionally, due to the significant differences between sketches and RGB images, Euclidean distance between independent samples may ignore the influence of overall sample distribution. Thus, we introduce a more comprehensive distance measurement method and propose triplet assignment loss L tal .During training, all these modules will be jointly optimized through identity loss [46] L id and L tal : L OLT M = L id + ηL tal , where η is a scaling factor. During inference, only R cls and S cls are used to match queries for practical application requirements.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: All formulas and proofs in the paper are numbered and cross-referenced. The paper provides complete proofs of the formulas, and for those appearing in the supplementary material, a brief proof sketch is provided in the main text. The proofs in the paper are rigorously reasoned, adhere to accepted mathematical principles, and do not omit any critical steps. The theorems and lemmas relied upon in the proofs are appropriately referenced and cross-referenced. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
text prompt reasoning: 
 Significant image differences and inherent abstract nature, cause semantic misalignment during knowledge acquisition, severely impacting model's reasoning and generalization capabilities. To address this issue, TPR introduces intermediate modality to guide alignment between modalities without additional costs. Moreover, TPR employs a dynamic consensus acquisition strategy to enhance the discriminative power of local text features.Text Attribute Generation Sketches, unlike conventional Re-ID tasks, are vulnerable to subjective emotions and drawing skills of artists, leading to a lack of detailed information crucial for model learning. The text's objectivity and flexibility prompt the model to focus more on semantic contextual information during knowledge acquisition. However, directly generating a comprehensive textual description of pedestrian images inevitably introduces irrelevant noise, thereby reducing model performance. Therefore, we retain the advanced modeling capabilities of large-scale language-visual models for images as possible. Specifically, for a given RGB image, TPR utilizes a pre-trained visual question answering model to address k specific details (cf. Appendix) and acquire corresponding descriptions for the target: att = {att 1 , att 2 , . . . , att k }. Importantly, this process introduces textual detail guidance during model training but excludes text-related components during inference.Learnable Prompt Strategy Inspired by [47] , we combine the learnable prompt with original text attributes, without incurring additional expert knowledge compared to the handcrafted prompt. Concretely, TPR initially transform these attributes into tokens through CLIP tokenizer, i.e., a = T okenizer(att). Then, l learnable prompts {p 1 , p 2 , . . . , p l } are embedded into these fixed attributes tokens, forming the textual description: q = {p 1 , a 1 , p 2 , a 2 , . . . , p l , a k }. This integration introduces a dynamic knowledge learning mechanism that reduces noise introduction compared to handcrafted prompts, while enhancing the flexibility of modal interaction and transferability of text embeddings. Subsequently, the whole token q is fed into a frozen text encoder to generate text embeddings T = {T sos , t 1 , t 2 , . . . , t n , T eos } ∈ R (n+2)×d , where T sos and T eos denote the [SOS] and [EOS] token, n is the number of d-dimensional word tokens. Based on widely-used token selection, T eos serves as the global feature, while T local = {t j } n j=1 represents a sequence of basic local tokens. Dynamic Consensus Acquisition To more effectively address fine-grained semantic variations (e.g., hats, shoes) across modalities, we explore methods to filter out non-informative features for enhancing the representational capacity of text embedding for detailed information. Therefore, based on local textual feature T local , we employ metric learning to draft a dynamic consensus acquisition strategy for capturing the discriminative prototypical representations X.To begin, in order to adaptively learn related-task knowledge, local text representations T local ∈ R n×d are fed into consensus multi-layer perceptron (ConMLP) blocks to achieve feature enhancement. Then, a prototypical descriptor is formed by assigning all enhanced features to a set of atoms. The cost matrix C ∈ R n×m + can be calculated for assignment, where the (i, j)-th element C i,j represents the cost of assigning a feature to an atom. In other words, C evaluates the affinity between the enhanced features and the prototypical atoms. Concretely, to introduce certain priors as guidance [23] , the enhanced features are used to learn the cost matrix through two fully connected layers initialized randomly. In addition, some irrelevant information in textual features, such as those representing association, may disrupt the model's ability to learn the target details. Inspired by the solutions used in graph matching and key-point matching [20, 23] , we set a learnable "bin" in C. Due to the differences between prior distributions, non-informative features can be assigned to it. Specifically, we extend the cost matrix fromC to C = [C, c] ∈ R n×(m+1) +, and c = w1 n , where w is a learnable parameter and 1 n = [1, . . . , 1] ⊤ ∈ R n represents n-dimensional vector of ones. Following Sec. 3.2, we consider that assignment where the enhanced features' mass, α = 1 n , should be assigned to the atoms or the "bin"EQUATIONThe optimal assignment plan can be solved by Sinkhorn algorithm (cf. Appendix) through iterative strategy. For better descriptors quality, the extra "bin" is discarded to obtain the optimal assignment P ∈ R n×m. Finally, the augmented comprehensive representation, i.e., consensus X ∈ R m×d , is obtained by aggregating enhanced textual features and optimal transport plan: X = P ⊤ T local .TPR develops a robust transitional means to facilitate reasonable and efficient modal alignment. In order to leverage effectively the multi-granularity textual information, TCA utilizes global embedding to achieve coarse-grained modal interaction, while CFI optimizes fine-grained alignment based on enhanced comprehensive representation.Text-injected Coarse-grained Alignment Module TCA utilizes transformer architecture, known for its efficiency in modeling long-distance dependencies, to capture global information, as shown in Fig. 2 . Furthermore, we implement a cross-attention mechanism using global text embedding to emphasize contextual information injection at the beginning. For an input pair (R, S), cross-attention mechanism computes the query using text global representation T eos and derives the key and value from modality-specific global features R cls and S cls :EQUATIONwhere R/S signifies identical operations across both modalities; w Q , w K and w V denote shared learnable parameters, while Q R/S , K R/S and V R/S represent query, key and value for either the RGB or sketch modality, respectively. After obtaining the fusing features that contain textual knowledge, TCA introduces the standard transformer blocks to refine modality-specific features. Finally, we can acquire the final global concept representations R ′ cls and S ′ cls for sketch Re-ID. Consensus-guided Fine-grained Interaction Module Due to the inherent complexity of sketches and RGB images and the potential for semantic misalignment during learning, capturing detail variations in modalities is crucial. Fortunately, the fine-grained consensus X provided by TPR, which contains rich detail information, offers a key solution to this problem. CFI adopts a transformer structure based on multi-head cross attention, and it converts the original local feature R local and S local to more discriminative representations through for robust Re-ID:EQUATIONwhere w Q, w K and w V denote the parameters of project layers of h-th head for both modalities. As a result, the text detail information about pedestrian characteristics can assist our model to address the problem caused by the diversity and uncertainty of sketches and RGB images. Eventually, the local concept representations R ′ local and S ′ local can be obtained.The triple loss, a commonly-used matching loss in cross-modal learning, performs well in performance through adjusting the distance of hardest negatives in scenarios like image-text matching [12, 48] and video-text retrieval [49] . However, this strategy independently trains each semantic part with equal contribution, disregarding the overall data distribution impact when multiple samples from Table 1 : Comparison with state-of-the-art methods on Market-Sketch-1K dataset. Both training and testing set uses all sketches. 'S' and 'M' represent single-query and multi-query, respectively. 'Backbone' refers to network structure used by each method, mainly including: ResNet50 [50] and CLIP [40] . Bold values represent the optimal results.Query Backbone Reference mAP Rank@1 Rank@5 Rank@10 different modalities exhibit slight differences. This oversight may lead to inaccurate estimation of sample distances and potentially result in sub-optimal local minima [27] . To this end, we propose a new triplet assignment loss (TAL) to establish a more rational measure for evaluating the proximity of local features.DDAGFor an input positive pair (R i , S i ) in a mini-batch x, the feature representations obtained through model inference are R ′ i and S ′ i . If we treat the feature sets of all samples for each of two modalities in x as two discrete distributions, their alignment can be considered an optimal transport problem. The cost matrix Ĉ is derived from pairwise feature similarities: Ĉi,j = [(R ′ i ) ⊤ S ′ j ] + . We aim to acquire the optimal transport matrix P * with the least amount of cost, where P * i,j represents the assignment weight of (R i , S j ) obtained after balancing the overall distribution. TAL can be represented based on triplet loss as the weighted sum of the original distance and the optimal assignment distance, dynamically updated at a certain rate γ:EQUATIONwhere[x] + = max(x, 0), Rh = argmax Rj ̸ =Ri D(R j , S i ) and Ŝh = argmax Sj ̸ =Si D(R i , S j ) are the most similar negatives in x for (R i , S i ), and E(R i , S i ) = ∥R ′ i -S ′ i ∥ 2denotes the Euclidean distance between feature representations.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The paper fully discloses all the information needed to reproduce the main experimental results. The experimental section of the paper provides detailed descriptions of experimental setups, configurations, and other relevant details. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiment setup: 
 Datasets Two publicly available benchmark datasets, namely PKU-Sketch [4] and Market-Sketch-1K [6] , are utilized for performance evaluation. Both of them are sketched and annotated by professional artists. PKU-Sketch is the first publicly Sketch Re-ID dataset, containing data for 200 pedestrians, with each individual being represented through one sketch and two photos. In accordance with the setting of [4] , we randomly select 150 identities for training and 50 for testing, and final results are derived from the average of 10 experimental runs. Market-Sketch-1K is a large-scale dataset derived from the Market-1501 [1] , which is created by six artists based on descriptions, featuring multiple perspectives and artistic styles. The training set consists of 2,332 sketches and 12,936 photos, while the testing set comprises 2,375 sketches and 19,732 photos. Following the experimental setup in [6] , our method will be evaluated in three settings: single-query, multi-query, and cross-style retrieval. Performance on Market-Sketch-1K. The experimental results for Market-Sketch-1K are shown in Tab. 1. OLTM significantly outperforms all state-of-the-art methods in both single-query and multi-query settings. In the single-query scenario, OLTM achieves an mAP of 38.35% and a Rank-1 of 36.75%, surpassing state-of-the-art method by 3.38% and 5.23%, respectively. In the multi-query scenario, OLTM achieves an mAP of 62.55% and a Rank-1 of 69.48%, exceeding state-of-the-art methods by 7.37% and 12.85%, respectively. This result is primarily due to the introduction of textual semantic information in the training process of OLTM, which implicitly guides images and sketches to focus on modality-invariant features. Through hierarchical and multi-granularity alignment, the model is able to uncover discriminative fine-grained information, leading to more Table 3 : Ablation studies on Market-Sketch-1K dataset. Training and testing are under the multi-query setting. "Handcrafted" and "VQA" denote manually annotated and VQA generated text attributes, respectively. "Template" represents the sentence template defined by experts. "Prompt" denotes the learnable text prompts. The 'Baseline' uses an image encoder to process both modalities and employs simple cross-attention to integrate the global features. 'L htl ' [67] accurate queries. The visualization results on Market-Sketch-1K are shown in Fig. 3 . When sketch possesses clearly distinguishable features, query results are satisfactory. In contrast, when sketch closely resembles multiple images, making its features challenging to distinguish with the naked eye, the model encounters errors that are justifiable. In addition, to test the generalization capability of OLTM on unknown styles, cross-style retrieval evaluation is performed on Market-Sketch-1K, and detailed results are in supplementary material.Performance on PKU-Sketch. Tab. 2 presents the model's performance on PKU-SKetch dataset.The results indicate that our OLTM outperforms all competitors by a significant margin. For example, mAP and Rank-1 of OLTM are remarkably high at 91.4 and 94.0, surpassing state-of-the-art method by 5.2% and 4.0%, respectively. Because the sketches on PKU-Sketch contain more detailed information, they can assist multi-granularity interaction in acquiring more fine-grained knowledge. Fig. 3 illustrates the top-5 visualization results of OLTM on PKU-Sketch. Our method can accurately identify the target pedestrians despite challenges such as variations in posture, viewpoint, occlusion, sketch abstraction, and different painting styles.In this section, ablation experiments are conducted on Market-Sketch-1K to evaluate the effectiveness of each component within the OLTM framework.The Effectiveness of Text Prompt Reasoning. To evaluate the contribution of different prompt setting, we train the model with different combination of each setting. As shown in Tab. 3, the combination of VQA and Prompt brings significant contribution. Compared to the strategy of directly aligning different modalities, leveraging TPR to implicitly guide the alignment results in improvements of 7.08% in mAP and 9.44% in rank-1. This significant improvement is primarily attributed to introducing textual information during the training phase, which enables the model to effectively capture semantic correlations between images and sketches during inference. Furthermore, compared to manually annotated fixed attributes, those generated by VQA enhance the model's performance by adaptively adjusting the level of detail on which it focuses. Moreover, the introduction of learnable prompts improved the mAP and rank-1 by 0.79% and 4.02%, respectively, compared to fixed templates. Prompt learning can enhance the network's learning and reasoning capabilities, allowing it to more flexibly adapt to diverse modalities and enhance its sensitivity to subtle distinctions.The Effectiveness of Our Designed Modules. To validate the effectiveness of the TCA and CFI module, we progressively integrate them into the baseline and evaluate performance. The results in Tab. 3 indicate that both modules significantly enhance the alignment and interaction capabilities of model across modalities. Specifically, TCA improves mAP and rank-1 by 3.36% and 4.82% compared to baseline, respectively. The introduction of textual information in TCA effectively provides semantic guidance for coarse-grained alignment between modalities, enabling the model to focus more on similar semantic relationships. Furthermore, the integration of CFI has increased mAP and rank-1 by 1.45% and 3.82%, respectively. CFI selectively focuses on key regions in visual concept representations through semantic consensus. This process optimizes feature interaction and ensures capturing more detailed information at a fine-grained level. The Effectiveness of Triplet Assignment Loss. The experimental results in Tab. 3 demonstrate that the combination of proposed TAL L tal and identity loss L ID achieves optimal performance. The identity loss ensures that the model correctly identifies different individual identities, while TAL further optimizes feature space by pulling positive samples closer and pushing negative samples apart. Additionally, to verify the generalization of TAL, as shown in Tab. 4, we achieve superior performance across various network frameworks by substituting TAL for the weighted regularization triplet loss (WRT) [59] and hard triplet loss (HTL). Balancing between Euclidean distance and optimal transport distance can significantly enhance model performance. Please refer to the supplementary materials for more verification experiments on the key role of overall data distribution in enhancing sample feature distance.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: This paper does not currently provide open-access code, but it is planned to be made public in the future.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we present a optimal transport-based labor-free text prompt modeling (OLTM) framework for sketch re-identification. OLTM embeds text prompt reasoning module and distance measurement into transformer for achieving hierarchical multi-granularity alignment through the guidance of text semantics, leveraging the advantages of prompt learning and optimal transport. In addition, to address the limitations of Euclidean distance in measuring sample similarity, we propose a triplet assignment loss that guarantees a more effective standard based on the overall data distribution. Extensive experiments conducted on two public datasets indicate outstanding performance compared to other state-of-the-art methods for sketch Re-ID.Discussion: In this work, we employ text injection and sample distance optimization to direct the model's attention toward key details, thereby minimizing performance losses due to modal gap and sample abstraction. However, our experiments revealed that when confronted with extremely vague sketch samples (i.e., those that human cannot discern features or match), the model's identification process deteriorates into a random selection among multiple potential outcomes. Therefore, sketch Re-ID heavily depend on the quality of sketches. Enhancing the discriminability of sketches without incurring additional labor costs is a topic worthy of further exploration in future research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all the training and test details necessary to understand the results. It includes information such as data splits, hyperparameters, how they were chosen, the type of optimizer used, and any other relevant details regarding the training and testing procedures.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not report error bars suitably and correctly defined, nor does it provide other appropriate information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: For each experiment, the paper provides sufficient information on the computer resources needed to reproduce the experiments (type of compute workers, memory).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 With the growing need for urban public safety, traditional person re-identification (Re-ID) methods [1, 2, 3] are gradually becoming inadequate for criminal investigations and missing person tracking, as the individuals of interest may not have been captured by surveillance cameras. To bolster social security management and combat criminal activities, sketch person re-identification (Sketch Re-ID), which utilizes eyewitness clues to draw professional sketches as queries and match target images in a photo gallery database, has received widespread attention from researchers and scholars [4, 5, 6] , as shown in Fig. 1 . Nonetheless, due to the considerable disparity in modal heterogeneity resulting from the varied sketch styles of different artists and the diverse postures of real pedestrians in monitoring, sketch Re-ID remains highly challenging and necessitates further exploration and investigation.Giving the high generalization and inherent abstraction of person characteristics in sketches, the pedestrian features depicted in a single sketch (such as clothing and gender) may match multiple similar real images, as illustrated in Fig. 1 . A viable solution is to leverage the inter-modality interaction within the feature space to achieve hard alignment. Such methods typically employ loss constraints to directly map different modalities into a generic latent space [4, 7, 8, 9] . However, this hard alignment manner may not fully capture the complex dependencies and correlations that exist within and across modalities. To compensate for the lack of details in the above manner, another branch is to introduce an intermediate modality to bridge two source modalities. For instance, [10, 11] generate simulated sketches through adversarial learning, but the generated sketches are inevitably corrupted with noise due to limited generation performance. Additionally, [7, 12] construct benchmarks that contain textual information to alleviate modal gap; [6] improves inference efficiency by introducing text only during the training process. Despite the fact that these additional texts do contribute to mitigating the modal gap, they are all manually labeled, requiring significant human labor in real-world applications. Moreover, existing text-guided methods [6, 7] only focus on global text embeddings as masks, neglecting finer and richer local features. Therefore, this paper aims to address two key challenges: i) developing sufficient textual information as a transition mechanism without incurring additional costs, and ii) further exploring fine-grained discriminative information for multi-granularity interaction.To address the above issues, we propose the Optimal Transport-based Labor-free Text Prompt Modeling (OLTM) framework, which implicitly incorporates text semantic information during training, facilitating hierarchical and multi-granularity modal alignment. In particular, OLTM is composed of three main components: i) text prompt reasoning (TPR); ii) text-injected coarse-grained alignment module (TCA); iii) consensus-guided fine-grained interaction module (CFI). On the one hand, to introduce text sequences without additional manual annotation, we dynamically transfer pre-trained language-visual knowledge into the downstream task. Specifically, TPR first generates multi-dimensional person attributes based on real images with a pre-trained visual question answering (VQA) model. Then, these attributes are inserted as fixed parts into learnable prompts to obtain the textual embedding representations. TCA integrates global parts of the embeddings to achieve the coarse-grained alignment across modalities. On the other hand, to explore fine-grained information, we employ optimal transport theory to enhance deep-level interaction. Concretely, TPR formulates the mapping from local parts of the textual embeddings to more discriminative feature representations, i.e., consensus, as an optimal transport problem. Subsequently, guided by consensus, CFI selectively focuses on key details, extracting fine-grained conceptual representations for sketch-ID. In addition, due to the significant heterogeneity gap between modalities, using Euclidean distance as a sole metric to measure feature similarity is inadequate. Thus, we propose a triplet assignment loss to optimize feature distance measurement and improve model performance. Extensive experiments are conducted on two challenging benchmarks, which demonstrate the favorable comparison of OLTM with other state-of-the-art methods.The core contributions of this work are summarized as follows: (1) This paper proposes a novel optimal transport-based labor-free text prompt modeling framework for sketch Re-ID. To our best knowledge, this is the first attempt to apply VQA-generated text responses as a means to achieve modal alignment in sketch Re-ID without any additional annotations. (2) A novel text prompt reasoning module is deployed to dynamically extract global textual embeddings and discriminative fine-grained consensus, which guide the hierarchical multi-granularity alignment module in injecting semantic knowledge into the modeling process. (3) A new triplet assignment loss is proposed, which optimizes inter-/intra-class distance by considering overall data distribution information. cross-domain adversarial learning to narrow the feature gap. Zhang et al. [9] proposed an advanced cross-modal learning mechanism for handling non-corresponding information between modalities. However, due to significant differences between modalities, this direct alignment paradigm inevitably loses fine-grained modality-specific cues [13] . Hence, some of the latter methods investigate gentler alignment techniques through transitional modality. For example, Chen et al. [10] designed a dynamic updatable auxiliary sketch modality to increases the diversity of training samples; Zhai et al. [12] introduced a multi-modal Re-ID task by combining text and sketch as query for retrieval, exploiting their complementary advantages. Obviously, auxiliary modalities lacking detailed information may introduce noise, while data annotation is a labor-intensive task. In this paper, we first attempt to use text attributes generated by a reasonable VQA model as guidance for achieving multi-granularity alignment across modalities in sketch Re-ID.Optimal Transport For optimizing the moving cost between distributions, Optimal Transport (OT) was first proposed by Kantorovich [14] , which has shown significant potential in machine learning and computer vision, e.g., domain adaptation [15, 16, 17] , learning with noisy labels [18, 19] , and feature matching [20, 21] . Zhang et al. [22] incorporated OT into the re-ranking phase of image retrieval, significantly improving accuracy and efficiency. Similarly, Sergio et al. [23] first applied OT in visual place recognition and introduced a novel local feature aggregation method. In semi-supervised person Re-ID, OT often achieves the mapping between pseudo labels and classes as a classifier [24, 25] . In addition, Ling et al. [26] designed a assignment strategy for alleviating the intra-identity variations; Wasserstein distance was used to rectify the original global distance between samples and provides aligned distance estimation for local features [27] . Considering the enormous potential of OT in feature aggregation and distribution mapping, our study adopts OT to assist fine-grained alignment between modalities and guide the model in extracting the overall sample distribution pattern.Prompt Learning Prompt learning initially garnered widespread attention and extensive research in natural language processing [28, 29, 30] , which has gradually demonstrated significant potential in vision-language (V-L) models [31, 32, 33] and pure vision models [34, 35, 36] . Prompt learning provides a flexible way to adapt pre-trained models to downstream tasks by training only additional parameters. This enables prompts to capture task-specific information while guiding the fixed model's performance [37, 38] . In sketch-based image retrieval, [39] innovatively learns a unified prompt for different branches in CLIP's [40] visual encoding layer, fully exploiting CLIP's zero-shot learning potential. In text-to-image person Re-ID, [41] introduces a multi-prompt strategy to integrate text prompts from various sources for fine-grained interaction. Furthermore, Li et al. [42] first attempt to conduct in-depth research on zero-shot multi-modal ReID through a large foundational model. In this paper, we delve into the significant application of prompt learning in sketch Re-ID, innovatively generating global text representations by integrating fixed and learnable prompts, and utilizing OT to reason consensus for effectively guiding detailed interactions across modalities. 
 
 Appendix: This section mainly introduces optimal transport and its corresponding algorithm, Sinkhorn-Knopp [44] . Let Γ r := {x ∈ R r + |x ⊤ 1 r = 1} represents the probability simplex, where 1 r is the rdimensional vector of ones. Given two probability simplex vectors α ∈ Γ m and β ∈ Γ n and a cost matrix C ∈ R m×n , the objective of OT is to seek the optimal transport plan P * mapping α to β at the minimum cost:EQUATIONwhere U (α, β) denotes the transport polytope of α and β, i.e., the solution space of P . The above problem is to find optimal solution P * in a set of all possible joint probabilities of (X, Y ), where X and Y represent random variables with marginal distribution α and β.Generally, optimal transport (OT) problem is a linear programming problem that can theoretically be solved in polynomial time. However, in the actual solving process, it involves the square of anchor feature dimensions at all scales, requiring near-cubic complexity [68] . Therefore, consider optimizing the problem through iteration. This method optimizes the solving process by adding an entropy constraint to the OT problem, transforming Eq. 7 into a non-linear convex form with a regularization term:d C (α, β) = min P ∈U (α,β)⟨C, P ⟩ + δE(P ),EQUATIONwhere δ is a hyper-parameter and E(P ) = P (log P -1) is the entropy of P . Introducing a regularization term is equivalent to introducing a prior knowledge: without considering the cost matrix C, the distribution of assignment matrix P is expected to be as uniform as possible. Eq. 8 is an entropy-regularized OT (EOT) problem and can be solved by Sinkhorn-Knopp algorithm [44] based on iterative updates of vectors. According to the Lagrange Multiplier Method, the conditional extremum problem (i.e., Eq. 8) can be transformed into an unconditional extremum problem: dC (α, β) = min P ⟨C, P ⟩ + δE(P ) + µ(PEQUATIONwhere, µ and ρ are the Lagrange multipliers. If we take its derivative and set it to 0, we can find P * :EQUATIONLet u = exp(-µ δ ), v = exp(-ρ δ ) and K = exp(-C δ ), two constraint of Eq. 8 needs to be met simultaneously. Thus, one possible solution is to iterate enough times according to the following iteration formula:EQUATIONEq. 11 is called as Sinkhorn-Knopp iteration. After z iterations, P * can be obtained by the following equation:EQUATIONIt is worth noting that the process of updating u and v (i.e., Eq. 11) alternately can be simplified to a single step: u z ← α/K(β/K ⊤ u z-1 ). Furthermore, when all elements of α are positive, this single step can be further simplified as: u z ← 1./ K(β/K ⊤ u z-1 ), where K = Diag(1./α)K. The overall flow of Sinkhorn-Knopp is described by Algorithm 1.This appendix provides some details on gradient derivation. To simplify representation and analysis, we focus on a single direction, following the approach in [69] , considering that sketch-to-image retrieval and image-to-sketch retrieval are symmetrical. Moreover, we assume that there is only oneRequire: Cost matrix C, weights α, β Require: Hyper-parameter δ, iterations i = 1, max iteration z 1: Initialize K = e -C/δ , K = Diag(1./α)K, u 0 = 1 n /n 2: while u changes or i is less than or equal z do 3:u i ← 1./ K(β/K ⊤ u i-1 )4:i ← i + 1 5: end while 6: Get optimal value u * = u 7: v * = β/K ⊤ u * 8: return optimal flow matrix P * = Diag(u * )KDiag(v * ) paired image for each sketch in the mini-batch. Consequently, we can simplify TRL and TAL as shown below:EQUATION)where δ is the hyper-parameter of OT problem, ri and r i are the hardest negative sample and positive sample of the anchor sample s i , respectively. These l 2 -normalized features are embedded by the modality-specific models, i.e., f θr (•) and f θs (•). Due to the truncation operation [x] + , we only discuss the case of L > 0 that could generate gradients. For TRL, the gradients to the parameters θ r and θ s are:EQUATIONSince the learning of normalized features can be viewed as the movement process of points on a unit hyperplane, we only consider the loss gradients with respect to r i , ri and s i are:EQUATIONFor our TAL, the gradients to the parameters θ r and θ s are:EQUATIONSince the result of exp(-r ⊤ s δ ) is obtained through several iterations of Sinkhorn-Knopp algorithm, this part does not conduct gradients and can be simplified as a coefficient δ = 1-exp(-r ⊤ s δ ) ∈ [0, 1]. Thus, the gradients for r i , ri and s i are:EQUATIONWe have included visual analysis in Fig. 4 , which illustrates the convergence curve and sample distances during training. Figure 1 (a) shows that conventional triplet loss converges prematurely. In contrast, our proposed triplet assignment loss exhibits higher volatility, reducing the risk of suboptimal local minima. Additionally, Figure 1 Our OLTM achieves the trade-off between performance enhancement and computational complexity.To this end, we select several methods for comparing parameters, floating-point operations (FLOPs), and frames per second (FPS), as shown in the Tab. 5 below. The results show that OLTM gets remarkable performance while maintaining reasonable computational costs. The reason is that: 1) only TCA module is required for inference; 2) the Visual Question Answering(VQA) model is used during data processing. Multi-query Setting Similar to [6] , "multi-query" involves combining multiple sketches of the same ID during both training and inference. Our paper employs a straightforward fusion method by averaging the image features from multiple sketches. Tab. 7 below provides a comparative analysis of various fusion strategies. The results demonstrate that the basic and simple fusion method achieves the best experimental performance. Parameter Analysis For the Triplet Assignment Loss proposed in our work, we compute the sample distance using Eq. 18. Fig. 5 illustrates an analysis of the hyper-parameter γ. It can be observed that setting γ = 0.3 yields the best performance in Rank-1 and mAP.EQUATION)whereE(R i , S i ) = ∥R ′ i -S ′ i ∥ 2denotes the Euclidean distance between feature representations. In Dynamic Consensus Acquisition, the cost matrix C ∈ R n×m + can be calculated for assignment, where the (i, j)-th element C i,j represents the cost of assigning a feature to an atom. Concretely, to introduce certain priors as guidance [23] , the enhanced features are used to learn the cost matrix through two fully connected layers initialized randomly. m is a hyper-parameter that needs to be set. Fig. 6 analyzes the values of m, showing that the best performance in terms of Rank-1 and mAP is achieved when m = 32. Qualitative Analysis Fig. 7 shows the Rank-5 results of OLTM and the baseline on the Market-Sketch-1K dataset. The left and right parts show the retrieval results of the baseline and OLTM, respectively. We can observe that OLTM can focus on more fine-grained discriminative information, such as bag and hat. In contrast, the baseline only considers global information matching, which leads to performance degradation. Through extensive statistical analysis, we have formulated 9 specific questions to obtain the corresponding fine-grained textual attributes for a given image. These nine questions are as follows:1. What is the gender of this person?2. Is this person with long or short hair? The VQA model is inherently substitutable. Essentially, any visual-language model which is capable of generating target attribute information from images can serve as an alternative. We also use other VQA models to demonstrate their substitutability, as shown in the Tab. 9 below. The VQA model has the capability to describe fine-grained recognition information for the following reasons: 1) The VQA model generates detailed attributes about various aspects of the pedestrian target (e.g., hair, backpack, hat), rather than relying on complete descriptive sentences. 2) We provide a visualization comparison, as shown in Fig. 8 . This comparison demonstrates that using text attributes can guide effectively the attention of model. We conduct additional background evaluation on Market-Sketch-1k dataset based on our initial studies. Specifically, we formulate the question 'What is the background of this image?' to extract textual attributes about background. The extracted background details are illustrated in Fig. 9 . However, the introduction of background information result in a decrease of 2.81 in Rank-1 and 1.62 in mAP. This decline can be attributed to the absence of corresponding background information in the sketches, which potentially interferes with the model's learning process. male short hair yellow sleeves short sleeves shorts gray shorts with bag without hat with glasses bike male short hair green sleeves short sleeves pants blue pants with bag without hat with glasses tree To verify the effectiveness of different text attributes, we have provided additional ablation experiments in Tab. 10 below. The results show a significant decrease in model performance after discarding several hard-distinguished attributes (e.g., color and gender) in sketches. As shown in Fig. 10 , sketches convey gender-related information through factors like body shape, and the contrast between light and dark areas effectively highlights specific color details. The TPR module injects detailed information into modal interactions during training. This enables the model to focus on these nuances autonomously, even without TPR during inference.Table 10 : Building on the original experimental setup, this comparison evaluates performance by removing the fine-grained textual attributes of gender, up color, and down color.gender up color down color mAP Rank@1 Rank@5 Rank@10 Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The abstract and introduction provide a concise overview of the research area, the novel approaches introduced, and the key contributions, ensuring that readers have a clear understanding of what the paper aims to achieve and the significance of its findings.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper introduces a novel framework for  Sketch Re-ID. After the author-reviewer discussion period, this paper received an average score of 5.75, with one negative score of 4. Since the reviewer with such a negative score did not engage in the discussion, I have checked the author's rebuttal carefully. I agree that the author has addressed the concerns.

Since the AC recognizes the overall quality of this paper, it is recommended to accept it to the NeruIPS'24 conference.

Reviews: review: paper_summary: paper_summary: This article proposes a framework for pedestrian re identification in sketch images based on optimal transportation theory, which utilizes visual question answering pre training models to address the current issue of high text annotation costs and only focuses on global feature representation. This framework utilizes a visual question answering model to automatically generate text prompts, and extracts coarse-grained features through interaction between the text encoder and the visual encoder. By combining clustering methods based on optimal transportation theory with multi head attention mechanism, fine-grained features are extracted, achieving hierarchical and multi granularity alignment, and obtaining a relatively complete feature representation. In addition, a triplet loss function was designed that prioritizes regions with maximum local feature similarity, providing a more reasonable representation for measuring local feature distance. Through experiments on two public datasets, it can be proven that the pedestrian re identification and generalization ability of this model framework have been improved compared to previous model methods.
strength_weakness: strength_weakness: (1) The calculation cost is relatively high. Due to the introduction of a new visual question answering model and the addition of multiple attention mechanism structures, the computational resources required by the model are significantly increased compared to previous models.
(2) Overfitting is prone to occur. During the experiment, it was found that due to the small sample size of the sketch dataset and the rich granularity and depth of the model's feature extraction, overfitting is prone to occur, which can also affect the model's generalization ability.
(3) The accuracy of text prompts is greatly affected by image quality. The text prompts designed in this article have some fixed prompts and some learnable units. Due to the accuracy of text information directly affecting the quality of feature extraction, the performance of visual question answering pre training models on sketch image modalities is easily affected by different styles of images.
questions: questions: (1) How is the number of learnable text prompt units added to a fixed text prompt vector determined?
(2) Does the visual question answering model have the ability to describe feature information that truly possesses fine-grained recognition ability?
(3) Is there a comparison of models with artificial text prompts in the experimental section? Will this method still have advantages in terms of re identification performance compared to models with manual text prompts that have been described with fine-grained features?
(4) Regarding the use of optimal transportation theory to remove irrelevant information, is it possible to increase recognition accuracy if the text description can involve background information?
limitations: limitations: (1) The performance of visual question answering models on sketch images will limit their ability to extract features, so it is necessary to evaluate whether the VQA model can provide fine-grained local feature descriptions on sketch images;
(2) There may still be misjudgments when performing re identification on two individuals with similar appearances but different identities, so text prompts may need to refer to more information, such as extracting content that is beneficial for judgment from discarded background information.
paper_summary: paper_summary: This paper proposes an optimal transport-based labor-free text prompt network for sketch re-identification. The authors address two primary challenges: the expense of text annotation and cross-modal interaction, leveraging generated text attributes for multi-granularity modal alignment. The experimental results on two datasets achieve significant improvements.
strength_weakness: strength_weakness: The paper is interesting and the topic authors discussed is very promising, but I feel there are some doubts:

1. The author suggests that the classic triplet loss may lead to inaccurate estimates of sample distances and potentially result in suboptimal local minima. The proposed Triplet Assignment Loss aims to address this issue. How can this viewpoint be substantiated from experiments?

2. Comparisons with the most related method [1] are missing in TABLE 1 and TABLE 2.

3. Do the transformer blocks in feature enhancement module for different modalities share weights?

4. The description of pedestrians comes from VQA's answers to 9 questions. These questions typically cover details like color and gender. However, sketches used in testing often lack such explicit information. How does the model then prioritize these aspects?

5. The experimental results in Table 1 include "multiple-query". However, the paper does not provide a description of how this is achieved.

6. The paper compares the structural and performance differences between OLTM and other methods. However, it does not discuss whether the additional structural design increases the computational burden. Including relevant cost comparison experiments would greatly enhance the study.

7. Is the VQA model irreplaceable? I did not come across a statement like "VQA model is optional" in the paper.

[1] Li H, Ye M, Zhang M, et al. All in One Framework for Multimodal Re-identification in the Wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 17459-17469.
questions: questions: Major concerns: see Weakness.
Minor concerns:
a)	In 4.1, the inference stage uses Rcls and Scls instead of r and s; the bolding of the formula needs to be checked.
b)	Add detailed descriptions of the cross-style retrieval experiment in the supplementary materials.
limitations: limitations: The author mentions that their proposed framework effectively tackles challenges like occlusion and perspective but still encounters misjudgments with highly similar but distinct individuals. I consider this a critical issue for collective resolution in advancing re-identification tasks. My positive evaluation of the research remains unchanged. I will, however, follow the authors' work and look forward to potential solutions for these challenges.
paper_summary: paper_summary: This paper focuses on sketch based person ReID. Especifically, a labor free text method with OT is proposed, which achieve a large performance improvement on two public databases. Overall, I think the idea of this paper is interesting and the experiments show its superiority.
strength_weakness: strength_weakness: 1. Some grammar errors should be corrected.
2. More detailed analysis on OT and TPR should be provided.
3. From the paper, I don't quite understand the settings of the parameters. In the proposed method, what is the mean of $\alpha$ and $\beta$? What is the relationship between them and cost matric $C$?
questions: questions: 1. In the related works, there is a subsection on optimal transport. What is the relationship of [26]/[27] and OT.
2. On Line 143 “represents” should be modified to “represent”. On page 6, P* should be bold, be consistent to above.
3. On line 185, authors argue that the proposed strategy would not introduce any noise. So, how to ensure it?
4. In eq.(7), it is better to give more description about how to calculate the P*.
limitations: limitations: The author has addressed the limitation.
paper_summary: paper_summary: This article extends image-based person and vehicle reid task to sketch field. First, authors introduce a novel sketch-based reid framework named OLTM, which utilizes text information to achieve modal alignment. Additionally, in sketch-based reid, authors apply the VQA model to generate textual attributes of persons, thus avoiding costly annotation efforts. Furthermore, authors introduce a novel triplet alignment loss to enhance distance calculation.
strength_weakness: strength_weakness: 1. Compared to CNN-based methods, using CLIP as a Backbone may suffer from high computation complexity.

2. Some minor issues: the symbols for K, Q, and V in Eq. 4 and 5 should be bolded.

3. The names in the paper and the supplementary material are inconsistent, i.e., HTL/TRL.

4. Figure 2 in the supplementary material is rather blurry.
questions: questions: 1. Why is the ViLT-based VQA model used instead of others, like Bert-based model?

2. Is there an error in Eq. 7? After checking the supplementary materials, my understanding is that the optimal transport matrix P* is a weight matrix, not a loss value.

3. Some details are not clear to me. It is unclear whether the process of obtaining text attributes occurs during model training or after pre-processing. According to the framework Figure 2, this process seems to be executed during the model training process, which can result in complex model calls.

4. The experiment doesn’t mention whether it is compared with existing image-based re-identification methods.
limitations: limitations: N/A

Comments: comment: We sincerely want to know if our response has addressed your concerns. If you are willing to, we would be eager to continue the discussion to better understand your thoughts.
Thank you for your time and efforts. We are encouraged that your concerns have been addressed, and we greatly appreciate your positive feedback on our work.
Thanks to the authors for the reply. The authors answered most of my questions. Therefore, I maintain my positive rating.
Thanks for your response and updating the rating. Your feedback is deeply valued and helps improve our paper.
Thank you for the author's reply. The detailed reply and analysis have dispelled my doubts about OT, TPR, and parameter settings.
Thank you for recognizing our work and raising your score. Your suggestions significantly help improve the quality of our paper.
This is a good job, and the author has also answered my questions well.