SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $d$  is the channel dimension of  $K$ ;  $F_{t}^{j}$  is the enhanced feature under the guidance of the  $j$ -th textual phrase. Particularly, in this work, we follow the multi-head attention [28], partitioning the  $Q / K / V$  in the channel dimension into multiple groups and applying attention to each group independently according to Equation 4. Through this attention operation, we could use explicit textual guidance to help our model to focus on distinguishable visual clues. For intuitive examples of how textual guidance aids our model in capturing discriminative clues, please refer to Figure 2 in the Appendix. Thereafter, we replay Equation 3 and Equation 4 to apply a self-attention over the  $F_{t}^{j}$ , in which all the  $Q$ ,  $K$ , and  $V$  are embedded from  $F_{t}^{j}$ . Finally, we use a global max pooling to extract the final feature  $F^{j} \in \mathbb{R}^{C}$ , which is used for loss computation and inference.
Training and Test Procedure. During the training stage, we use these 4 phrase features to respectively guide the refinement of visual features and utilize refined visual features to calculate triplet loss independently. The objective function could be written as,
$$
L = \sum_ {j = 1} ^ {4} L _ {t r i} \left(F ^ {j}, y\right), \tag {5}
$$
where  $L$  is the overall objective function of our UniReID;  $L_{tri}$  indicates triplet loss;  $y$  is the ground truth. During the test phase, we concatenate these 4 refined visual features together for inference.
Datasets. Wildlife-71. In this work, we evaluate the generalization capability of our UniReID model on our constructed Wildlife-71 dataset. Briefly, our Wildlife-71 consists of 67 training wildlife
Table 1: Comparison results of our UniReID and other ReID methods under the ReID-AW setting. "ReID" and "DGReID" indicate models designed for general and domain generalization ReID settings.
Type	Method	Reference	Zebra	Seal	Giraffe	Tiger	AVG
mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1
ReID	TransRelD [6]	ICCV2021	26.3	45.3	19.7	50.3	42.2	67.1	62.8	62.0	37.8	56.2
CAL [24]	ICCV2021	26.8	45.0	21.6	52.7	44.0	69.3	64.1	63.4	39.1	57.6
AGW [31]	TPAMI2021	24.0	40.1	23.8	55.9	42.7	68.6	58.5	59.5	37.3	56.0
DGReID	IBN-Net [20]	ECCV2018	26.2	42.6	25.2	58.0	44.0	67.5	60.5	61.5	39.0	57.4
SNR [12]	CVPR2020	25.5	42.4	24.1	55.0	42.5	67.0	61.5	63.4	38.4	57.0
MetaBIN [4]	CVPR2021	26.4	45.6	23.9	56.7	42.6	69.3	61.2	62.0	38.5	58.4
DTIN-Net [11]	ECCV2022	26.0	43.2	25.2	57.3	43.5	69.6	61.9	61.8	39.2	58.0
ReID-AW	UniRelD	-	29.60.1	48.70.4	28.50.2	62.60.4	46.50.2	73.90.7	66.70.8	65.21.4	42.8	62.6
categories and 4 test wildlife categories, namely zebra [21], seal [18], giraffe [21], and tiger [15]. Each training category contains an average of 29 identities, with each identity including approximately 56 images. To further enrich the training data, we include a person ReID benchmark MSMT17 [30] and a vehicle ReID benchmark VehicleID [16] into the training set of Wildlife-71 as two additional object categories. During the evaluation phase, we evaluate our model on the four test (unseen) categories. For more details about Wildlife-71, please refer to Section 1 in the Appendix.
Implementation Details. We employ the pre-trained CLIP [23] as the backbone model. During the training and test stage, we resize all images into  $224 \times 224$ . In the training phase, we adopt the category-specific sampling strategy, as mentioned in Section 3.2. Our model is trained for 20 epochs. The learning rate is initialized to  $1.0 \times 10^{-4}$  and divided by 10 at the 15th epochs. Random flipping, random erasing, and color jittering are employed for data augmentation. Two NVIDIA Titan GPUs are used for model training. We employ the CMC [29] and mAP [34] metrics for evaluation. All ablation experiments in this section are repeated times, and the mean $^{\text{std}}$  results are reported.
4.2 Comparison between Our UniReID with Other State-of-the-art Algorithms
For fair comparisons, all methods (exclude our UniReID) compared on our Wildlife-71 are trained on base categories and then fine-tuned with the support data (a triplet of images) of each test category for 10 iterations.
Comparison on Wildlife-71. To evaluate the category generalization capability of our UniReID model, we compare it with other state-of-the-art ReID models on our Wildlife-71 dataset. In this part, we compare our UniReID with models designed for general ReID setting (ReID) and domain generalization ReID setting (DGReID), both of which concentrate on single-category object identification. Specifically, ReID aims to develop effective models that achieve optimal identification performance in a fixed environment, while DGReID aims to design robust models that perform well in unseen environments. The results are presented in Table 1, revealing two key findings. Firstly, on our Wildlife-71 dataset, DGReID methods only marginally outperform general ReID methods, with an average improvement of  $1.1\%$  in CMC-1. This observation suggests that re-identifying unseen object categories may be more complex than the DGReID task, requiring ReID models to adaptively accommodate the unique characteristics of novel categories rather than solely addressing style divergence across domains [12]. Secondly, we can find that our UniReID model considerably surpasses all compared methods, achieving an average  $3.6\%$  better performance in mAP than the best competitor, DTIN-Net [11]. This outcome implies that the dual-modal guidance employed in our model effectively enables dynamic adaptation to novel categories, leading to enhanced generalization capabilities.
To clarify, in the rest of this paper, the "Baseline" indicates a CLIP model trained with visual prompt tuning [10] using static prompt vectors shared for all categories.
The Effectiveness of Our Designed Modules. To investigate the effectiveness of our proposed modules, i.e., the category-specific visual prompts and the text-guided attentive module, we incrementally incorporate them into the "Baseline" model and compare the performance improvements. The results, as presented in Table 2, show that both of our designed modules effectively help the ReID model to handle unseen categories. In comparison to the "Baseline" model, our category-specific visual prompts ("Ada.Prompt") result in an average improvement of  $1.1\%$  in mAP. This enhancement sug-
Table 2: Ablation experiments of the designed components. The "Visual Prompt Mode" indicates which kind of prompt vectors are used in the current model. "Static" and "Adaptive" represent static prompts shared for all categories and our category-specific prompts, respectively. We can find that both of our designed modules, namely category-specific prompts ("Ada.Prompt") and text-guided attentive module ("Textual Attention"), effectively help the ReID model to handle unseen categories.
Visual Prompt Mode	Textual Attention	Zebra	Seal	Giraffe	Tiger	AVG
mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1
Baseline	Static	×	26.60.5	43.80.9	26.20.4	58.20.4	43.80.2	68.41.2	62.10.1	60.30.4	39.7	57.7
Ada.Prompt	Adaptive	×	27.30.3	45.40.3	26.90.7	59.70.4	45.60.2	70.40.7	63.50.3	60.71.4	40.8	59.1
UniRelD	Adaptive	✓	29.60.1	48.70.4	28.50.2	62.60.4	46.50.2	73.90.7	66.70.8	65.21.4	42.8	62.6
Figure 3: Ablation of our adaptive category-specific prompts ("Ada.Prompt"). CMC-1 to CMC-5 results on 4 test categories are given. We can find that our category-specific prompts tuning strategy outperforms all compared fine-tuning approaches.
gests that these adaptive visual prompts are indeed capable of extracting knowledge from support data and tailoring our UniReID to the target category. Moreover, the text-guided attentive module leads to a further increase of  $2.0\%$  in mAP on average. This indicates that using explicit semantic guidance from large-scale language models could indeed help ReID models to capture the discriminative clues of novel categories.
The Effectiveness of Adaptive Prompt Tuning. In our UniReID, we use a set of category-specific prompts to adapt our model to novel categories dynamically. To evaluate the effectiveness of our adaptive prompt tuning strategy ("Ada.Prompt"), we compare it with other fine-tuning approaches, namely "Full Tuning" that fine-tunes all parameters of CLIP, "Norm Tuning" [8] that fine-tunes only normalization layers inside CLIP, and "Prompt Tuning" that introduces a set of static prompts [10] shared for all categories on the basis of "Norm Tuning". The comparison results are given in Figure 3, from which we can get three important findings. Firstly, the "Full Tuning" approach demonstrates the poorest performance among all strategies compared. This could be attributed to the fact that modifying the original parameters of CLIP leads to overfitting the base categories, thereby failing to generalize. Secondly, the performances of "Norm Tuning" and "Prompt Tuning" are roughly comparable. These results are expected, as the prompt vectors used in "Prompt Tuning" are especially learned for the base categories, potentially limiting its generalization capabilities towards novel categories. Thirdly, it is evident that our implemented "Ada.Prompt" outperforms all other compared approaches. This could indicate that our adaptive category-specific prompts indeed learn and transfer crucial knowledge to our model, enabling dynamic adaptation to target categories.
The Effectiveness of Textual Guidance. From the results in Table 2, we can find that our text-guided attentive module plays a pivotal role, contributing a significant improvement of  $2.0\%$  in mAP on average. However, a question may arise: does the model actually learn to utilize the textual guidance, or is this improvement simply the consequence of introducing attention modules? In response to this question, we evaluate the effectiveness of textual guidance from two aspects. 1) Firstly, we replace the textual tokens in our text-guided attentive module with a set of learnable vectors, thereby converting it into a self-attention module. The results, presented in Table 3, reveal that the self-attention module ("Self-Attn") only brings a  $1.0\%$  CMC-1 improvement to the model that does not use attention modules ("w/o Attn"). This could be attributed to the learnable vectors inside the self-attention module potentially overfitting the base categories, leading to poor generalization. This finding indicates that the superiority of our model does not solely stem from the additional attention module. 2) Secondly, we replace the textual guidance used in our text-guided attentive module with mismatched ones ("Mismatched"), e.g., providing the zebra with monkey guidance, or with a general one ("General") in the format of "Clue one, Clue two, ...". Notably, for the mismatched guidance, we randomly select guidance from three different mismatched categories and averaged the
Table 3: Ablation experiments of textual guidance. The "Self-Attn" is the version that replaces our text-guided attentive module with a self-attention module. The "Mismatched" indicates the version replaces the textual guidance used in our text-guided attentive module with mismatched ones, e.g., providing the zebra with monkey guidance. The "General" is the version that replaces the textual guidance with a group of general phrases like "Salient clue one, ...". We can find that our UniReID with matched guidance ("Matched") outperforms all variations, which indicates our model could make good use of semantic knowledge to adapt to novel categories.
Textual Guidance Mode	Zebra	Seal	Giraffe	Tiger	AVG
mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1
w/o Attn Self-Attn	-	27.30.3	45.40.3	26.90.7	59.70.4	45.60.2	70.40.7	63.50.3	60.71.4	40.8	59.1
-	28.70.1	47.00.1	27.40.2	60.20.4	45.50.1	70.50.2	65.20.2	62.60.3	41.7	60.1
UniReID	Mismatched General	29.10.2	47.40.5	27.80.1	61.20.8	46.00.1	72.60.4	65.70.3	64.30.2	42.2	61.4
29.20.2	47.60.1	28.20.4	61.40.7	46.20.1	73.10.1	65.90.4	64.30.9	42.4	61.6
Matched	29.60.1	48.70.4	28.50.2	62.60.4	46.50.2	73.90.7	66.70.8	65.21.4	42.8	62.6
Table 4: Comparison results of our UniReID model trained on different datasets, namely vehicle ReID dataset VehicleID, person ReID dataset MSMT17, and our Wildlife-71 dataset. The results indicate that our Wildlife-71 dataset significantly enhances the performance compared to single-category benchmarks.
Training Set	Zebra	Seal	Giraffe	Tiger	AVG
mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1	mAP	CMC-1
MSMT17 (M) [30]	23.70.5	40.51.3	19.81.5	45.91.8	39.11.4	65.70.4	45.90.3	48.20.3	32.1	50.1
VehicleID (V) [16]	21.11.3	35.31.7	23.71.2	55.61.9	36.40.2	59.50.1	44.60.9	47.00.3	31.5	49.4
M&V	24.00.4	39.90.8	21.80.7	51.40.1	39.51.2	63.40.9	47.40.4	49.90.3	33.2	51.2
Wildlife-71	29.60.1	48.70.4	28.50.2	62.60.4	46.50.2	73.90.7	66.70.8	65.21.4	42.8	62.6
final results. The results, presented in the "Mismatched" and "General" rows of Table 3, show that unsuitable textual guidance causes substantial performance degradation compared to using matched ones ("Matched"), i.e.,  $1.1\%$  CMC-1 on average. This finding indicates that our model indeed learns to utilize semantic knowledge effectively. For intuitive cases of how textual guidance aids our model in capturing salient clues, please refer to Figure 2 in the Appendix.
Effect of Our Wildlife-71 Dataset. In this work, we construct the Wildlife-71 dataset, which comprises 71 wildlife categories. To the best of our knowledge, our Wildlife-71 dataset is the first ReID benchmark that encompasses multiple object categories. To evaluate the value of this diverse dataset, we compare the performance of our UniReID trained on Wildlife-71 with versions trained on two mainstream ReID benchmarks, namely VehicleID [16] and MSMT17 [30], which only contain pedestrian or vehicle instances. As depicted in Table 4, the performance of the UniReID model trained on the Wildlife-71 dataset surpasses the models trained on single-category benchmarks by a significant margin, achieving an average  $12.8\%$  CMC-1 improvement. Furthermore, compared to the version trained on both VehicleID and MSMT17, the UniReID model trained on Wildlife-71 also displays superior performance, with an increase of  $11.4\%$  in CMC-1. These results indicate that the diverse categories and the plentiful images in the Wildlife-71 dataset are indeed effective in endowing ReID models with sufficient knowledge for generalization. Therefore, we believe that our Wildlife-71 dataset is a right step toward constructing universal wildlife ReID models for open-world applications.
In this work, we propose a ReID-AW task, which aims to construct universal ReID models to handle unseen animal categories in the open world. To support the ReID-AW task, we construct a Wildlife-71 dataset, which contains ReID data from 71 wildlife categories. To the best of our knowledge, Wildlife-71 is the first ReID dataset encompassing multiple object categories. Furthermore, we also design a UniReID model, which utilizes dual-modal guidance to dynamically adapt to novel categories. We hope our proposed task, dataset, and approach could support future research on category generalizable ReID methods.
Limitation. As pioneers in designing category-generalizable ReID models, our Wildlife-71 dataset only includes limited wildlife categories. In our future work, we will continually enhance our Wildlife-71 dataset and incorporate other categories and more instances.
Acknowledgements This work was supported by National Key R&D Program of China (No.2020AAA0106900), the National Natural Science Foundation of China (No.U19B2037), Shaanxi Provincial Key R&D Program (No.2021KWZ-03), and Natural Science Basic Research Program of Shaanxi (No.2021JCW-03).
[34] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In Proc. IEEE Int. Conf. Comp. Vis., pages 1116–1124, 2015.