SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bingliang Jiao $^{1,2,3,5}$  Lingqiao Liu $^{4}$  Liying Gao $^{1,2,3}$  Ruiqi Wu $^{1,2,3}$
Guosheng Lin $^{5\dagger}$  Peng Wang $^{1,2,3\dagger}$  Yanning Zhang $^{1,2,3}$
$^{1}$ School of Computer Science, Northwestern Polytechnical University, China
$^{2}$ Ningbo Institute, Northwestern Polytechnical University, China
$^{3}$ National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean, China
<sup>4</sup>The University of Adelaide, Australia <sup>5</sup>Nanyang Technological University, Singapore
The current state of re-identification (ReID) models poses limitations to their applicability in the open world, as they are primarily designed and trained for specific categories like person or vehicle. In light of the importance of ReID technology for tracking wildlife populations and migration patterns, we propose a new task called "Re-identify Any Animal in the Wild" (ReID-AW). This task aims to develop a ReID model capable of handling any unseen wildlife category it encounters. To address this challenge, we have created a comprehensive dataset called Wildlife-71, which includes ReID data from 71 different wildlife categories. This dataset is the first of its kind to encompass multiple object categories in the realm of ReID. Furthermore, we have developed a universal re-identification model named UniReID specifically for the ReID-AW task. To enhance the model's adaptability to the target category, we employ a dynamic prompting mechanism using category-specific visual prompts. These prompts are generated based on knowledge gained from a set of pre-given images within the target category. Additionally, we leverage explicit semantic knowledge derived from the large-scale pre-trained language model, GPT-4. This allows UniReID to focus on regions that are particularly useful for distinguishing individuals within the target category. Extensive experiments have demonstrated the remarkable generalization capability of our UniReID model. It showcases promising performance in handling arbitrary wildlife categories, offering significant advancements in the field of ReID for wildlife conservation and research purposes. Our work is available in https://github.com/JiaoBL1234/wildlife.
Object Re-identification (ReID) aims to match identical target objects across non-overlapping camera views. Although existing ReID methods [7, 22, 24, 31, 33] have made the remarkable process, their identification capabilities heavily rely on their training benchmarks. This reliance often results in overfitting to base categories in the training set, e.g., vehicle [22, 33] and person [7, 24, 31]. However, in open-world scenarios, the categories of target objects could be diverse. For example, ReID technologies are widely used toward animal objects for the purposes of analyzing population counts and migration patterns. To adapt these ReID methods to new animal categories, users have to compile large-scale datasets first. This process is time-consuming and poses a significant hurdle to the open-world applications of ReID methods. This limitation motivates us to propose a novel and practical task named "Re-identify Any Animal in the Wild", which aims to construct a universal
†Corresponding author, bingliang.jiao@mail.nwpu.edu.cn, peng.wang@nwpu.edu.cn
Figure 1: The sketch of the ReID-AW task, which aims to train a ReID model able to handle any animal category under the guidance of a triplet of images within the target category.
ReID model to break the category boundaries and handle unseen animal categories in the open world, as shown in Figure 1.
An effective universal ReID model must fulfill two essential criteria. Firstly, it should have sufficient knowledge learned from an extensive range of seen categories. Secondly, the model should possess the capability to transfer valuable knowledge from seen categories for distinguishing individuals within novel categories. The first criterion necessitates a training dataset that covers diverse object categories, multiple viewpoints, and abundant identities for each category. This ensures that the ReID model can acquire the necessary knowledge for generalization. Unfortunately, existing ReID benchmarks primarily focus on two categories, namely vehicle [22, 33] and person [7, 24, 31], which is insufficient for training a universal ReID model. The second criterion requires the ReID models to be able to adaptively transfer the knowledge learned from the training categories to new categories. It is crucial to recognize that not all the knowledge acquired from the base categories is applicable to the novel ones. For instance, the ReID model might be trained to consider color as a significant attribute for distinguishing many animals, such as dogs and cats. However, such a clue is ineffective or even counterproductive for differentiating chameleons.
In this work, we present the first attempt to construct a universal ReID model. To ensure that the ReID model meets the two aforementioned criteria, we first propose the Wildlife-71 dataset, which contains ReID data from 67 training (base) animal categories and 4 test (novel) categories. In our Wildlife-71 dataset, each training category contains an average of 29 identities, with each identity including approximately 56 images. Our Wildlife-71 dataset is collected from three sources: integrating existing datasets [21, 18, 16, 30, 15], extracting target bounding boxes from a large-scale tracking dataset GOT-10k [9], and crawling web videos to extract target bounding boxes using a tracking algorithm [32]. To the best of our knowledge, Wildlife-71 is the first ReID dataset encompassing multiple object categories. Moreover, in Wildlife-71, we provide a triplet of images (two from the same identity and another from a different identity) as support data for each category in the test set. These support data are pre-accessible for the ReID model to help it adapt to the novel categories. It is worth noting that a triplet image is the smallest unit for learning knowledge to distinguish between individuals, and it is readily available in real-world applications.
In addition to our dataset, we construct a novel universal ReID framework named UniReID. One of the major challenges we faced is equipping UniReID with the ability to select and apply appropriate knowledge for adapting to unseen categories. This step is non-trivial, as it requires the ReID model to both learn the knowledge about the target categories promptly and possess sufficient adaptability to the obtained knowledge. To accomplish this, we train our UniReID to efficiently acquire knowledge about target categories from dual-modal guidance, i.e., visual and textual guidance, and use the knowledge to dynamically adapt to various categories. For visual guidance, we use a triplet of images within the target category to guide the adaption of our UniReID. Particularly, for each training category, we randomly sample an image triplet in training set as guidance, and for test categories, we adopt the given support data. Inspired by visual prompt tuning [10], which efficiently adapts deep models into novel tasks by injecting task-relevant prompt vectors, we employ a transformer layer to transfer the knowledge from the support images into a group of learnable vectors. These vectors then act as visual prompts, adapting our backbone model to the target categories. In addition to the implicit guidance from these visual prompts, we also derive semantic knowledge from a large-scale language model as explicit textual guidance. In this step, we construct a natural language template in the form of "Please give me 4 short phrases describing visual clues beneficial for distinguishing [cls] individuals", where [cls] is the category label, to derive knowledge from the GPT-4 model [19]
in a question-answering manner. GPT-4 is a large-scale pre-trained language model with extensive knowledge and remarkable zero-shot question-answering capability. By this means, we could obtain textual descriptions of crucial clues for distinguishing individuals within the target category. Then, we utilize text-guided attention to help our model focus on these clues. As far as we know, this is the first work introducing knowledge from large-scale language models into ReID tasks. Leveraging such dual-modal guidance, our model could rapidly adapt to unseen categories in the open world.
Tasks Related to Our ReID-AW: 1) Re-identification. Re-identification (ReID) aims to match the same object across different camera views. In recent years, numerous studies [24, 6, 31, 27, 22, 14] have been proposed and made significant progress. However, most existing ReID works are specifically designed for a specific category, such as person [6, 27] and vehicle [22, 14], which limits their generalization capability in real-world applications. For instance, many part-based methods [27, 6] assume that pedestrian instances always stand vertically, and they consequently partition pedestrian images into several horizontal stripes to extract local features from head to feet. Although these methods enhance pedestrian re-identification performance, they fail to handle other object categories, such as reptiles, that do not meet the vertically standing assumption. To address this limitation, we propose the ReID-AW task, which aims to design universal re-identification models capable of handling unseen wildlife categories.
2) Domain Generalizable Re-identification. To facilitate the application of ReID models, Song et al. [26] propose the domain generalizable ReID (DGReID) task, which aims to train a ReID model that can work well on unseen environments (domains). Many recent works [26, 12, 4] have been dedicated to addressing the influence caused by style divergence, e.g., viewpoint, illumination, and season variation, across domains to help ReID models generalize to unseen domains. However, DGReID models only consider style divergence and environment variation but fail to account for changes in the target category, which still limits their application in the open world.
3) Few-shot Classification. Few-shot classification [25, 17] aims to train a robust classifier capable of achieving sufficient performance with a limited amount of support (labeled) data. Notably, there are two major differences between our ReID-AW task and few-shot classification. First, classification focuses on category-level distinctions, e.g., distinguishing between elephant and mouse, where inter-class divergence is considerably large. In contrast, in our ReID-AW, we aim to identify instances within the same category where inter-class divergence can be very subtle. This necessitates ReID models to dynamically adapt to target categories and capture their unique discriminative clues. Second, classification models need only to classify input images into a limited number of categories provided by the support data. As a result, many few-shot classification models [25, 17, 1] can achieve classification by directly calculating the feature similarity between support data and input images without specifically understanding these categories. In contrast, ReID models need to learn sufficient knowledge about the target category to support image matching across unlimited unseen identities.
Large-Scale Pre-trained Models. Recently, many works [19, 3, 23] have focused on leveraging abundant training data to endow deep models with comprehensive knowledge and robust representation capabilities. These models have shown their superiority in various vision-language tasks, such as zero-shot classification [23] and visual question answering [19]. For example, CLIP [23] is a lightweight vision-language model trained on 400 million image-text pairs using a contrastive learning strategy, achieving promising zero-shot classification performance. Furthermore, GPT-4 [19] is a robust NLP model trained with 300 billion words, providing it with comprehensive knowledge and remarkable question-answering capabilities. In this work, we aim to derive knowledge from these pre-trained models to facilitate the adaptation of ReID models toward novel categories.
Prompt Tuning. As large-scale pre-trained models have shown their superiority in many tasks, numerous works [3, 10, 13, 2] focus on transferring these models to downstream tasks. For example, Brown et al. [3] propose to use manually designed natural language templates to derive knowledge from the pre-trained language model, GPT-3, and adapt it to different tasks, such as translation and question-answering. Similarly, Jia et al. [10] propose to use a group of task-specific visual prompts to adapt pre-trained vision models to the downstream tasks. In this work, we go one step further to construct category-specific prompts to adapt ReID models to various target categories dynamically.
Figure 2: The sketch of our UniReID model. We use pre-trained CLIP as the backbone model. In our UniReID, the visual guidance generator is responsible for utilizing a triplet of images to construct category-specific visual prompts. The textual guidance generator is designed to derive semantic knowledge about the target category from GPT-4 model. The obtained semantic knowledge is then used to guide our UniReID focus on salient visual content via the text-guided attentive module.
In this work, we propose a novel task named "Re-identify Any Animal in the Wild" (ReID-AW). This task seeks to develop a universal ReID model capable of re-identifying animals from unseen classes  $C^u$  after being trained on multiple seen classes  $C^s$  in our Wildlife-71 dataset. In ReID-AW, ReID models are trained with the training data  $\mathbf{D}^{\mathbf{t}} = \{D^{k}\}_{k = 1}^{C^{s}}$ .  $D^{k} = \{(x_{i},y_{i})\}_{i = 1}^{N_{k}}$  is the labeled training data of the  $k$ -th category, where  $N_{k}$  is the number of training images in this category. After being trained on  $\mathbf{D}^{\mathbf{t}}$ , the ReID model is expected to effectively identify individuals belonging to arbitrary unseen class  $C^u$ . Notably,  $C^u$  and  $C^s$  is non-overlapped, termed  $C^u \cap C^s = \emptyset$ . The statistics of our Wildlife-71 are given in Section 1 in the Appendix. In summary, Wildlife-71 includes 67 training (seen) categories and 4 test (unseen) categories. To help ReID models adapt to unseen categories, we provide a triplet of labeled images for each test category, serving as support data. The image triplet is denoted as  $\{I_a,I_p,I_n\}$ , where  $I_{a}$  and  $I_{p}$  are images of the same identity, distinct from  $I_{n}$ . ReID models are permitted to pre-access these support data for adaptation before commencing inference on the target categories.
The overall framework of our UniReID is depicted in Figure 2. As illustrated in this figure, our UniReID model receives dual-modal guidance, i.e., visual and textual guidance, to facilitate adaptation to the target category. The visual guidance consists of a triplet of images, from which our model could learn to identify and capture visual patterns that differentiate target individuals. For textual guidance, we directly derive explicit semantic knowledge from GPT-4 model to inform our model about the discriminative clues of the target category. There are three major components in our UniReID: 1) visual guidance generator, designed to transfer knowledge from a triplet of images into a group of category-specific visual prompts, which is used to adapt our model to the target category; 2) textual guidance generator, employed to derive semantic knowledge about the target category from the large-scale language model GPT-4; 3) text-guided attentive module, which uses the obtained semantic knowledge to adapt our model to the target category through an attention mechanism. These three modules will be introduced sequentially in the following subsections. Additionally, in our UniReID, we utilize the pre-trained CLIP [23] model as the backbone network for extracting features of both input images and textual guidance. This design is based on two key benefits. Firstly, CLIP
is extensively trained on a large number of images containing various object categories, thereby endowing it with robust pattern recognition capabilities that benefit generalization. Secondly, CLIP is trained on an image-text matching task, enabling its visual and textual encoders to naturally embed image and text inputs into a shared space, which greatly facilitates textual guidance over image features.
Category-specific Sampling Strategy. During the training stage, we employ a category-specific sampling strategy. Specifically, in each mini-batch, we only sample instances from a single randomly selected category for training. The rationale behind this strategy is that emphasizing cross-category distinguishing could guide ReID models to concentrate more on coarse-grained clues while ignoring fine-grained clues that are actually beneficial for intra-category identification.
3.3 Visual Guidance Generator: Category-specific Visual Prompts
The first component of UniReID is the visual guidance generator, designed to use a triplet of images to help our model adapt to target categories. In this step, we find that visual prompt tuning [10] could be an ideal solution, which could efficiently adapt deep models to the target task by introducing task-relevant prompt vectors. Inspired by this, we develop a set of category-specific visual prompts to help our model dynamically adapt to various target categories. Specifically, in our UniReID, we keep all the parameters of the CLIP model frozen (except for normalization layers) and inject it with a set of category-specific visual prompts, which are dynamically constructed based on a triplet of images. Here, for each training category, the image triplet is randomly sampled from the training set, and for test categories, we adopt the support data. As depicted in Figure 2 (1), in this module, we employ a transformer layer to transfer knowledge from the image triplet into a set of learnable vectors, which then serve as category-specific visual prompts. This design has two primary benefits. First, by keeping the parameters unchanged, we can prevent our model from overfitting to seen categories. Second, through the integration of category-specific visual prompts, we can transfer essential knowledge to our model, allowing it to adapt to target categories dynamically.
In practice, given the image triple of target category  $\{I_a,I_p,I_n\}$ , we first send them into a convolutional encoder to extract their features  $\{f_a,f_p,f_n\}$  with  $f_{*}\in \mathbb{R}^{C,H,W}$ . Here,  $C$  indicates channel dimension, and  $(H,W)$  represents spatial scale. Thereafter, we spatially flatten these features and concatenate them together in the flattened spatial dimension. The concatenated features are denoted as  $f_{tri.}\in \mathbb{R}^{C,(H\times W\times 3)}$ , which is then used to construct category-specific visual prompts. In this work, we adopt deep prompt tuning [10]. This method requires us to construct a visual prompt vector for each layer of the CLIP's visual encoder, i.e., ViT [5]. To do so, we use a transformer layer to transfer the knowledge from  $f_{tri.}$  into a group of learnable vectors  $L = \{l^1,l^2,\dots,l^n\}$ , where  $l^i\in \mathbb{R}^{C,20}$  is the prepared prompt vector for the  $i$ -th transformer layer, and  $n$  is the depth of visual encoder. This process could be written as,
$$
[ p ^ {i}, \underline {{\quad}} ] = \operatorname {T r a n s} ([ l ^ {i}, f _ {t r i}. ]), \tag {1}
$$
where  $p^i$  is the output visual prompts; Trans is the transformer layer;  $_-$  is the output of  $f_{tri.}$ , which is discarded; [,] indicates concatenation operation.
By this means, we could obtain category-specific visual prompts  $P = \{p^1, p^2, \dots, p^n\}$ , where  $p^i \in \mathbb{R}^{C,20}$  is the prompt vector for  $i$ -th layer of visual encoder. Then, we follow deep prompt tuning [10] to insert  $P$  into the CLIP's visual encoder to adapt it to the current category. Formally, we represent the input features of the  $i$ -th layer of encoder as  $\{g^i, h_1^i, h_2^i, \dots, h_N^i\}$ , where  $g^i$  is the class token,  $H^i = \{h_1^i, h_2^i, \dots, h_N^i\}$  denote hidden features, and  $N$  is the length of image tokens. In this step, we first insert  $p^i$  into the above feature sequence as  $\{g^i, p^i, h_1^i, h_2^i, \dots, h_N^i\}$ . Then we send them into the  $i$ -th transformer layer of the visual encoder, which could be written as,
$$
[ g ^ {i + 1}, -, H ^ {i + 1} ] = \operatorname {L a y e r} ^ {i} ([ g ^ {i}, p ^ {i}, H ^ {i} ]), \tag {2}
$$
where  $\text{Layer}^i$  is the  $i$ -th layer of visual encoder;  $\_$  is the output of  $p^i$ , which is discarded. By this means, we could adapt our model toward the target category.
3.4 Textual Guidance Generator
In addition to employing implicit guidance from visual prompts, we acquire explicit textual knowledge from pre-trained language models to adapt our model to the target categories. To achieve this, we
design a textual guidance generator to derive explicit textual descriptions toward distinguishable clues of the target category from the GPT-4 model [19]. The GPT-4 is a large-scale pre-trained language model that possesses a wealth of semantic knowledge and boasts impressive zero-shot question-answering performance.
In this part, we follow the approach described in [3] to utilize a natural language template to extract knowledge from the GPT-4 model. Our template is constructed as "Please give me 4 short phrases describing visual clues beneficial for distinguishing [cls] individuals," where [cls] represents the class label of target categories, such as dog, cat, and panda. By populating this template and submitting it to the GPT-4 model, we obtain four textual phrases that outline visual clues useful for distinguishing individuals within the target category, as illustrated in Figure 2 (2). Once we obtain these phrases, we feed them into the textual encoder of the pre-trained CLIP model to extract their features. Notably, to preserve the knowledge of CLIP, we also keep the parameters of its textual encoder frozen. We then pass the extracted features through a trainable transformer layer to adapt them to the current task. The final output textual features are denoted as  $T = \{t^1, t^2, t^3, t^4\}$ , where  $t^j \in \mathbb{R}^{M,C}$  represents the features of the  $j$ -th phrase and  $M$  is the sequence length. These textual features  $T$  are then employed as guidance to help our model focus on salient visual content through an attention-based module.
3.5 Text-guided Attentive Module
In this subsection, we aim to use the acquired textual features  $T = \{t^1, t^2, t^3, t^4\}$  as guidance to assist our model in focusing on visual content that is beneficial in distinguishing individuals within the target category. To accomplish this, we employ a text-guided attentive module to capture salient content over visual features. Formally, given the textual features  $t^j$  of the  $j$ -th textual phrase and patch-level image features  $H = \{h_1, h_2, \dots, h_N\}$  extracted from CLIP, we first apply linear projection to both of these features as,
$$
Q ^ {j} = \operatorname {F C} ^ {Q} \left(t ^ {j}\right), \quad K = \operatorname {F C} ^ {K} (H), \quad V = \operatorname {F C} ^ {V} (H), \tag {3}
$$
where  $\mathrm{FC}^Q$ ,  $\mathrm{FC}^K$ , and  $\mathrm{FC}^V$  are fully connected layers used to embed textual and visual features;  $Q^j$ ,  $K$ , and  $V$  represent Query, Key, and Value. Then we use these vectors to calculate attention as,
$$
\text {A f f i n i t y :} S ^ {j} = \operatorname {S o f t m a x} \left(\frac {Q ^ {j} K ^ {\mathrm {T}}}{\sqrt {d}}\right), \tag {4}
$$
$$
\text {A g g r e g a t i o n :} F _ {t} ^ {j} = S ^ {j} \cdot V,
$$