SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shijie Wang $^{1}$ , Jianlong Chang $^{2}$ , Haojie Li $^{3*}$ , Zhihui Wang $^{1}$ , Wanli Ouyang $^{4}$ , Qi Tian $^{2}$
$^{1}$ International School of Information Science & Engineering, Dalian University of Technology, China
$^{2}$  Huawei Cloud & AI, China
<sup>3</sup>College of Computer and Engineering, Shandong University of Science and Technology, China
$^{4}$ Shanghai Artificial Intelligence Laboratory, China
Open-set fine-grained retrieval is an emerging challenging task that allows to retrieve unknown categories beyond the training set. The best solution for handling unknown categories is to represent them using a set of visual attributes learnt from known categories, as widely used in zero-shot learning. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. Therefore, it is worth to investigate how to transform retrieval models trained by image-level supervision from category semantic extraction to attribute modeling. To this end, we propose a novel Visual Attribute Parameterization Network (VAPNet) to learn visual attributes from known categories and parameterize them into the retrieval model, without the involvement of any attribute annotations. In this way, VAPNet could utilize its parameters to parse a set of visual attributes from unknown categories and precisely represent them. Technically, VAPNet explicitly attains some semantics with rich details via making use of local image patches and distills the visual attributes from these discovered semantics. Additionally, it integrates the online refinement of these visual attributes into the training process to iteratively enhance their quality. Simultaneously, VAPNet treats these attributes as supervisory signals to tune the retrieval models, thereby achieving attribute parameterization. Extensive experiments on open-set fine-grained retrieval datasets validate the superior performance of our VAPNet over existing solutions.
Fine-grained image retrieval attempts to build a well-generalized embedding space where the visual discrepancies among categories are clearly reflected. It plays a vital role in numerous vision applications from fashion industry, e.g., retrieval of different types of shoe or clothes [22; 1], to environmental conservation, e.g., retrieval endangered species [6; 38; 36; 32]. However, real-world applications probably face the input of unknown categories, and the model will treat them as known ones. As a result, the retrieval performance decays, which is unbearable in real-world applications. Open-set fine-grained retrieval is thus proposed to conduct training on known categories but retrieve unknown ones during evaluation.
Facing the unknown inputs of novel categories, an intuitive way to identify them is to capture the discriminative discrepancies of unknown categories from these inputs for identifying them. However, most existing works [19; 24; 28; 20] still focus on discriminative concepts of known categories and only capture them from unknown instances, consequently making it hard to precisely identify unknown categories. Interestingly, zero-shot learning [11; 18; 47] has proven that an unknown instance can be described integrally using a variety of visual attributes, and these attributes can be
*Corresponding author: hjli@sdust.edu.cn
discovered on multiple known categories. For example, the unknown birds in Fig. 1 can be represented using visual attributes discovered from seen instances, and the combination of these attributes can clearly reflect their discrepancies, thus alleviating the problem behind open-set settings. Though important, attribute modeling usually requires significant manual annotations and thus is labor-intensive. When attribute annotations are unavailable, how to transform retrieval models trained by image-level supervision from category semantic prediction to attribute modeling is worthy of investigation.
In this paper, we present a Visual Attribute Parameterization Network, termed as VAPNet, aiming to distill visual attributes from various semantics presented on seen fine-grained objects, and utilize these attributes to tune the retrieval model. Consequently, VAPNet describes the appearance of the
Figure 1: Motivation of the proposed VAPNet as well as the main process of retrieving unknown categories based on our visual attribute parameters. The key idea promoted throughout the paper is that the visual attributes within known categories promote the understanding of unknown inputs of novel categories. The backbone network  $(CNN)$  with this key idea can be transformed into a set of visual attribute parameters  $\theta (\cdot)$ . Therefore, given visually similar images of unknown categories  $(C_A,C_B)$ , we feed them into our VAPNet to generate the retrieval embeddings containing various attributes, and then calculate their cosine similarity (Sim : 0.08) to determine whether to be distinguished. VAPNet with specific parameters transformed by visual attributes can procure in-depth semantic pattern understanding for unknown categories, improving the open-set retrieval performance eventually.
input instances of novel categories based on its parameters tuned by visual attributes, thus transforming the retrieval model from category semantic prediction to attribute modeling. Notably, due to lacking of attribute annotations, the attributes derived from VAPNet will be not restricted to pre-defined attributes like supervised-based attribute learning works [10; 49].
Technically, VAPNet needs to parse various semantics presented in fine-grained objects, which is a prerequisite for attribute modeling. However, a feature extractor trained by image-level labels tends to focus on a few primary semantic regions (e.g., bird's head) while ignoring other visual clues (e.g. bird's body). We empirically observe that vision models can discover these overlooked object regions with rich details when taking local image patches as input compared to the whole image. Therefore, VAPNet attains some rich semantics presented in objects via parsing multiple local views randomly cropped from the input image. After that, we further apply an encoder to project these discovered semantics to a set of visual attributes. Nevertheless, due to lacking the attribute annotations, these attributes usually include some noisy patterns. To handle this limitation, we incorporate the online refinement of these attributes into the training process to iteratively improve their quality and simultaneously regard these attributes as supervision signals to tune the retrieval model, thus achieving attribute parameterization. Specifically, we design another encoder with the same structure as the above one to produce another set of visual attributes from the global features, which are used to match the visual attributes inferred by local views for providing a rich supervisory signal. To avoid optimizing two encoders instead of the retrieval model, we design the counterparts of two encoders by accumulating their parameters of all previous iterations to make supervisory signals provided by attributes tune the retrieval model directly. In this way, the features outputted by the retrieval model can be iteratively improved and fed into the optimized encoders to provide more accurate visual attributes, which, in turn, better tunes the retrieval model for visual attribute modelling.
Contributions of this paper are summarized as below:
- To the best of our knowledge, we are the first to transform the retrieval model trained by image-level supervisions from category semantic prediction into attribute modeling, thus alleviating the problem behind open-set fine-grained retrieval settings.
- We propose a novel Visual Attribution Parameterization Network, which distills visual attributes from various semantics discovered on seen fine-grained objects, and transcribes these attributes into parameters within the retrieval model, thus representing unknown categories precisely based on its parameters transformed by visual attributes.
- Extensive experiments show that open-set fine-grained retrieval task can benefit from the proposed method, and thus our VAPNet obtains significant gains of  $8.6\%$  average accuracy over recent state-of-the-art work [33] on three open-set fine-grained retrieval benchmarks.
Open-set fine-grained retrieval. Existing open-set fine-grained retrieval works can be roughly divided into several groups. The first group, localization-based scheme, utilizes the supervision of category signals to learn discriminative embeddings [42; 52; 24; 40]. CRL [52] designs an attractive object feature extraction strategy to facilitate the retrieval task. Despite the inspiring achievement, the shortcoming of these works is that they only focus on individual samples while neglecting the inter-class and intra-class correlations between subcategories, thus reducing the retrieval performance. Therefore, the second group, metric-based scheme, is learning an embedding space where similar examples are attracted, and dissimilar examples are repelled [33; 41; 4; 15; 27; 51; 14; 50]. NIA [28] enforces unique translatability of samples from their respective class proxies to bring the distance of samples with the same subcategory closer. However, they still capture the discriminative details of known categories from unknown instances but neglect more details on undiscovered semantic regions, consequently impairing the retrieval performance.
Unlike the above works, FRPT [35] steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompting and feature adaptation. PLEor [34] could leverage pre-trained CLIP model to infer the discrepancies encompassing both pre-defined and unknown subcategories, and transfer them to the backbone network trained in the close-set scenarios. Nevertheless, it is worth noting that both of these approaches typically require more computational resources to optimize the retrieval models. This can potentially limit their practical applicability in real-world scenarios. To alleviate the problem behind open-set scenarios, we design VAPNet to explore and exploit visual attributes learnt from known instances instead of learning discriminative clues to anticipate open-set class data, improving retrieval performance in open-world scenarios accordingly.
Visual attributes. Attributes belong to intuitive properties of objects, which contain low-level semantics (e.g., color, texture and shape), high-level semantics (e.g., head, body and tail of objects), or even common sense (e.g., birds living on the tree) [7]. Utilizing visual attributes makes great progress on various vision tasks, including image search [17], fine-grained recognition [49; 37], scene understanding [25], and so on. Most of the previous works based on attribute learning [10; 17; 49] usually require significant manual attribute annotations and therefore is labor-intensive. Besides, the attributes learnt by these works are also restrained to pre-defined attribute labels, consequently ignoring some potentially vital information lying in visual semantics. To alleviate the aforementioned issues, recent works [43; 39] formulate an unsupervised learning strategy to project the learnt features into an attribute space. However, although the two works achieve superior performance on their corresponding vision tasks, they still are rooted in the close-set scenarios and thus make it hard to handle unknown instances. Therefore, we propose VAPNet to process more challenging scenarios, i.e., open-set fine-grained retrieval tasks, by making full use of known data.
The overall structure of VAPNet is shown in Fig. 2. It is clear that our network is mainly organized by three modules: retrieval module, attribute exploration module and attribute parameterization module. The retrieval module could extract retrieval embeddings encompassing various attributes of input objects for retrieving visually similar objects. The attribute exploration module is designed to randomly extract visual attributes from known categories. In addition, the attribute parameterization module is responsible for improving visual attributes and utilizing them as supervisory signals to tune the retrieval model.
3.1 Retrieval Module
The retrieval module aims at extracting basic image representations using the backbone network and producing retrieval embeddings. Thereby, the backbone network can be regarded as the retrieval model. Formally, given an image  $\mathbf{X}$ , let  $\mathbf{F} \in \mathbb{R}^{W \times H \times C}$  be the C-dimensional with  $H \times W$  feature
Figure 2: Detailed illustration of visual attribute parameterization framework. Our algorithm can be clearly divided into three main components: a retrieval module, an attribute exploration module (AEM), and an attribute parameterization module (APM). The AEM is responsible for extracting a set of visual attributes from local views, while the APM is designed to iteratively enhance the quality of these attributes. These attributes are then used as supervisory signals to fine-tune the parameters of the retrieval model. It is important to note that during testing, the single retrieval module acts as a retrieval model. The AEM and APM are only utilized during the training phase to improve the attributes and fine-tune the retrieval model parameters.
planes encoded by a backbone network  $\mathbf{F} = \mathcal{F}_{\mathrm{CNN}}(\mathbf{X})$ . Thus the most common way for retrieval is to embed the final feature  $\mathbf{F}$  by using global average pooling operations (GAP), calculating mean values on the  $\mathrm{H} \times \mathrm{W}$  feature plane and producing the final retrieval embeddings  $\mathbf{E}_{\mathbf{R}} \in \mathbb{R}^{\mathrm{C}}$ . It should be clarified that our VAPNet does not introduce extra computation overhead during evaluation.
3.2 Attribute Exploration Module
Facing the unknown input of novel categories, VAPNet aims to explore all the attributes presented in known instances as much as possible and utilize them to understand unknown categories. A non-negligible problem is that a feature extractor trained by image-level labels tends to focus on a few primary semantic regions (e.g., bird's head) while ignoring other visual clues (e.g. bird's body). Fortunately, a feature extractor could discover object regions with rich details when replacing the input image with its local patches, as verified in Fig. 3. This suggests a proper way to focus on some overlooked object regions by making use of local image patches. Therefore, an attribute exploration module is proposed to attain some semantic clues of an input object via randomly cropping local patches from the input image. These collected semantic clues could be translated into visual attributes describing pre-defined and unknown categories.
Input sampling. Given an input image  $\mathbf{X}$ , we equally split it into  $n \times n$  patches which have  $3 \times \left\lfloor \frac{\mathrm{H}}{\mathrm{n}} \right\rfloor \times \left\lfloor \frac{\mathrm{W}}{\mathrm{n}} \right\rfloor$  dimensions. Here, the granularities of patches are controlled by the hyper-parameter  $n$ . In our experiments,  $n$  respectively equals to 2, 4, 8 and 16, and thus the number of patches is 340. We randomly sample  $M$  ( $M \ll 340$ ) patches from the candidate set at each iteration to construct a set of different local views  $\mathbf{V} = [\mathbf{V}_1, \mathbf{V}_2, \dots, \mathbf{V}_M]$ . These local views are resized to the same scale as inputs via using bilinear interpolation. The magnification operation could directly highlight the subtle yet discriminative details in the local views, further making the backbone network more sensitive to these subtle details.
Then, the local views  $\mathbf{V}$  are passed through the backbone network  $\mathcal{F}_{\mathrm{CNN}}$ :
$$
\mathbf {F} _ {\mathbf {L}} = \mathcal {F} _ {\mathrm {C N N}} (\mathbf {V}), \tag {1}
$$
where  $\mathbf{F}_{\mathbf{L}}\in \mathbb{R}^{M\times C\times H\times W}$  is the local attribute feature set.
It should be clarified that the input sampling step samples local patches randomly, so it can treat all object patches equally, regardless of their discriminative ability. However, it should not be bad news for open-set fine-grained retrieval tasks, since the more diverse the visual attributes are, the
better the understanding of the unknown categories is. Furthermore, although this module picks out a few patches at each iteration, it could effectively parse the holistic structures of objects based on the accumulation of selected patches by multiple iterations.
Attribute exploration. In real-world applications, especially for the fine-grained tasks with small inter-class variances, attribute annotations are expensive and therefore labor-intensive. Nevertheless, unsupervised learning [26] can capture regularities in data for the purpose of extracting useful knowledge or for restoring corrupted data. Therefore, many unsupervised works [26; 2] explicitly produce internal latent units or codes from feature representation. Inspired by this, we design a visual attribute exploration to project these visual semantics attained by local views into a latent space, i.e., the attribute space:
$$
\mathbf {A} _ {\mathbf {L}} ^ {\mathbf {i}} = \mathcal {T} _ {\mathrm {L}} (\mathbf {F} _ {\mathbf {L}} ^ {\mathbf {i}}) = \mathrm {W} _ {\mathrm {L}} \cdot \mathrm {g} (\mathbf {F} _ {\mathbf {L}} ^ {\mathbf {i}}) + \mathrm {b} _ {\mathrm {L}}, \tag {2}
$$
where  $\mathcal{T}_{\mathrm{L}}$  denotes the encoder with the weight matrix  $\mathrm{W_L}\in \mathbb{R}^{c\times k}$  and bias vector  $\mathrm{b_L}\in \mathbb{R}^k$  to process the local features produced by local views, and  $\mathrm{g}(\cdot)$  is the global average pooling operation. By the projection operation, we successfully transform the visual semantics into the visual attributes  $[\mathbf{A}_{\mathbf{L}}^{\mathbf{i}}\in \mathbb{R}^{\mathbf{k}}|\mathbf{i} = 1,\dots ,\mathbf{M}]$ . In this way, these visual attributes could correspond to local semantic knowledge of fine-grained objects (e.g., red head, spotted wings, etc.), so the group of certain attributes can clearly describe an unknown category and reflect its discriminative discrepancies.
3.3 Attribute Parameterization Module
Visual attributes produced by local views serve as supervisory signals to tune the retrieval model, further make it be transformed from category semantic extraction to attribute modelling. Nevertheless, due to lacking the attribute annotations, these visual attributes substantively include some noisy patterns. The retrieval model supervised by these noisy attributes is harmful to parsing fine-grained objects. To this end, we propose an attribute parameterization module to incorporate the online refinement of these attributes into the training process to iteratively improve them and simultaneously regard these attributes as supervisory signals to tune the retrieval model. In this way, the retrieval model could capture visual attributes from input instances, thus achieving attribute parameterization.
Attribute sampling. To match visual attributes provided by local views, we need to process the final features  $\mathbf{F}$  inferred by an input image  $\mathbf{X}$  to produce another group of attributes, which are used to match these visual attributes provided by local views. Concretely, forwarding  $\mathbf{F}$  into a sampler extracts the corresponding local attribute feature sets  $\mathbf{F}_{\mathbf{G}} = [\mathbf{F}_{\mathbf{G}}^{1}, \mathbf{F}_{\mathbf{G}}^{2}, \dots, \mathbf{F}_{\mathbf{G}}^{\mathbf{M}}]$  according to four coordinates of local views. Specifically, we utilize RoIAlign operation proposed in Mask-RCNN [8] to accurately extract the corresponding local features. Then, we project these local representation in another attribute space:
$$
\mathbf {A} _ {\mathbf {G}} ^ {\mathbf {i}} = \mathcal {T} _ {\mathrm {G}} \left(\mathbf {F} _ {\mathbf {G}} ^ {\mathbf {i}}\right) = \mathrm {W} _ {\mathrm {G}} \cdot \mathrm {g} \left(\mathbf {F} _ {\mathbf {G}} ^ {\mathbf {i}}\right) + \mathrm {b} _ {\mathrm {G}}, \tag {3}
$$
where  $\mathcal{T}_{\mathrm{G}}$  represents an encoder with the weight matrix  $\mathrm{W_G} \in \mathbb{R}^{c \times k}$  and bias vector  $b_{\mathrm{G}} \in \mathbb{R}^{\mathrm{k}}$  to process the local features produced by the final features, and  $\mathbf{A}_{\mathbf{G}}^{\mathrm{i}} \in \mathbb{R}^{k}$  denotes the  $i$ -th attribute in the attribute set  $\mathbf{A}_{\mathbf{G}} \in \mathbb{R}^{\mathrm{M} \times \mathrm{k}}$ .
With these visual attributes including  $\mathbf{A_L}$  and  $\mathbf{A_G}$  and local features containing  $\mathbf{F_L}$  and  $\mathbf{F_G}$ , the attribute pairs  $\mathbf{A}$  and local feature pairs  $\mathbf{F_P}$  can be organized as:
$$
\mathbf {A} = \left[ \left(\mathbf {A} _ {\mathbf {L}} ^ {1}, \mathbf {A} _ {\mathbf {G}} ^ {1}\right), \left(\mathbf {A} _ {\mathbf {L}} ^ {2}, \mathbf {A} _ {\mathbf {G}} ^ {2}\right), \dots , \left(\mathbf {A} _ {\mathbf {L}} ^ {\mathrm {M}}, \mathbf {A} _ {\mathbf {G}} ^ {\mathrm {M}}\right) \right],
$$
$$
\mathbf {F} _ {\mathbf {P}} = \left[ \left(\mathbf {F} _ {\mathbf {L}} ^ {1}, \mathbf {F} _ {\mathbf {G}} ^ {1}\right), \left(\mathbf {F} _ {\mathbf {L}} ^ {2}, \mathbf {F} _ {\mathbf {G}} ^ {2}\right), \dots , \left(\mathbf {F} _ {\mathbf {L}} ^ {\mathbf {M}}, \mathbf {F} _ {\mathbf {G}} ^ {\mathbf {M}}\right) \right]. \tag {4}
$$
Attribute parameterization constraint. This constraint is responsible for improving these visual attributes and utilizing them to tune the retrieval model. Specifically, as the local attribute features fed to each encoder come from global and local views, this encoder only distills visual attributes only from its corresponding view. Thus, given local attribute features, no matter which view they come from, if two encoders provide the same attribute, it means this feature can be regarded as from both two sources. In other words, the feature discrepancy between the local attribute features inferred from both global and local views is effectively eliminated. Thereby, the above process could optimize two encoders to iteratively improve visual attributes, and utilize these attributes to modify the features inferred by local and global views, thus achieving attribute parameterization.
To this end, the attribute parameterization constraint  $\mathcal{L}_c$  can be formulated as:
$$
\mathcal {L} _ {\mathrm {c}} (\mathbf {A}, \mathbf {F} _ {\mathbf {P}}) = \sum_ {\mathrm {i} = 1} ^ {\mathrm {M}} \left[ \mathcal {T} _ {\mathrm {G}} \left(\mathbf {F} _ {\mathbf {L}} ^ {\mathrm {i}}\right) \log \frac {\mathcal {T} _ {\mathrm {G}} \left(\mathbf {F} _ {\mathbf {L}} ^ {\mathrm {i}}\right)}{\mathbf {A} _ {\mathbf {L}} ^ {\mathrm {i}}} + \mathcal {T} _ {\mathrm {L}} \left(\mathbf {F} _ {\mathbf {G}} ^ {\mathrm {i}}\right) \log \frac {\mathcal {T} _ {\mathrm {L}} \left(\mathbf {F} _ {\mathbf {G}} ^ {\mathrm {i}}\right)}{\mathbf {A} _ {\mathbf {G}} ^ {\mathrm {i}}} \right]. \tag {5}
$$
This loss encourages the two encoders to produce the same attributes for the same visual content, no matter which view it comes from, thus achieving attribute parameterization. However, training the model with Eq. (5) directly will make the attributes provided by two encoders become similar quickly since the encoders learn the attributes from another view according to Eq. (5). Therefore, using Eq. (5) more optimizes the parameters of two encoders, but has less impact on the parameters of the retrieval model.
To handle this limitation, we propose two mean encoders with the same structure as the above ones to produce attributes for features of another view. In this way, Eq. (5) can be written as