SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\mathcal {L}} _ {\mathrm {c}} (\mathbf {A}, \mathbf {F} _ {\mathbf {P}}) = \sum_ {\mathrm {i} = 1} ^ {\mathrm {M}} [ \operatorname {E} [ \mathcal {T} _ {\mathrm {G}} ] (\mathbf {F} _ {\mathbf {L}} ^ {\mathrm {i}}) \log \frac {\operatorname {E} [ \mathcal {T} _ {\mathrm {G}} ] (\mathbf {F} _ {\mathbf {L}} ^ {\mathrm {i}})}{\mathbf {A} _ {\mathbf {L}} ^ {\mathrm {i}}} + \operatorname {E} [ \mathcal {T} _ {\mathrm {L}} ] (\mathbf {F} _ {\mathbf {G}} ^ {\mathrm {i}}) \log \frac {\operatorname {E} [ \mathcal {T} _ {\mathrm {L}} ] (\mathbf {F} _ {\mathbf {G}} ^ {\mathrm {i}})}{\mathbf {A} _ {\mathbf {G}} ^ {\mathrm {i}}} ], \tag {6}
$$
where  $\operatorname{E}[\mathcal{T}_{\mathrm{G}}]$  and  $\operatorname{E}[\mathcal{T}_{\mathrm{L}}]$  denote the mean encoders without learnable parameters, respectively. Their parameters can be updated in a temporal average manner. Concretely, at the t-th iteration, parameters  $\operatorname{E}[\mathcal{T}_{\mathrm{G}}](\theta_{\mathrm{G}})$  and  $\operatorname{E}[\mathcal{T}_{\mathrm{L}}](\theta_{\mathrm{L}})$  are accumulated by
$$
\mathrm {E} ^ {\left(\mathrm {t}\right)} \left[ \mathcal {T} _ {\mathrm {G}} \right] \left(\theta_ {\mathrm {G}}\right) = (1 - \alpha) \mathrm {E} ^ {\left(\mathrm {t} - 1\right)} \left[ \mathcal {T} _ {\mathrm {G}} \right] \left(\theta_ {\mathrm {G}}\right) + \alpha \theta_ {\mathrm {G}},
$$
$$
\mathrm {E} ^ {(\mathrm {t})} \left[ \mathcal {T} _ {\mathrm {L}} \right] \left(\theta_ {\mathrm {L}}\right) = (1 - \alpha) \mathrm {E} ^ {(\mathrm {t} - 1)} \left[ \mathcal {T} _ {\mathrm {L}} \right] \left(\theta_ {\mathrm {L}}\right) + \alpha \theta_ {\mathrm {L}}, \tag {7}
$$
where  $\mathrm{E}^{(t)}[\mathcal{T}_{\mathrm{G}}](\theta_{\mathrm{G}})$ ,  $\mathrm{(E^{(t)}[T_L](\theta_L))}$  and  $\mathrm{E}^{(t - 1)}[\mathcal{T}_{\mathrm{G}}](\theta_{\mathrm{G}})$ ,  $\mathrm{(E^{(t - 1)}[T_L](\theta_L))}$  respectively denote the parameters of the mean encoders in current iteration and last iteration, and  $\theta_{\mathrm{G}} = (\mathrm{W}_{\mathrm{G}},\mathrm{b}_{\mathrm{G}})$  and  $\theta_{\mathrm{L}} = (\mathrm{W}_{\mathrm{L}},\mathrm{b}_{\mathrm{L}})$  are the learnable parameters of  $\mathcal{P}_{\mathrm{G}}$  and  $\mathcal{P}_L$  at the current iteration, respectively. The mean encoders are initialized as  $\mathrm{E}^{(0)}[\mathcal{T}_{\mathrm{G}}](\theta_{\mathrm{G}}) = \theta_{\mathrm{G}}$  and  $\mathrm{E}^{(0)}[\mathcal{T}_{\mathrm{L}}](\theta_{\mathrm{L}}) = \theta_{\mathrm{L}}$ . The hyper-parameter  $\alpha$  is the updating ratio within the range of [0, 1).
Since the two mean encoders do not introduce learnable parameters, the attribute parameterization constraint could directly penalize the retrieval model and make its parameters be adjusted through back propagation. More importantly, the mean encoders could consider the knowledge learned from all previous stages to form more robust attributes for the current stage. Therefore, they have another important property that could remain sensitive even for rare attributes, consequently staying well generalized when facing unknown categories. After the attribute parameterization operation, VAPNet will transform the retrieval model from category semantic extraction to attribute modeling, allowing the utilization of visual attributes to anticipate open-set class data.
The retrieval model with specific parameters supervised by visual attributes can extract the attributes presented in objects and further procure in-depth semantic pattern understanding. For fine-grained understanding, these extracted visual attributes should clearly reflect the discrepancies of an objects, so that we can better identify visually similar objects. Thereby, we propose an auxiliary constraint based on the cross-entropy loss to ensure that these extracted visual attributes can contribute to decision boundary:
$$
\mathcal {L} _ {\mathrm {a}} = \mathrm {y l o g} (\mathrm {C} (\mathrm {g} (\mathbf {F}))), \tag {8}
$$
where  $y$  denotes the ground-truth label of the corresponding input, and  $C(\cdot) \in \mathbb{R}^{c \times N}$ ,  $N$  is the number of category in the training set.
The total loss  $\mathcal{L}$  of VAPNet can be formulated as:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {a}} + \lambda \hat {\mathcal {L}} _ {\mathrm {c}}, \tag {9}
$$
where  $\lambda$  is the hyper-parameter to balance the contributions of the individual loss item.
Datasets. CUB-200-2011 dataset [5] contains 200 bird subcategories with 11,788 images. We utilize the first 100 classes (5,864 images) in training and the rest (5,924 images) in testing. The Stanford Cars dataset [16] contains 196 car models of 16,185 images. The spilt in Stanford Cars [16] is also similar to CUB, which is split into the first 98 classes (8,054 images) for training and the remaining classes (8,131 images) for testing. FGVC Aircraft dataset [23] is divided into first 50 classes (5,000 images) for training and the rest 50 classes (5,000 images) for testing. In Shop Clothes Retrieval (In-Shop) [21] contains 7,982 subcategories with 52,712 images, and we use the 3,997 classes (25,882 images) in training and the rest 3,985 classes in testing. In-Shop is divided between a query (14,218 images) and a gallery set (12,162 images).
Evaluation protocols. We evaluate the retrieval performance by  $Recall@K$  with cosine distance, which is average recall scores over all query images in the test set and strictly follows the setting in the pioneer work [31]. Specifically, for each query, our model returns the top  $K$  similar images. In the top  $K$  returning images, the score will be 1 if there exists at least one positive image, and 0 otherwise.
Implementation Details. For backbone network, we apply the widely-used Resnet-50 [9] in our experiments with the pre-trained parameters. The input raw images are resized to  $256 \times 256$  and cropped into  $224 \times 224$ . We train our models using Stochastic Gradient Descent (SGD) optimizer with weight decay of 0.0001, momentum of 0.9, and batch size of 32. We adopt the commonly used data augmentation techniques, i.e., random cropping, left-right flipping, and color jittering for robust feature representations. Our model is trained end-to-end on one NVIDIA 2080Ti GPUs for acceleration. The initial learning rate is set to  $10^{-5}$ , with exponential decay of 0.9 after every 5 epochs. The total number of training epochs is set to 200.
The proposed VAPNet is optimized by a combination of two loss functions, an auxiliary loss  $\mathcal{L}_a$  and an attribute parameterization constraint  $\mathcal{L}_c$  or  $\hat{\mathcal{L}}_c$ , which play different roles in guiding our model to understand unknown categories. Here, we perform thorough ablation experiments on CUB-200-2011 and Stanford Cars datasets to further validate the effectiveness of each loss function. Tab. 1 shows quantitative comparisons between different combinations of loss functions. The
Table 1: Comparison of performance and efficiency on CUB200-2011 and Stanford Cars datasets using different combinations of constraints. "R@1" denotes the Recall@1 retrieval performance. "Time" is the time of extracting retrieval embeddings.
La	Lc	ˆc	CUB R@1	CAR R@1	Time
✓	69.5%	89.3%	21.1ms
✓	✓	71.4%	91.2%
✓	74.1%	92.7%	21.1 ms
✓	✓	76.2%	94.8%
baseline method only using  $\mathcal{L}_a$  obtains  $69.5\%$  and  $89.3\%$  Recall@1 accuracy on CUB-200-2011 and Stanford Cars datasets, respectively. The results reflect that the network only learns the discriminative object regions instead of the visual attributes, consequently impairing the retrieval performance of unknown categories. An addition of  $\mathcal{L}_c$  improves Recall@1 from  $69.5\%$  to  $71.4\%$ . However,  $\mathcal{L}_c$  optimizes the encoders to translate the semantics into visual attributes, instead of parameterizing the attributes into the retrieval model. During testing, our VAPNet still makes it hard to handle unknown categories, thus limiting the performance gains. To handle this limitation, we improve the attribute parameterization constraint to force this loss function to directly optimize the parameters within backbone network. As expected,  $\hat{\mathcal{L}}_c$  can effectively make the backbone network sensitive to visual attributes and understand unknown categories accordingly. Furthermore, we also verify the effectiveness of  $\mathcal{L}_a$ , which can ensure that these visual attributes learnt from known categories keep discriminative. As shown in Tab. 1, the proposed VAPNet achieves  $76.2\%$  and  $94.8\%$  Recall@1 performance owing to the combination of  $\hat{\mathcal{L}}_c$  and  $\mathcal{L}_a$  on two widely-used benchmarks. Additionally, during testing, the retrieval embedding extraction time remains the same as that of the baseline model, as the additional attribution exploration modules (AAM and APM) are only used during training.
Table 2: Comparison of different methods on CUB-200-2011, Stanford Cars 196 and FGVC Aircraft datasets.
Method	CUB-200-2011	Stanford Cars 196	FGVC Aircraft
1	2	4	8	1	2	4	8	1	2	4	8
SCDA [42]	57.3	70.2	81.0	88.4	48.3	60.2	71.8	81.8	56.5	67.7	77.6	85.7
PDDM [3]	58.3	69.2	79.0	88.4	57.4	68.6	80.1	89.4	-	-	-	-
CRL [52]	62.5	74.2	82.9	89.7	57.8	69.1	78.6	86.6	61.1	71.6	80.9	88.2
CEP [4]	69.2	79.2	86.9	91.6	89.3	93.9	96.6	98.1	-	-	-	-
HDCL [46]	69.5	79.6	86.8	92.4	84.4	90.1	94.1	96.5	71.1	81.0	88.3	93.3
DGCRL [53]	67.9	79.1	86.2	91.8	75.9	83.9	89.7	94.0	70.1	79.6	88.0	93.0
DCML [50]	68.4	77.9	86.1	91.7	85.2	91.8	96.0	98.0	-	-	-	-
DRML [51]	68.7	78.6	86.3	91.6	86.9	92.1	95.2	97.4	-	-	-	-
DAS [20]	69.2	79.3	87.1	92.6	87.8	93.2	96.0	97.9	-	-	-	-
IBC [29]	70.3	80.3	87.6	92.7	88.1	93.3	96.2	98.2	-	-	-	-
NIA [28]	70.5	80.6	-	-	89.1	93.4	-	-	-	-	-	-
Proxy [13]	71.1	80.4	87.4	92.5	88.3	93.1	95.7	97.5	-	-	-	-
HIST [19]	71.4	81.1	88.1	-	89.6	93.9	96.4	-	-	-	-	-
ETLR [14]	72.1	81.3	87.6	-	89.6	94.0	96.5	-	-	-	-	-
PNCA [33]	72.2	82.0	89.2	93.5	90.1	94.5	97.0	98.4	-	-	-	-
VAPNet	76.2	84.6	90.1	94.0	94.8	96.3	98.0	98.6	87.2	91.7	95.0	96.3
4.3 Comparison with the State-of-the-Art Methods
Open-set Fine-grained Object Retrieval. We compare our VAPNet with some state-of-the-art approaches. In Tab. 2, the performance of different methods on CUB-200-2011, Stanford Cars-196, and FGVC Aircraft datasets is reported, respectively. In the table from top to bottom, the methods are roughly divided into three groups, i.e., localization-based networks, metric-based frameworks, and our VAPNet.
As shown in Tab. 2, it is obvious that the retrieval performance obtained by our VAPNet is better than other methods no matter whether the localization-based or metric-based schemes are adopted. Concretely, existing works based on localization schemes, i.e., CEP [4] and HDCL [46], tend to project the final retrieval embeddings into a category space. Despite the encouraging achievement, the shortcoming of these works is that they only focus on individual samples while neglecting the correlations among subcategories, thus limiting the retrieval performance. To address this problem, the effectiveness of these models based on metric schemes, i.e., ETLR [14] and PNCA [33], can be largely attributed to their precise identification of negative/positive pairs through the manipulation of distances, which indirectly enhances the discriminative power of features. However, these existing works, e.g., CEP [4], HIST [19] and PNCA [33], follow a close-set learning setting, where all the categories are pre-defined, to learn the discriminative and generalizable embeddings for identifying the visually similar objects of unknown subcategories. It is thus very challenging for a feature extractor trained in closed-set scenarios with classification or metric supervisions to capture discriminative discrepancies from unknown subcategories, consequently impairing the retrieval performance.
To handle this limitation, our VAPNet focuses on learning visual attributes instead of discriminative clues to understand the unknown categories and clearly reflect their discriminative discrepancies, thus achieving a clear improvement of state-of-the-art methods.
Large-scale Product Retrieval. Our VAPNet exceeds all the existing methods and achieves the best performance with a retrieval accuracy of  $93.9\%$ , as shown in Tab. 3. Besides, we beat the second-best work CEP [4] and get a better result with a relative accuracy improvement of  $3.0\%$ . By leveraging the visual attributes learned from
Table 3: Comparison of different state-of-the-art methods on In-shop dataset.
method	1	10	20	30	40
HDC [45]	62.1	84.9	89.0	91.2	92.3
ABE [12]	87.3	96.7	97.9	98.2	98.5
EPSHN [44]	87.8	95.7	96.8	-	-
NSM [48]	89.4	97.8	98.7	99.0	-
MS [41]	89.7	97.9	98.5	98.8	99.1
CEP [4]	90.6	98.0	98.6	98.9	99.1
PNCA [33]	90.9	98.2	98.9	99.1	99.4
Our VAPNet	93.9	98.7	99.1	99.4	99.6
known instances to identify category-specific discrepancies, our VAPNet demonstrates impressive generalization capabilities.
Patch number  $M$ . Tab. 4 ablates the role of patch number  $k$  in the attribute exploration module (§3.2). The optimal value is  $M = 4$  (our default). Moreover, VAPNet is robust when  $M$  is in  $[4,\dots ,16]$ , showing that it is beneficial
Table 4: The retrieval accuracy on CUB-200-2011 of model trained with different number  $M$  of local views in §3.2.
Number M	1	2	4	8	16
Recall@1	73.6%	74.9%	76.2%	76.1%	76.2%
to spot visual attributes in a relatively many local regions. It is worth mentioning that when  $M$  is too large, the training time grows exponentially. However, when  $M$  is too small, the performance degrades due to easily overlooking some undiscovered regions. The results reveal that the local regions help the model attain accurate attributes, leading to better understanding unknown categories.
Attribute dimension  $k$ . We investigate the necessity of diverse attribute dimensions  $k$  for retrieval performance. As reported in Tab. 5, the dimension  $k$  stores the attribute knowledge. Although the large dimension could hold more infor
Table 5: Comparison of model trained with different dimension  $k$  of visual attributes on CUB-200-2011.
Dim k	32	64	128	256	512
Recall@1	74.7%	75.9%	76.1%	76.2%	76.0%
mation related to attributes and has less impact on retrieval performance, it is easy to contain more useless information and increase storage overhead. However, when the dimension is small, it is not enough to precisely represent visual attributes, leading to degraded performance. Therefore, the optimal dimension is  $k = 256$ .
Updating ration  $\alpha$ . Tab. 6 reports the accuracy of using diverse updating ratios in Eq. (7). Notably, after increasing the updating ratio, the retrieval performance reduces progressively. These results reveal that a large updating ratio
Table 6: Evaluation results on CUB-200-2011 of model trained with different updating ratio  $\alpha$  in Eq. (7).
Ratio α	0.1	0.2	0.4	0.6	0.8
Recall@1	75.9%	76.2%	75.1%	73.9%	72.6%
quickly updates the projectors more relying on the learning parameters on the current stage, thus easily degrading the discrepancies between two projectors. Moreover, when using a small updating ratio, the projectors keep sensitive to previous learning knowledge and easily keep different during optimization, thus extracting precisely visual attributes from given features.
Balanced parameter  $\lambda$ . There is one balanced parameter in Eq. (10). The sensitivity analysis of the parameter are performed on CUB-200-2011 and the evaluation results are presented in Tab. 7. It is observed that the performances
Table 7: Quantitative performance of model trained with different weight  $\lambda$  in loss function in Eq. (10) on CUB-200-2011.
Weight λ	1	5	10	15	20
Recall@1	73.8%	75.2%	76.2%	75.0%	74.9%
of our VAPNet are not stable with the variation of  $\lambda$  (from 1 to 20). The retrieval performance increases as  $\lambda$  grows to 10, the consistency constraint with a large balanced parameter would force the network to focus on visual attributes. However, this constraint has a larger balanced parameter, which makes the network neglect the discriminative ability of visual attributes, thus reducing retrieval performance. The upper bound of retrieval performance may saturate at  $\lambda = 10$  for learning visual attributes from known categories.
4.5 Visual Attribute Analysis
Interpreting visual attributes is difficult because these attributes are optimized in a latent space. We resort to an indirect way to interpret these attributes by visualizing their sources (i.e., Fig. 3) to display the content within them, and the features influenced by them (i.e., Fig. 4) to indirectly track these visual attributes.
Our VAPNet distills visual attributes from some local regions randomly cropped from inputs. Therefore, we provide some activation maps generated by Grad-CAM [30] to display some visual clues of interest in the attributes. In Fig. 3, these semantics provided by local regions could grab some rich details, and thus attributes projected by them could clearly represent these regions. Besides, we can also observe that the response maps of the local views highlight more object details compared to that
Figure 3: Visualization of the source of attributes. The top row shows the global view. The second and third rows show the multiple image patches after random cropping (local views). (a), (c) and (e) denote the input views. (b), (d) and (f) show the response maps.
Figure 4: Illustration of class activation maps produced by baseline and our VAPNet. (a) and (d) are the input images. (b) and (e) are the referred class activation maps by baseline. (c) and (f) denote the class activation maps provided by our VAPNet.
of the global view. Based on the above observation, we draw a conclusion that vision models can discover more semantic clues when replacing the input image with its local patches.
We exhibit the visualization results to demonstrate the influence of visual attributes. The referred visualizations of baseline and our model are shown in Fig. 4. It is shown that our model focuses on multiple local parts (e.g., head, wings and abdomen, etc.) instead of the fixed part predicted by baseline, (e.g., head of birds, front of cars, and middle of aircraft, etc). This verifies that using visual attributes learnt from known categories reasonably is beneficial for describing novel categories, thus improving the retrieval performance under open-set scenarios. More importantly, as shown in Fig. 4 (c) and (f), two sub-figures in the same row can roughly correspond to certain kinds of attributes of the fine-grained objects, e.g., "wings of birds", "tires of cars", "head or wing of planes", etc. The results reflect that the activation of objects parts is apparently attribute-related and contains the visual discrepancies among unknown categories accordingly, which could provide a clear explanation of the success in retrieving unknown categories.
In this paper, we propose a novel Visual Attribution Parameterization Network (VAPNet) to handle unknown categories using visual attributes learnt from known instances in open-set fine-grained retrieval tasks. VAPNet focuses on distilling visual attributes from semantic clues presented in objects and utilizing these attributes as supervisory signals to tune the retrieval model. In this way, we could transform the retrieval model trained by image-level supervisions from category semantic extraction to attribute modeling, and precisely represent unknown categories based on its parameters supervised by visual attributes. Therefore, VAPNet successfully alleviates the problem behind facing instances from unseen novel categories. Last but not the least, the overall retrieval pipeline is simple and flexible. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods by a significant margin, indicating the effectiveness of attribute modelling on facing unknown categories.
Limitations & Broader Impacts: By introducing VAPNet, we aim to extract visual attributes from seen classes without relying on attribute annotations to differentiate unseen classes. This innovation has the potential to greatly impact open-domain tasks. In particular, annotating a large number of attributes for unseen categories in open-domain tasks can be a costly and time-consuming endeavor. By enabling the model to automatically capture knowledge about unseen classes, our approach reduces the reliance on attribute annotations, resulting in decreased manual labeling costs. Furthermore, our approach exhibits improved adaptability to data from domains resembling the training set, such as natural images or medical images. This heightened adaptability contributes to stronger generalization capabilities, allowing the model to perform well in real-world scenarios. Ultimately, our solution has the potential to propel the advancement of open-domain tasks and facilitate their practical applications.
This work is supported in part by the National Natural Science Foundation of China (NSFC) under Grants (No.61976038 and No.61932020), and The Taishan Scholar Program of Shandong Province (tstp20221128).
[50] Wenzhao Zheng, Chengkun Wang, Jiwen Lu, and Jie Zhou. Deep compositional metric learning. In CVPR, pages 9320-9329. Computer Vision Foundation / IEEE, 2021.
[51] Wenzhao Zheng, Borui Zhang, Jiwen Lu, and Jie Zhou. Deep relational metric learning. In ICCV, pages 12045-12054. IEEE, 2021.
[52] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Feiyue Huang, and Yanhua Yang. Centralized ranking loss with weakly supervised localization for fine-grained object retrieval. In Jérôme Lang, editor, IJCAI, pages 1226-1233. ijcai.org, 2018.
[53] Xiawu Zheng, Rongrong Ji, Xiaoshuai Sun, Baochang Zhang, Yongjian Wu, and Feiyue Huang. Towards optimal fine grained retrieval via decorrelated centralized loss with normalize-scale layer. In AAAI, pages 9291-9298. AAAI Press, 2019.