SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hierarchical Cosine Margin Manner On this basis, the HCM manner constrains the distance distribution between sample pairs to be consistent with the hierarchical distribution of sample labels, so that sample pairs with different grained labels have different gaps in the embedding space, thus obtaining a more refined hierarchical embedding space that is conducive to fine-grained recognition. Specifically, we first group features  $\mathbf{K}$  according to coarse-grained labels and store samples with the same coarse-grained labels in the same group in memory. We keep the latest  $M$  samples for each coarse-grained category in memory. Then, we perform  $k$ -means clustering on each group of features respectively, dividing each group of sample features into smaller groups of clusters. We regard these more subdivided clusters as fine-grained categories under coarse-grained categories and give different clusters different fine-grained pseudo-labels. From this point on, a feature in the embedding space has three levels of granularity labels: instance-level label (each input sample has its own instance-level label, a sample's different augmentations have the same instance-level label), fine-grained pseudo-labels and coarse-grained labels. Therefore, for a feature vector of an anchor sample in  $\mathbf{Q}$ , there are four hierarchical relationships in  $\mathbf{K}$ : same instance-level label, same fine-grained label, same coarse-grained label, and different coarse-grained labels.
The hierarchical cosine distance distribution is demonstrated in Fig. 3(b). From bottom to top, the visual similarity between the samples and the anchor decreases, resulting in larger cosine distances between the samples and the anchor in the embedding space.
This creates a hierarchical cosine distance distribution that follows the same pattern as the visual similarity distribution between categories. Our proposed HCM manner is more refined and complex than the traditional cosine distance constraint, as shown in Fig. 3(a). The traditional cosine distance only needs to consider two types of relationships, i.e., similar or dissimilar, while the HCM manner needs to consider the four hierarchical similarity relationships mentioned above in our setting.
In order to constrain the sample features by the hierarchical cosine distance, we calculate the cosine distance distribution between each feature  $\{q_i\}_{i=1}^{N_B}$  in  $\mathbf{Q}$  and all features  $\{k_j\}_{j=1}^{N_B}$  in  $\mathbf{K}$  by
$$
\boldsymbol {W} _ {i} = \left[ d _ {\cos} \left(\boldsymbol {q} _ {i}, \boldsymbol {k} _ {1}\right), d _ {\cos} \left(\boldsymbol {q} _ {i}, \boldsymbol {k} _ {2}\right), \dots , d _ {\cos} \left(\boldsymbol {q} _ {i}, \boldsymbol {k} _ {N _ {B}}\right) \right], \tag {5}
$$
where  $d_{\mathrm{cos}}(\pmb {u},\pmb {v}) = 1 - \cos (\theta) = 1 - \frac{\pmb{u}\pmb{v}}{\|\pmb{u}\| \|\pmb{v}\|} \in [0,2]$  is a cosine distance function, and  $N_B$  is the training batch size. As shown in Fig. 3(b), the distance between two samples in the embedded space becomes larger as the category gap between them increases. For example, when the relationship between sample pairs is the same physical object, different objects of the same kind, different species of the same genus, and different genera of the same family, the sample pairs become less and less
similar. To simplify the model, we set the distances between sample pairs with the same sample, the same fine-grained category, the same coarse-grained category, and different coarse-grained categories as  $d_0$ ,  $d_1$ ,  $d_2$ , and  $d_3$ , respectively, where  $d_0 < d_1 < d_2 < d_3$ . We obtain the target semantic distance between each feature  $\{q_i\}_{i=1}^{N_B}$  in  $\mathbf{Q}$  and all features  $\{k_j\}_{j=1}^{N_B}$  in  $\mathbf{K}$  for each sample pairs by
$$
\boldsymbol {M} _ {i} = \left[ d _ {t} \left(\boldsymbol {q} _ {i}, \boldsymbol {k} _ {1}\right), d _ {t} \left(\boldsymbol {q} _ {i}, \boldsymbol {k} _ {2}\right), \dots , d _ {t} \left(\boldsymbol {q} _ {i}, \boldsymbol {k} _ {N _ {B}}\right) \right], \tag {6}
$$
where  $d_t(\pmb{u}, \pmb{v}) = \left\{ \begin{array}{ll} d_0 & \text{if } \pmb{u}, \pmb{v} \text{ are same instance-level labels,} \\ d_1 & \text{else if } \pmb{u}, \pmb{v} \text{ are same fine-grained pseudo labels,} \\ d_2 & \text{else if } \pmb{u}, \pmb{v} \text{ are same coarse-grained labels,} \\ d_3 & \text{otherwise.} \end{array} \right.$ . To ensure that the
feature distance distribution of samples in the embedding space aligns with the hierarchical class semantic similarity, we approximate the distributions of the pairwise feature cosine distances of samples and targets using the Kullback-Leibler (KL) divergence
$$
\mathcal {L} _ {h c m} = \frac {1}{N _ {B}} \sum_ {i = 1} ^ {N _ {B}} \operatorname {K L} \left(\boldsymbol {M} _ {i}, \boldsymbol {W} _ {i}\right). \tag {7}
$$
Finally, we combine Eq. (4) and Eq. (7) to obtain the overall loss during the training process
$$
\mathcal {L} = \mathcal {L} _ {\text {c l s}} + \alpha \mathcal {L} _ {\text {h c m}}, \tag {8}
$$
where  $\alpha$  is a trade-off hyperparameter.
During testing, we utilize the trained  $f_{embed}(\cdot)$  to embed the support set and query set samples into the hyperbolic space. In the following, we classify the query samples by computing their distance to the support samples through the  $k$ -nearest neighbors method.
Adaptive Hierarchical Cosine Distance In Eq. (6), we use  $[d_0, d_1, d_2, d_3]$  in  $d_t(\cdot)$  as the target values of the feature cosine distance distribution, which should be determined by the intrinsic hierarchy in different datasets, so we propose an Adaptive Hierarchical Cosine Distance (AHCD) strategy to update the target values during the training process according to the training data. Specifically, we set the target value  $d_0$  of samples with the same instance-level label to 0, and the target value  $d_3$  of samples with different coarse-grained labels to 1 which correspond to angles of  $90^\circ$ . We initialize  $d_1$  and  $d_2$  to 0.134 and 0.5 which correspond to angles of  $30^\circ$  and  $60^\circ$ , respectively, and using  $d_0$  and  $d_3$  as anchor values to uniformly distribute them. During training, we calculate the average feature cosine distance between different examples with the same grained label in each batch of data
$$
\bar {d} _ {l} = \frac {1}{N _ {l}} \sum_ {\boldsymbol {q} _ {i}, \boldsymbol {k} _ {j} \in \Omega_ {l}} ^ {N _ {l}} d _ {c o s} (\boldsymbol {q} _ {i}, \boldsymbol {k} _ {j}), \tag {9}
$$
where  $l \in \{1,2\}$  respectively represent fine-grained label hierarchy and coarse-grained hierarchy (i.e.,  $\bar{d}_1$  and  $\bar{d}_2$  respectively represent average distance of sample pairs with same fine-grained label and same coarse-grained label),  $\Omega_l$  represents a set of the sample pairs that belong to same grained labels,  $N_l$  is the size of  $\Omega_l$ . Then, we update the target semantic distance by momentum update
$$
d _ {l} = \beta d _ {l} + (1 - \beta) \bar {d} _ {l}. \tag {10}
$$
We use the momentum update method to dynamically update the target semantic cosine distance of sample pairs with the same fine-grained label and the same coarse-grained label, so that  $d_{1}$  and  $d_{2}$  can adjust values adaptively according to the distribution of data in different datasets.
Benchmark Datasets Table 1 summarizes the benchmark datasets CIFAR-100 [16] and BREEDS [23]. The BREEDS dataset includes four subsets derived from ImageNet, namely LIVING-17, NONLIVING-26, ENTITY-13, and ENTITY-30, each of which has a class hierarchy calibrated to ensure that classes at the same level have similar visual granularity.
Table 1: Summaries of the benchmark datasets. L-17, NL-26, E-13, E30 are the LIVING-17, NONLIVING-26, ENTITY-13, ENTITY-30 sub-datasets from BREEDS [23].
Datasets	L-17	NL-26	E-13	E-30	CIFAR-100
# Coarse classes	17	26	13	30	20
# Fine classes	68	104	260	240	100
# Train images	88K	132K	334K	307K	50K
# Test images	3.4K	5.2K	13K	12K	10K
Image resolution	224	224	224	224	32
Table 2: Comparisons on BREEDS [23]. Red and blue bold numbers are the top two best results.
Methods	LIVING-17	NONLIVING-26	ENTITY-13	ENTITY-30
5-way	all-way	5-way	all-way	5-way	all-way	5-way	all-way
Fine upper-bound	90.75±0.48	62.65±0.18	90.33±0.47	60.68±0.14	94.72±0.33	65.18±0.09	94.02±0.36	63.72±0.10
MoCo-v2	56.66±0.70	18.57±0.11	63.51±0.75	21.07±0.11	82.00±0.67	33.06±0.07	80.37±0.62	28.62±0.06
MoCo-v2-ImageNet	82.21±0.73	40.29±0.14	77.07±0.78	34.78±0.13	85.24±0.60	35.62±0.08	83.06±0.62	31.73±0.08
SWAV-ImageNet	79.83±0.65	38.79±0.15	76.26±0.71	33.94±0.11	81.15±0.65	33.57±0.07	79.91±0.54	31.15±0.07
ANCOR	89.23±0.55	45.14±0.12	86.23±0.54	43.10±0.11	90.58±0.54	42.29±0.08	88.12±0.54	41.79±0.08
ANCOR-fc	90.41±0.57	46.19±0.16	88.77±0.54	45.34±0.13	89.05±0.58	38.52±0.08	91.84±0.49	42.33±0.10
SCGM-G	89.72±0.54	48.74±0.15	89.87±0.51	49.25±0.13	90.15±0.51	40.00±0.08	92.90±0.46	42.17±0.08
SCGM-A	90.97±0.55	49.31±0.16	88.78±0.55	46.93±0.13	88.48±0.59	41.07±0.09	91.22±0.51	44.14±0.09
Ours	90.94±0.43	53.09±0.11	89.97±0.42	50.12±0.11	91.24±0.38	41.64±0.09	92.95±0.40	44.53±0.09
Table 3: Comparisons on CIFAR-100 [16]. Red and blue bold numbers are the top two best results.
Methods	5-way	all-way
Fine upper-bound	75.53±0.68	31.35±0.11
ANCOR	74.56±0.70	29.84±0.11
ANCOR-fc	74.73±0.73	27.32±0.10
SCGM-G	76.19±0.73	29.92±0.11
SCGM-A	77.37±0.77	25.91±0.10
Ours	81.42±0.69	36.28±0.12
Table 4: Comparisons of intra-class (all fine-grained class recognition within a random coarse class) on the BREEDS [23]. Red and blue bold numbers are the top two best results.
Methods	Living17	Nonliving26	Entity13	Entity30
Fine upper-bound	70.72±0.92	74.02±0.91	72.24±0.60	73.86±0.67
ANCOR	48.77±0.71	49.64±0.88	42.00±0.47	45.17±0.59
ANCOR-fc	51.07±0.82	53.51±1.00	41.83±0.48	47.82±0.65
SCGM-G	53.48±0.81	57.32±1.04	43.89±0.58	46.80±0.78
SCGM-A	53.88±0.90	55.12±1.00	45.09±0.58	50.02±0.71
Ours	57.11±0.93	58.63±1.01	44.85±0.54	51.24±0.65
Baselines We compare PE-HCM with the most relevant state-of-the-art models on embedding learning: (1) MoCo-v2 [5], trained on the above datasets; (2) MoCo-v2-ImageNet [5], pre-trained on the official full ImageNet; (3) SWAV-ImageNet, pre-trained by Caron [3]; (4) ANCOR [2], combine supervised and self-supervised contrastive learning with their proposed Angular normalization module; (5) SCGM [18], based on the superclass-conditional Gaussian mixture model. (6) 'Fine upper-bound' [2], naturally trained on the fine-grained labels.
**Implementations** For fair comparisons with ANCOR [2] and SCGM [18], we use ResNet-12 [12] and ResNet-50 [12] as the backbone network for CIFAR-100 [16] and BREEDS [23]. We use a three-layer MLP as the projector, and the exponential mapping follows it to embed the features in the hyperbolic space. The input, hidden, and output layer dimensions of the MLP are  $a \to a \to b$ , where  $a$  is 640 and 2048 for ResNet-12 and ResNet-50, respectively, and the output layer dimension  $b$  is 128, following ANCOR and SCGM. For the hyperparameters, we set  $c = 0.001$  in Eq. (1),  $\alpha = 800$  in Eq. (8),  $\beta = 0.999$  in Eq. (10). We used the Adam optimizer to train the model on 4 GeForce RTX 3090 Ti GPUs, and training a total of 200 epochs. For CIFAR-100 and BREEDS, the batch size were 1024 and 256, the initial learning rates were  $5 \times 10^{-4}$  and  $1.25 \times 10^{-4}$ , the learning rates were reduced by 10 times when the epoch was 120 epoch and 160 epoch. In order to prevent the distance distribution from getting stuck in local optima, we reinitialize  $d_{1} = 0.134$  and  $d_{2} = 0.5$  at the 120-th epoch and 160-th epoch. We followed ANCOR [2] to implement random data enhancement using random resized crop, random horizontal flipping, random color Jitter, random grayscale, and random Gaussian smoothing during training. We evaluate the performance using 5-way and all-way 1-shot settings during testing. The evaluation is conducted on 1000 random episodes, and we report the mean accuracy along with the  $95\%$  confidence interval.
Main Results Table 2 and Table 3 present the average accuracy rates for fine-grained learning from coarse labels on BREEDS and CIFAR-100. For each dataset, we report both 5-way and all-way fine-grained recognition results during testing. As shown in these tables, our proposed model is significantly better than other baseline methods on the above datasets. Especially on CIFAR-100, we achieve  $4.05\%$  and  $6.36\%$  improvements over state-of-the-art methods, even surpassing the fine upper-bound baseline significantly. Table 4 evaluates an intra-class case when all fine-grained classes of random coarse-grained classes were sampled in each episode. The results demonstrate that our method can better distinguish fine-grained categories in the embedding space.
Retrieval Task Results In addition to evaluating the ability of learning fine-grained embeddings from coarse labels in few-shot learning tasks, Grafit [29] and MaskCon [8] also investigated the effectiveness of the 'coarse to fine' approach by using retrieval tasks for validation. As shown in Table 5, we conducted fair comparisons of our method with these state-of-the-art approaches, using
Table 5: Comparisons on CIFAR-100. Red and blue bold numbers are the top two best results.
Methods	5-way	All-way	Recall@1	Recall@5	mAP
Graft (Reported in paper)	-	-	60.57	82.32	-
MaskCon (Reported in paper)	-	-	65.52	83.64	-
MaskCon (Our re-implementation)	81.05	40.81	65.35	83.57	60.27
Ours	85.45	47.76	62.11	80.68	62.04
the same backbone and epoch settings as MaskCon. Our method still achieves superior results over Grafit and MaskCon (almost  $7\%$  improvement on all-way FSFG) under FSFG, and achieves a mean Average Precision (mAP) of  $62.04\%$ , surpassing MaskCon's  $60.27\%$ . This also reinforces the efficacy of our approach. Additionally, it is worth noting that our method is lower than MaskCon when we used Recall@k as an evaluation metric (by following MaskCon). The reason might be that image retrieval involves identifying relevant images from a gallery of diverse images, usually by utilizing a query image as a reference. In this scenario, the task entails finding any images belonging to the same category as the query image from the gallery, which might contain multiple images per category. On the other hand, few-shot recognition, exemplified by the one-shot scenario, presents a more challenging scenario. In this setting, the goal is to recognize objects/classes with very limited training examples, often relying on only a single image per category. This constraint inherently magnifies the difficulty of the task.
Proposals in Our Method As shown in Table 6, we explored the improvement of each module in our method on LIVING-17 and CIFAR-100. The results of the first two rows show that the hyperbolic space is more suitable for the fine-grained embedding from coarse labels with a hierarchical structure, which is consistent with the conclusion shown in [11, 15]. The results of the last three rows show that the introduction of our HCM manner and AHCD strategy in the hyperbolic space can bring great improvement. This indicates that all proposals of our method can better learn the discriminative patterns between finer-grained categories from coarse labels.
Table 6: Comparisons of our proposals on LIVING-17 [23] and CIFAR-100 [16]. E: Coarse-grained supervised learning in the Euclidean space. H: Coarse-grained supervised learning in the hyperbolic space. HCM: Our proposed Hierarchical Cosine Margin manner. AHCD: Our proposed Adaptive Hierarchical Cosine Distance strategy.
Methods	LIVING-17	CIFAR-100
E	H	HCM	AHCD	5-way	all-way	5-way	all-way
✓	87.72	32.71	76.34	24.92
✓	88.06	33.78	76.62	26.19
✓	✓	90.02	46.53	77.64	33.22
✓	✓	✓	90.94	53.09	81.42	36.28
Visualization As shown in Fig. 4, it can be observed that compared to training the model using only coarse labels, our method leads to a more concentrated distribution of sample points for each category, while also accentuating the distinctions between different classes. The visualization validates the effectiveness of our method of contributing to the discriminative power from the qualitative aspect.
Figure 4: Visualization by  $t$ -SNE [13] on CIFAR-100. Left: training the model using only coarse labels. Right: our method.
Adaptive Ability We extensively explored the effective-
ness of the AHCD strategy. As shown in Fig. 5, we tracked the update processes of  $d_{1}$  and  $d_{2}$  during training on CIFAR-100 and LIVING-17 datasets. In general, there are the same trends and different details. In the early stages of training, as the model lacks discriminative ability, the values of  $d_{1}$  and  $d_{2}$  gradually decrease. As the model continues to train, it gradually acquires discriminative ability, resulting in an increase in the values of  $d_{1}$  and  $d_{2}$  followed by stabilization. From the final stable results, we observe that the margin between coarse-grained categories  $(d_{2})$  is larger than the margin between fine-grained categories  $(d_{1})$ , and the margin between fine-grained categories on CIFAR-100 is larger than the margin between fine-grained categories on LIVING-17, which aligns with the true category relationships in the data distribution. This demonstrates that our proposed adaptive strategy can adjust  $d_{1}$  and  $d_{2}$  during training according to the actual data distribution.
Hyperparameters As shown in Fig. 6, we changed the trade-off hyperparameter  $\alpha$  of the hierarchical cosine loss in Eq. (8) on the CIFAR-100 and LIVING-17 datasets to observe its impact for the final result. It can be seen that, as  $\alpha$  increases exponentially, the performance of the model increases
Figure 5: The update processes of  $d_{1}$  (left) and  $d_{2}$  (right) in adaptive hierarchical cosine distance strategy on CIFAR-100 and LIVING-17.  $d_{1}$  and  $d_{2}$  are the target distance for the same fine-grained and coarse-grained sample pairs.
Figure 6: Comparisons with different value of  $\alpha$  on CIFAR100 and LIVING-17.
first and then decreases. The value of  $\alpha$  is also sensitive, with a smaller value of  $\alpha$  limiting the discriminative hierarchical embedding ability of the HCM manner, and a larger value of  $\alpha$  suppressing the generalization ability to fine-grained categories.
As shown in Fig. 7, we change the number of  $k$ -means clusters for each coarse-grained class in the HCM manner on CIFAR-100 and LIVING-17, and report the recognition results for all fine-grained categories (all-class) and intra-coarse class fine-grained categories (intra-class). From the experimental results, it can be observed that the number of clusters has a significant impact on the fine-grained recognition performance. When the number of clusters is small, the model performs poorly in fine-grained recognition tasks. Having a larger number of clusters than the actual number of fine-grained categories leads to better results. Additionally, in our experiments across five datasets, we found a recurring pattern that offers a guideline. Typically, when clustering coarse-grained categories, the number of clusters that yields optimal results is roughly twice the
Figure 7: Comparison results in 1-shot learning with different numbers of clusters on CIFAR-100 and LIVING-17.
actual number of fine-grained subclasses within that coarse category. While this heuristic emerged consistently across our tested datasets, we acknowledge that it may not be universally optimal. However, it does provide a good starting point for users. In practice, fine-tuning based on validation performance is always recommended to ascertain the best hyperparameter for a specific task.
We proposed a novel method (PE-HCM) for fine-grained learning from coarse labels. We used the hyperbolic space to embed the samples and enhanced the discriminative ability with a hierarchical cosine margins manner. On the one hand, we performed supervised learning using coarse-grained labels to distinguish coarse-grained category samples initially. On the other hand, we constructed instance-level labels and fine-grained pseudo-labels using data enhancement and clustering methods, for the hierarchical cosine margins manner constraining the distance between sample pairs of four relationships, which resulted in a more refined hierarchical division of distances for fine-grained learning. By optimizing the hierarchical cosine distance using the hierarchical cosine margins manner, the learned embedding could be well generalized to fine-grained visual recognition tasks. Experiments on five benchmark datasets demonstrated the effectiveness of the proposed method.
Throughout our experiments, we have occasionally observed a particular scenario where our method yields comparatively lower results. This typically occurs when classes within the same hierarchy level exhibit substantial variations in their scopes. For instance, when some classes possess a significantly broader scope while others have a notably narrower scope. We believe this phenomenon arises from the fact that our method employs a uniform margin across all levels. In the future, more refined category-specific semantic hierarchical distance constraints are worth further study, that is, further constraining the distances between sample pairs of different categories within the same grain of granularity to make them more consistent with the real data distribution.