SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shu-Lin Xu $^{1}$ , Yifan Sun $^{2}$ , Faen Zhang $^{3}$ , Anqi Xu $^{4}$ , Xiu-Shen Wei $^{1*}$ , Yi Yang $^{5}$
$^{1}$ School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University
$^{2}$ Baidu Inc.  $^{3}$ AlInnovation Technology Group Co., Ltd  $^{4}$ University of Toronto
$^{5}$ CCAI, College of Computer Science and Technology, Zhejiang University
Learning fine-grained embeddings from coarse labels is a challenging task due to limited label granularity supervision, i.e., lacking the detailed distinctions required for fine-grained tasks. The task becomes even more demanding when attempting few-shot fine-grained recognition, which holds practical significance in various applications. To address these challenges, we propose a novel method that embeds visual embeddings into a hyperbolic space and enhances their discriminative ability with a hierarchical cosine margins manner. Specifically, the hyperbolic space offers distinct advantages, including the ability to capture hierarchical relationships and increased expressive power, which favors modeling fine-grained objects. Based on the hyperbolic space, we further enforce relatively large/small similarity margins between coarse/fine classes, respectively, yielding the so-called hierarchical cosine margins manner. While enforcing similarity margins in the regular Euclidean space has become popular for deep embedding learning, applying it to the hyperbolic space is non-trivial and validating the benefit for coarse-to-fine generalization is valuable. Extensive experiments conducted on five benchmark datasets showcase the effectiveness of our proposed method, yielding state-of-the-art results surpassing competing methods.
In deep learning, rich supervised information is crucial for the generalization ability of deep models. Fine-grained visual recognition [36] is a fundamental problem in computer vision, which requires accurately identifying subordinate (fine-grained) categories within the same meta (coarse-grained) category. However, in many domains, annotating fine-grained data requires the involvement of domain experts, such as in pipeline failure detection [1], biodiversity recognition [30] and medicine analyses [25], leading to high annotation costs or even infeasibility. In contrast, obtaining the coarse labels, e.g., the meta category of a species in biodiversity recognition, is much easier. In such cases, we attempt to use coarse-grained labels to train models and apply them to finer-grained recognition tasks. Such a task is challenging as it lacks the detailed supervisory information required for fine-grained tasks, preventing us from obtaining sufficiently detailed label information to train models to recognize fine-grained features and distinctions. In this paper, we attempt to train models using coarse-grained labels and apply them to challenging few-shot fine-grained recognition tasks, which are highly demanding and have practical significance.
*Corresponding author. This work was supported by National Key R&D Program of China (2021YFA1001100), National Natural Science Foundation of China under Grant (62272231), Natural Science Foundation of Jiangsu Province of China under Grant (BK20210340), the Fundamental Research Funds for the Central Universities (NJ2022028), and CAAI-Huawei MindSpore Open Fund.
To tackle these challenges, we propose a novel method termed Poincaré embedding with hierarchical cosine margins (PE-HCM). Specifically, PE-HCM embeds samples into a Poincaré embedding (which is also known as an embedding in the hyperbolic space [21]) and enhances their discriminative ability with a hierarchical cosine margin manner. Our PE-HCM has two coupled features, i.e., hyperbolic space and hierarchical cosine margins:
- Hyperbolic space. We first embed the samples into the hyperbolic space instead of the regular Euclidean space, as the hyperbolic space provides stronger expressive power for hierarchical relationships [19]. Since the tasks require an embedding space that can not only capture the coarse-grained information provided during training but also generalize well to fine-grained categories during the test, the embedding space should cover multiple granularities from coarse to fine. The hyperbolic space well satisfies this prerequisite with its natural hierarchical structure.
- Hierarchical cosine margins. Based on the hyperbolic space, we further incorporate hierarchical cosine distance constraints to impose a hierarchical proximity relationship among sample pairs. We construct finer-grained labels than the coarse-grained training labels through unsupervised approaches such as data augmentation and clustering. Specifically, similar to SimCLR [4], we apply two sets of data augmentations to a batch of data. For the same sample, different augmentations are assigned with the same instance-level label, while different samples are assigned with different instance-level labels. By clustering the samples within a coarse-grained category, we obtain multiple clusters that represent the fine-grained categories between the coarse-grained and the instance-level categories. As a result, we have instance-level, fine-grained, and coarse-grained labels. Correspondingly, there are four types of pairwise relationships: same instance-level category, same fine-grained category, same coarse-grained category, and different coarse-grained categories.
Moreover, to derive the appropriate target distances that will be used for different levels, we propose an adaptive strategy to update the target distances during training. It utilizes the average distance between sample pairs in each batch to update the target cosine distances with momentum. The adaptive strategy plays a crucial role in ensuring that the target distances reflect the real data distribution. By dynamically adjusting the target distances during training, our method can better capture the underlying relationships and adapt to the fine-grained characteristics of the data. In a batch of training data, we derive the target cosine distance distributions for the pairwise relationships based on their category relationships in the hyperbolic embedding space. By enforcing the consistency between the feature distribution and the category relationship branch, we enhance the hyperbolic embedding space's generalization and discriminative capability for implicit finer-grained categories.
In experiments, we perform PE-HCM on five popular benchmark datasets, i.e., CIFAR-100 [16] and four sub-datasets {LIVING-17, NONLIVING-26, ENTITY-13, ENTITY-30} from BREEDS [23]. Our method has achieved state-of-the-art recognition accuracy on these datasets, thereby demonstrating its effectiveness and its potential for practical applications.
In summary, our major contributions are three-fold:
- We propose a novel method that addresses the challenging task of fine-grained learning from coarse-grained labels. It bridges the gap between coarse-grained and fine-grained labels and leverages the knowledge learned from coarse-grained categories for fine-grained recognition tasks.
- We develop a hyperbolic embedding with hierarchical cosine margins (HCM). HCM enforces relatively large/small similarity margins between coarse/fine classes. The key point of HCM, i.e., the target similarity margins, are dynamically updated using an adaptive strategy.
- We conduct comprehensive experiments on five popular benchmark datasets, and our proposed method achieves superior recognition accuracy over competing solutions on these datasets.
Coarse and Fine Learning Coarse and fine learning has been an important topic in computer vision and machine learning. Numerous methods [2, 7, 8, 26, 27, 29, 35, 40, 42] and theoretical studies [10] have been proposed to address this problem, with the goal of leveraging coarse-grained labeled data to improve fine-grained recognition. On the one hand, several methods have been proposed to tackle the coarse and fine learning problem. For instance, Stretcu et al. [26] proposed a coarse-to-fine curriculum learning method that can dynamically generate training sample sequences
based on task difficulty and data distribution, thereby accelerating model convergence and improving generalization ability. Xiang et al. [40] proposed a coarse-to-fine incremental few-shot learning method that can use coarse-grained labels to perform contrastive learning on the embedding space and then used fine-grained labels to normalize and freeze the classifier weights, thereby solving the class incremental problem. Sun et al. [27] developed a dynamic metric learning method that can adaptively adjust the metric space according to different semantic scales, thereby improving the performance of multi-label classification and retrieval. Cui et al. [7] designed a coarse-to-fine pseudo-labeling guided meta-learning method that can use coarse labels to generate pseudo-labels, and updated model parameters through meta-optimizer, thereby achieving fast adaptation in few-shot classification tasks. Bukchin [2] proposed a fine-grained angular contrastive learning method with coarse labels that can use coarse labels as prior knowledge to constrain the angular loss function, and constructed positive and negative sample pairs through random sampling and data augmentation, thereby achieving robust feature representation in fine-grained image recognition tasks. On the other hand, Fotakis et al. [10] provided a theoretical analysis of the generalization error of learning with coarse labels, which can recover the true distribution statistically without requiring additional information or assumption conditions.
Overall, previous work on coarse and fine learning demonstrates the significance of this problem and highlights the potential benefits of leveraging coarse-grained labeled data for fine-grained recognition.
Few-Shot Learning Few-shot learning [9, 20, 24, 28, 31, 33, 37-39, 41, 44-46] has become a popular research direction in computer vision due to the difficulty of collecting and annotating large datasets. The goal of few-shot learning is to enable models to quickly acquire knowledge from a small number of new samples, which is particularly suitable for fine-grained recognition as fine-grained level annotations are often costly. To address this problem, various methods have been proposed, which can be broadly categorized into three types: metric-based methods, optimization-based methods, and attention-based methods. In concretely, metric-based methods, e.g., prototypical networks [24], relation networks [28], matching networks [31], and feature generating networks [39], learn a metric space where the similarity between support and query samples is computed. These methods aim to learn a feature representation that captures the discriminative information of categories and a distance metric that can compare samples in this feature space. Optimization-based methods, e.g., model-agnostic meta-learning [9] and piecewise classifier mappings [37], aim to learn a model that can quickly adapt to new categories with only a few examples. These methods learn an initialization that can be fine-tuned quickly on a new task with a small amount of data. Attention-based methods, e.g., MultiAtt [33], MattML [46] and Dual Att-Net [41], use attention mechanisms to identify the most informative parts or regions of the input images. These methods aim to learn a feature representation that is not only discriminative but also informative for few-shot recognition.
In summary, few-shot learning has been extensively studied in recent years, and various methods have been proposed to tackle this problem. While the aforementioned papers have shown success in addressing the problem, they still rely on a considerable amount of labels with the same granularity as the testing granularity during training. In contrast, our work aims to learn fine-grained recognition only from coarse labels during training, which is a more challenging task.
Hyperbolic Geometry In the field of deep learning, the hyperbolic space was first proposed by Nickel and Kiela [19] to learn hierarchical representations of symbolic data such as text and graphs by embedding them into an  $n$ -dimensional Poincaré ball, which showed that hyperbolic embeddings can capture the hierarchical relationships in knowledge graphs more effectively than Euclidean embeddings. Similarly, Ganea et al. [11] proposed a hyperbolic neural network for modeling tree-structured data, which was observed to outperform Euclidean-based models. In addition, Sala et al. [22] utilized the hyperbolic space to embed words in natural language processing tasks. Khrulkov et al. [15] proposed a hyperbolic image embedding method to represent hierarchical structures in images.
Compared to the Euclidean space which has a constant sectional curvature of 0, the hyperbolic space has a constant negative curvature, which enables more efficient low-dimensional embedding for modeling hierarchical structure property data [19]. As illustrated in the left of Fig. 1, in this Poincaré disk, the exponential growth of distances matches the exponential growth of nodes with the tree depth [11]. This property has been shown to be particularly powerful for representing tree-structured data, where the hierarchy of the data can be more naturally captured in the hyperbolic space.
Figure 1: Left: An example of modeling a regular tree using a 2D Poincaré disk, where all connected nodes are equally spaced apart. Middle: Decision boundary in a Poincaré disk, where  $p_1$ ,  $a_1$  are a point and a normal vector determining the decision boundary, and the region bounded by the arc and the disk represents a category. C1, C2, C3 and C4 represent four coarse-grained categories, while C4.1 and C4.2 represent fine-grained subcategories of category C4. Right: Decision boundary with the cosine distance constraint in a Poincaré disk.
In contrast, Euclidean space has a flat geometry, which limits its ability to represent hierarchical structures efficiently.
Task Formulation In the coarse-to-fine setting [2], our goal is to train the model using only coarse-grained labels during training and achieve strong generalization performance on fine-grained categories. To evaluate the model's coarse-to-fine capability, we test it on the challenging few-shot fine-grained recognition tasks, where the model needs to recognize fine-grained categories with limited training samples. In concretely, let  $\mathcal{Y}_{\text{coarse}} = \{y_1, y_2, \ldots, y_{N_{\text{coarse}}}\}$  be a set of coarse-grained classes (e.g., cat, dog, bird), and  $\mathcal{Y}_{\text{fine}} = \{y_1', y_2', \ldots, y_{N_{\text{fine}}'}\}$  be a set of fine-grained classes (e.g., husky, samoyed, burmese). Given an auxiliary training set  $\mathcal{B}$ , it contains  $N_{\mathcal{B}}$  labelled training images  $\mathcal{B} = \{(I_1, y_1), (I_2, y_2), \ldots, (I_{N_{\mathcal{B}}}, y_{N_{\mathcal{B}}})\}$ , where  $I_i$  is an example image and  $y_i \in \mathcal{Y}_{\text{coarse}}$  is its corresponding label. Our goal is to use  $\mathcal{B}$  to learn a model that recognizes different fine-grained patterns when given a small number (e.g., 1 for each class, as assumed in this paper) of novel fine-grained labeled samples  $\mathcal{N} = \{(I_1', y_1'), (I_2', y_2'), \ldots, (I_{N_{\mathcal{N}}}', y_{N_{\mathcal{N}}}')\}$ , where  $I_i'$  is a support image and  $y_i' \in \mathcal{Y}_{\text{fine}}$  is its corresponding label. If  $\mathcal{N}$  contains  $N$  categories, each with  $K$  support images, then it is usually regarded as an  $N$ -way  $K$ -shot task.
Recap of the Hyperbolic Space We use the Poincaré ball [19] to perform the hyperbolic space. Formally, an  $n$ -dimensional Poincaré ball is defined by a manifold  $\mathbb{B}^n = \{\pmb {z}\in \mathbb{R}^n:c\| \pmb {z}\| < 1,c>$  where  $\| \cdot \|$  denotes the Euclidean norm and the hyperparameter  $c$  is used to modify the curvature.  $\mathbb{B}^n$  is an open ball of radius  $1 / \sqrt{c}$ , and with  $c\to 0$ ,  $\mathbb{B}^n$  is close to the regular Euclidean space.
In the application of the hyperbolic space, hyperbolic neural networks [11] were first proposed to adopt the formalism of Möbius gyrovector spaces to define the hyperbolic versions of feed-forward networks and Multinomial Logistic Regression (MLR) [11]. In concretely, after using the traditional neural network (e.g., CNN) with Multi-Layer Perceptron (MLP) to get an Euclidean representation vector  $\pmb{x} \in \mathbb{R}^n$ , we map it to hyperbolic representation using the exponential map:
$$
\boldsymbol {z} = \exp^ {c} (\boldsymbol {x}) = \boldsymbol {x} \tanh  (\sqrt {c} \| \boldsymbol {x} \|) / (\sqrt {c} \| \boldsymbol {x} \|), \tag {1}
$$
where  $\exp^c (\cdot)$  donates the exponential mapping with curvature  $c$  and  $\tanh (\cdot)$  refers to the hyperbolic tangent function. Then, we use a hyperbolic hyperplane
$$
H _ {\boldsymbol {p} _ {k}, \boldsymbol {a} _ {k}} = \left\{\boldsymbol {x} \in \mathbb {R} ^ {n}: \left\langle - \boldsymbol {p} _ {k} + \boldsymbol {x}, \boldsymbol {a} _ {k} \right\rangle = 0 \right\}, \tag {2}
$$
determined by two learnable vectors  $\pmb{p}_k, \pmb{a}_k \in \mathbb{R}^n$  as the decision boundary (e.g.,  $\pmb{p}_1$  and  $\pmb{a}_1$  as shown in the middel of Fig. 1) to predict the category  $k$ :
$$
p (y = k | \boldsymbol {z}) \propto \exp (\operatorname {s i g n} (\langle - \boldsymbol {p} _ {k} + \boldsymbol {z}, \boldsymbol {a} _ {k} \rangle) \| \boldsymbol {a} _ {k} \| d (\boldsymbol {z}, H _ {\boldsymbol {p} _ {k}, \boldsymbol {a} _ {k}})), \tag {3}
$$
where the  $\langle \cdot, \cdot \rangle$  represents the inner product,  $a_k$  is a normal vector to  $H_{\pmb{p}_k, \pmb{a}_k}$  and  $d(\pmb{z}, H_{\pmb{p}_k, \pmb{a}_k})$  is the hyperbolic space distance [11] from point  $\pmb{z}$  to the hyperplane  $H_{\pmb{p}_k, \pmb{a}_k}$ .
Motivation We hereby present the motivation behind our work. In recent years, there have been a lot of efforts to explore the hyperbolic space to make it better suitable for a variety of different
Figure 2: Illustration of our method during training. Our method uses two views of the same input images to obtain the representations  $\mathbf{Q} = \{\pmb{q}_1, \pmb{q}_2, \dots, \pmb{q}_{N_B}\}$  and  $\mathbf{K} = \{\pmb{k}_1, \pmb{k}_2, \dots, \pmb{k}_{N_B}\}$  via an encoder and a projector. The representations  $\mathbf{Q}$  are classified into coarse classes by the Poincaré Multinomial Logistic Regression (MLR) head, which is followed by the cross-entropy loss  $\mathcal{L}_{cls}$ . We calculate the cosine distance between each pair of examples ( $e.g., \pmb{q}_i$  and  $\pmb{k}_j$ ), and use the corresponding target distance to constrain it via  $\mathcal{L}_{hcm}$  according to their class hierarchical relationship. We calculate the average hierarchical distance in each mini-batch to carry out momentum updates to the target distance. By using coarse-grained labels for supervised learning in a hyperbolic space and imposing hierarchical cosine distance constraints on features, we can obtain more discriminative representations for unlabeled fine-grained classes.
applications [6, 11, 14, 15, 19, 22, 34, 43]. Some efforts strive [6, 34] to obtain large-margin decision boundaries to enhance the expressive power of hyperbolic models. However, since these methods directly perform complex operations in the hyperbolic space and cannot be extended to multi-grained label tasks (e.g., an object having Labrador, dog and mammal labels at the same time), these methods cannot make good use of the potential of the hyperbolic space to deal with multi-grained label data.
In the regular Euclidean space, researchers have proposed many methods [17, 32] based on the cosine distance to increase the angular boundary between different categories, which brings a considerable improvement to the performance of the model. Therefore, we introduce the cosine distance constraint into the hyperbolic space, and as far as we know, we are the first to do so in the hyperbolic space. According to Eq. (1), it is known that a vector does not change its direction after being mapped from the Euclidean space to the hyperbolic space by exponential mapping. This property, aka Conformality [21], plays a crucial role in our context. It ensures that the angles between curves or vectors in the Euclidean space remain consistent when projected onto the Poincaré disk. Therefore, as the comparison between the middle and right images in Fig. 1, the introduction of large-margin cosine distance constraint in the hyperbolic space should have similar properties as introducing the cosine distance constraints in the Euclidean space. After that, it could make the decision boundaries have larger / small similarity margins between coarse/fine classes, which enhances the generalization and discriminative capabilities of the hyperbolic space for fine-grained categories.
Overall Framework The method we proposed aims at learning a mapping function  $f_{embed}(\cdot)$  using only coarse-grained label training data that embeds raw inputs into a hyperbolic space with a hierarchical distance structure for fine-grained visual recognition. Fig. 2 depicts our method at the training time, where our method is termed as Poincaré embedding with hierarchical cosine margins (PE-HCM). Specifically, we first introduce two separate random augmentations for minibatches examples to get two correlated views, and then obtain two groups of representations  $\mathbf{Q}$  and  $\mathbf{K}$  by an encoder and a projector. For  $\mathbf{Q}$  and  $\mathbf{K}$  these two groups sample points in embedding spaces, we respectively use Poincaré Multinomial Logistic Regression (MLR) and our proposed Hierarchical Cosine Margin (HCM) manner to optimize the embedding space.
Figure 3: Comparisons between traditional and hierarchical cosine margins. Different colors indicate different categories. (a) In the latent space, we just need to make a decision boundary with a large-margin cosine distance for each pair of classes. (b) Sample pairs with three-grained labels correspond to four-level complex hierarchical target cosine distances. For instance, a pair of examples can belong to the same instance, the same fine-grained class, the same coarse-grained class, or two different coarse-grained classes.
Concretely, the Poincaré MLR module first predicts the softmax probability distribution of coarse-grained categories. Then, we use the coarse-grained labels  $y$  for supervised learning
$$
\mathcal {L} _ {c l s} = \operatorname {C r o s s E n t r o p y} (p (y = k | \boldsymbol {x}), y), \tag {4}
$$
where CrossEntropy  $(\cdot)$  is the cross-entropy loss and  $p(y = k|x)$  is the probability prediction distribution as Eq. (3). Based on this,  $f_{embed}(\cdot)$  can separate different coarse-grained samples in the embedding space.