Hyperbolic Space with Hierarchical Margin Boosts Fine-Grained Learning from Coarse Labels
Authors: ShuLin Xu
Yifan Sun
Faen Zhang
Anqi Xu
Xiu-Shen Wei
Yi Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Fine-grained learning
Coarse-to-fine learning
Hyperbolic space
Hierarchical margin

Abstract: Learning fine-grained embeddings from coarse labels is a challenging task due to limited label granularity supervision, i.e., lacking the detailed distinctions required for fine-grained tasks. The task becomes even more demanding when attempting few-shot fine-grained recognition, which holds practical significance in various applications. To address these challenges, we propose a novel method that embeds visual embeddings into a hyperbolic space and enhances their discriminative ability with a hierarchical cosine margins manner. Specifically, the hyperbolic space offers distinct advantages, including the ability to capture hierarchical relationships and increased expressive power, which favors modeling fine-grained objects. Based on the hyperbolic space, we further enforce relatively large/small similarity margins between coarse/fine classes, respectively, yielding the so-called hierarchical cosine margins manner. While enforcing similarity margins in the regular Euclidean space has become popular for deep embedding learning, applying it to the hyperbolic space is non-trivial and validating the benefit for coarse-to-fine generalization is valuable. Extensive experiments conducted on five benchmark datasets showcase the effectiveness of our proposed method, yielding state-of-the-art results surpassing competing methods.

Hyperbolic Space with Hierarchical Margin Boosts Fine-Grained Learning from Coarse Labels
Shu-Lin Xu', Yifan Sun', Faen Zhang', Anqi Xu', Xiu-Shen Wei'; Yi Yang"
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University
Baidu Inc. 3 Annnvvation Technology Group Co., Ltd *University of Toronto
CCAIL College of Computer Science and Technology, Zhejiang University
Abstract
Learning fine-grained embeddings from coarse labels is challenging task due limited label granularity supervision, i.e., lacking the detailed distinctions required for fine-grained tasks. The task becomes even more demanding when attempting few-shot fine-grained recognition, which holds practical significance in various applications. To address these challenges, we propose a novel method that embeds visual embeddings into hyperbolic space and enhances their discriminative ability with a hierarchical cosine margins manner. Specifically, the hyperbolic space offers distinct advantages, including the ability to capture hierarchical relationships and increased expressive power, which favors modeling fine-gr une-grained objects. Based on the hyperbolic space, we further enforce relatively large/small similarity margins between coarse/fine classes, respectively, yielding the so-called hierarchical cosine margins manner. While enforcing similarity margins in the regular Euclidean space has become popular for deep embedding learning, applying to the hyperbolic space is non-trivial and validating the benefit for coarse-to-fine generalization is valuable. Extensive experiments conducted on five benchmark datasets show- case the effectiveness of our proposed method, yielding state-of-the-art results surpassing competing methods.
Introduction
In deep learning, rich supervised information is crucial for the generalization ability of deep models. Fine-grained visual recognition [36] is a fundamental problem in computer vision, which requires accurately identifying subordinate (fine-grained) categories within the same meta (coarse- (coarse-grained) category. However, in many domains, annotating fine-gra seee-grrined data requires the involvement of domain experts, such in pipeline failure detection [1] biodiversity recognition (30) and medicine analyses (25). leading to high annotation costs or even infeasibility.. In contrast, obtaining the coarse labels, e.g., the meta category of species in biodiversity recognition, is much easier. In such cases, we attempt to use coarse- caaseegrained labels to train models and apply them to finer-grained recognition tasks. Such a task is challenging as lacks the detailed supervisory information required for fine- grained tasks, preventing us from obtaining sufficiently detailed label information to train models to recognize fine-grae grainer features and distinctions. In this paper, we attempt to train models using coarse- grained labels and apply them to challenging few-shot fine-grained recognition tasks, which are highly demanding and have practical significance.
*Corresponding author. This work was supported by National Key R&D Program of China (2021 2O02YYAAOON 1FA1000 100), National Natural Science Foundation of China under Grant (62272231), Natural Sci- ence Foundation of Jiangsu Province of China under Grant (BK20210340) 10340), the Fundamental Research Funds for the Central Universities (NJ2022028), and CAAl-Huawei MindSpore Open Fund.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To tackle these challenges. we propose a novel method termed Poincaré embedding with hierarchical cosine margins (PE-HCM). Specifically, PE-HCM embeds samples into a Poincaré embedding (which is also known as an embedding in the hyperbolic space (2I]) and enhances their discriminative ability with a hierarchical cosine margin manner. Our PE-HCM has two coupled features, i.e., hyperbolic space and hierarchical cosine margins:
Hyperbolic space. we first embed the samples into the hyperbolic space instead of the regular Euclidean space, as the hyperbolic space provides stronger expressive power for hierarchical relation- ships [9]. Since the tasks require an embedding space that can not only capture the coarse-graine -aarined information provided during training but also generalize well to fine-grained categories during the test, the embedding space should cover multiple granularities from coarse fine. The hyperbolic space well satisfies this prerequisite with its natural hierarchical structure.
Hierarchical consine margins. Based on the hyperbolic space, we further incorporate hierarchical cosine distance constraints to impose a hierarchical proximity relationship among sample pairs. We construct finer-grained labels than the coarse searse-grained training labels through unsupervised approaches such as data augmentation and clustering. Specifically, similar to SimCLR 4 we apply two sets of data augmentations to a batch of data. For the same sample, different augmentations are assigned with the same instance-level label, while different samples are assigned with different instance-level labels. By clustering the samples within a coarse- -rraine category, we obtain multiple clusters that represent the fine- ane-grained categories between the coarse- -gaiiied and the instance- categories. As a result, we have instance-level, une-grained, and coarse- -rrsined labels. Correspondingly, there are four types of pairwise relationships: same instance-level category, same fine fine-grained category, same saars-ggrained category, and different coarse-grained categories.
Moreover, to derive the appropriate target distances that will be used for different levels, we propose an adaptive strategy to update the target distances during training. utilizes the average distance between sample pairs in each batch to update the target cosine distances with momentum. The adaptive strategy plays crucial role in ensuring that the target distances reflect the real data distribution. By dynamically adjusting the target distances during training, our method can better capture the underlying relationships and adapt to the fine-grain une-grained characteristics of the data. In a batch of training data, we derive the target cosine distance distributions for the pairwise relationships based on their category relationships the hyperbolic embedding space. By enforcing the consistency between the feature distribution and the category relationship branch, we enhance the hyperbolic embedding space's 's generalization and discriminative capability for implicit grained categories.
In experiments, we perform PE-HCM on five popular benchmark datasets, i.e., CIFAR- 16 and four sub-datasets (LIVING-17, NONLIVING-26, ENTITY-13, ENTITY-30] from BREEDS (23).
Our method has achieved state-of-the-ar recognition accuracy on these datasets, thereby demonstrat- ing its effectiveness and its potential for practical applications.
In summary, our major contributions are three-fold:
We propose novel method that addresses the challenging task of fine-grained learning from coarse-grand -ourse-graind labels. It bridges the gap between coarse marse-grained and fine- greiird labels and leverages the knowledge learned from coarse-grained categories for fine-grained recognition tasks.
We develop a hyperbolic embedding with hierarchical cosine margins (HCM). HCM enforces relatively large/small similarity margins between coarse/fine classes. The key point of HCM, iee. the target similarity margins, are dynamically updated using an adaptive strategy.
We conduct comprehensive experiments on five popular benchmark datasets, and our proposed method achieves superior recognition accuracy over competing solutions these datasets.
2 Related Work
Coarse and Fine Learning Coarse and fine learning has been an important topic in computer vision and machine learning. Numerous methods 7 8 26 27 29,
35
40 42 and theoretical
studies (10) have been proposed to address this problem, with the goal of leveraging coarse-grained labeled data to improve fine-grainel recognition. On the one hand, several methods have been proposed to tackle the coarse and fine learning problem. For instance, Stretcu et al. [26) proposed a coarse-to-fine curriculum learning method that can dynamically generate training sample sequences

---

based on task difficulty and data distribution, thereby accelerating model convergence and improving generalization ability. Xiang et al. 40 proposed a coarse-to-fine incremental few-shot learning method that can use poarse-grained labels to perform contrastive learning on the embedding space and then used fine- ane-grained labels to normalize and freeze the classifier weights, thereby solving the class incremental problem. Sun et al. (27] developed dynamic metric learning method that can adaptively adjust the metric space according to different semantic scales, thereby improving the performance of multi-label classification and retrieval. Cui et al. 7 designed a coarse-to-fine to-nee pseudo-labeling guided meta-learning method that can use coarse labels to generate pseudo-labels, and updated model parameters through meta-optimizer, thereby achieving fast adaptation in few-shot classification tasks. Bukchin [2 proposed a fine-gret grained angular contrastive learning method with coarse labels that can use coarse labels as prior knowledge to constrain the angular loss function, and constructed positive and negative sample pairs through random sampling and data augmentation, thereby achieving robust feature representation in fine-grained image recognition tasks. On the other hand, Fotakis er al. [TO) provided a theoretical analysis of the generalization error of learning with coarse labels, which can recover the true distribution statistically without requiring additional information or assumption conditions.
Overall, previous work on coarse and fine learning demonstrates the significance of this problem and highlights the potential benefits of leveraging coarse- saarse-rrinedd labeled data for fine- -neegrained recogmition.
46) has become a popular research direction in computer vision due to the difficulty of collecting and annotating large datasets. The goal of few-shot learning is to enable models to quickly acquire knowledge from a small number of new samples, which is particularly suitable for fine-grained recognition as fine ine-grained level annotations are often costly. To address this problem, various methods have been proposed, which can be broadly categorized into three types: metric-based methods, aptimization-based methods, and attention-based methods. In concretely, metric-based methods, e.g., prototypical networks 124), relation networks (28), matching networks [31, and feature generating networks 339 learn metric space where the similarity between support and query samples is computed. These methods aim to learn feature representation that captures the discriminative information of categories and distance metric that can compare samples in this feature space. Optimization based methods, e.g., model- agnostic meta-learning [9] and piecewise classifier mappings (37,, aim to learn a model that can quickly adapt to new categories with only a few examples. These methods learn an initialization that can be fine-tuned quickly
Few-Shot Learning Few-shot learning
20
new task with small amount of data. Attention-based methods, e.g., MultiAtt [33, MattML [46] and Dual Att-Net [41] use attention mechanisms to identify the most informative parts or regions of the input images. These methods aim learn feature representation that is not only discriminative but also informative for few-shot recognition.
In summary, few-shot learning has been extensively studied in recent years, and various methods have been proposed to tackle this problem. While the aforementioned papers have shown success in addressing the problem, they still rely as the testing granularity during training. In contrast, our work aims to learn fine-grained recognition only from coarse labels during training, which is more challenging task.
considerable amount of labels with the same granularity
Hyperbolic Geometry In the field of deep learning, the hyperbolic space was first proposed by Nickel and Kiela [19 to learn hierarchical representations of symbolic data such as text and graphs by embedding them into an ni-dimensional Poincaré ball, which showed that hyperbolic embeddings can capture the hierarchical relationships in knowledge graphs more effectively than Fuclidean embeddings. Similarly, Ganea el al. (II] proposed a hyperbolic neural network for modeling tree-structured data, which was observed to outperform Euclidean- bassdde models. In addition, Sala et al. (22) utilized the hyperbolic space to embed words in natural language processing tasks. Khrulkov et al. [5] proposed a hyperbolic image embedding method to represent hierarchical structures in images.
Compared the Euclidean space which has a constant sectional curvature of 0, the hyperbolic space has constant negative curvature, which enables more efficient low-dimensinn for modeling hierarchical structure property data [9). As illustrated in the left of Fig. in this Poincaré disk, the exponential growth of distances matches the exponential growth of nodes with the tree depth [T]. This property has been shown to be particularly powerful for representing tree- structured data, where the hierarchy of the data can be more naturally captured in the hyperbolic space.
embedding
3

---

p C1
el C2
C4.2
C4.1
C4.1
Figure 1: Left: An example of modeling regular tree using 2D Poincaré disk. where all connected nodes equally spaced apart. Middle: Decision boundary in Poincaré disk, where Pt, a normal vector determining the decision boundary, and the region bounded by the are and the disk represents a category. CI, C2, C3 and C4 represent four coarse grained categories, while C4.1 and C4.2 represent fine -ra-grained subcategories of category C4. Right: Decision boundary with the cosine distance constraint in Poincaré disk.
point and
In contrast, Euclidean space has flat geometry, which limits its ability to represent hierarchical structures efficiently.
Methodology
Task Formulation In the coarse-to-fim, setting (2). goal is to train the model using only coarse-grained lahels during training and achieve strong generalization performance on fine-grained categories. To evaluate the model's coarse-to-fine capability, test it the challenging few-shot fine-grained recognition tasks, where the model needs to recognize fine-grained categories with limited training samples. In concretely, let Vecarse (v1 92. UNocevse } be set of coarse- grained classes (e.g, cat, dog, bi bird), and fine = {V1, Ve...., VNy be a set of fine-grained classes (e.g., husky, samoyed, burmese). Given an auxiliary training set B, it contains NB labelled training images B {(I1. v1). (12, 12)..... (INe yN ), where I4 is an example image and y: E Veoorse is its corresponding label. Our goal is to use B to learn a model that recognizes different fine-grained patterns when given a small number (e.g., 1 for each class, as assumed in this paper) of novel fine- the-grained labeled samples N = {(T 11), (T2, 16)..... (N'w VN }}, where T/ is support image and y E Viine is its corresponding label. If N contains N categories, each with K support images, then is usually regarded an N-way K K-shot task.
Recap of the Hyperbolic Space We use the Poincaré ball 119 to perform the hyperbolic space. Formally, an r--iiensional Poincaré ball is defined by manifold [B" {z E R² c||=|| 1, 0}. where || denotes the Euclidean norm and the hyperparameter c used to modify the curvature. B" is an open ball of radius 1/vē. and with - 0, IB is close to the regular Euclidean space.
In the application of the hyperbolic space, hyperbolic neural networks [ were first proposed to adopt the formalism of Möbius gyrovector spaces define the hyperbolic versions of feed-forward networks and Multinomial Logistic Regression (MLR) 11. In concretely, after using the traditional neural network (e.g., CNN) with Multi-Layer Perceptron (MLP) get an Fuclidean representation vector æ E R", we map it to hyperbolic representation using the exponential map:
z exp" (x) tanh( /cl== [)/(c)])]
where exp*(.) donates the exponential mapping with curvature and tanh(-) refers to the hyperbolic tangent function. Then, we use a hyperbolic hyperplane
Hp..a {x E IR", (-pa x. ax) 0}
determined by two learnable vectors Pk: E Rn as the decision boundary (e.g., Pı and a as shown in the middel of Fig. D predict the category k:
ply klz) exp(sign(( (aagen(((peee)) Ps +
where the ( represents the inner product, a% is a normal vector to Hp and d(z, Hp..a)) ) is the hyperbolic space distance [I] from point z to the hyperplane Hp...u.
Motivation We hereby present the motivation behind our work. recent years, there have been lot of efforts explore the hyperbolic space to make it better suitable for a variety of different

---

Aug.1
Encoder
Projector
Poincuré MLR
Logit
Lls
share
Aug.2
Encoder
Projector
HCM module
hnnm
deacch
mini-butch data
Momentum update
Mcan distance
Target distance
K
Memory (Group by coarse lahels)
Instance labels
A-means
Pscudo fine labels
Coarse labels
Figure 2: Illustration of our method during training. Our method uses views of the same input images obtain the representations Q = {qg, 92...., IN and = (K1, k2,..., .2....,ku ......,,nn.. } via an encoder and a projector. The representations Q are classified into coarse classes by the Poincaré Multinomial Logistic Regression (MLR) head, which is followed by the cross-entropy loss Lete. We calculate the cosine distance between each pair of examples (e.g., l.g... q, and k;,), and the corresponding target distance constrain it via Lkem according their class hierarchical relationship. We calculate the average hierarchical distance in each mini-batch to carry out momentum updates to the target distance. By using coarse-graine labels for supervised learning in a hyperbolic space and imposing hierarchical cosine distance constraints on features, we can obtain more discriminative representations for unlabeled fine-grained classes.
34 43). Some efforts strive 34 obtain sarge-margin decision boundaries to enhance the expressive power of hyperbolic models. However, since these methods directly perform complex operations the hyperbolic space and cannot extended to multi-graine label tasks (e.g., an object having Labrador, dog and mamma labels at the same time), these methods cannot make good use of the potential of the hyperbolic space deal with multi multi-grained label data.
applications 66
In the regular Euclidean space, researchers have proposed many methods 17 32 based the cosine distance to increase the angular boundary between different categories, which brings a considerable improvement to the performance of the model. Therefore, we introduce the cosine distance constraint into the hyperbolic space. and as far as we know, we are the first to do so in the hyperbolic space. According Eq. (I. is known that vector does change its direction after being mapped from the Euclidean space to the hyperbolic space by exponential mapping. This property. aka Conformality 21 plays crucial role in our context. ensures that the angles between curves or vectors in the Euclidean space remain consistent when projected onto the Poincaré disk. Therefore, large-margin
as the comparison between the middle and right images in Fig. [ the introduction cosine distance constraint in the hyperbolic space should have similar properties introducing the cosine distance constraints the Euclidean space. After that, it could make the decision boundaries have larger / small similarity margins between coarse/fine classes, which enhances the generalization and discriminative capabilities of the hyperbolic space for fine- ane-grained categorics.
Overall Framework The method we proposed aims at learning a mapping function fembea(-) using only coarse-grained label training data that embeds raw inputs into a hyperbolic space with a hierarchical distance structure for fine-gre fine-grained visual recognition. Fig. 2 depicts our method at the training time, where our method termed Poincaré embedding with hierarchical cosine margins (PE-HCM). Specifically, we first introduce two separate random augmentations for mini- batch examples to get two correlated views, and then obtain two groups of representations Q and K by an encoder and projector. For Q and K these two groups sample points in embedding spaces, we respectively use Poincaré Multinomial Logistic Regression (MIR) and our proposed Hierarchical Cosine Margin (HCM) manner optimize the embedding space.
5

---

Largel angle
sample with ground-truth/pseee
label
anchor
dissimilar
similar
(a) Traditional cosine margin
(b) Hierarchical cosine margin
Figure 3: Comparisons between traditional and hierarchical cosine margins. Different colors indicate different categories. (a) In the latent space, just need make a decision boundary with large-margh cosine distance for each pair of classes. (h) Sample pairs with three-grainee lahels correspond to four-level complex hicrarchical target cosine distances. For instance, a pair of examples can belong to same instance, the same fine-grained class, the same coarse-grained class, or two diffcrent coarse-grained classes.
Concretely, the Poincaré MLR module first predicts the softmax probability distribution of coarse- grained categories. Then, we use the coarse-grained labels y for supervised learning
Leie = CrossEntropy( = k=
where CrossEntr)pyy() is the cross-entropy loss and ply k/(x) is the probability prediction distribution as Eq. (3). Based on this, fembed(-) can separate different coarse- coarse-grained samples in the embedding space.
Hierarchical Cosine Margin Manner On this basis, the HCM manner constrains the distance distribution between sample pairs to be consistent with the hierarchical distribution of sample labels, so that sample pairs with different grained labels have different gaps in the embedding space, thus obtaining more refined hierarchical embedding space that is conducive to fine aneeggrined recognition. Specifically, we first group features K according to coarse coarse-grained labels and store samples with the same coarse-grained labels in the same group in memory. We keep the latest M samples for each coarse-grained category in memory. Then, we perform k-means clustering on each group of features respectively, dividing each group of sample features into smaller groups of clusters. We regard these more subdivided clusters as fine-grained categories under coarse-grained categories and give different clusters different fine-grained pseudo-labels. From this point on, a feature in the embedding space has three levels of granularity labels: instance-level label (each input sample has its own instance-level label, a sample's different augmentations have the same instance-level label), fine-gres grained pseudo-labell and coarse- -rannneg labels. Therefore, for a feature vector of an anchor sample in Q, there are four hierarchical relationships in K: same instance-level label, same fine-grained label, same coarse -raiind label, and different coarse coarse-grained labels.
The hierarchical cosine distance distribution is demonstrated in Fig. (b). From bottom top, the visual similarity between the samples and the anchor decreases, resulting in larger cosine distances between the samples and the anchor in the embedding space.
This creates a hierarchical cosine distance distribution that follows the same pattern as the visual similarity distribution between categories. Our proposed HCM manner is more refined and complex than the traditional cosine distance constraint, as shown in Fig. B(a). The traditional cosine distance only needs to consider two types of relationships, i.e., similar dissimilar, while the HCM manner needs to consider the four hicrarchical similarity relationships mentioned above in our setting.
Fig-B(a)
In order to constrain the sample features by the hierarchical cosine distance, we calculate the cosine distance distribution between each feature {q: Ne in Q and all features (k; J== in K by
W [dcos (q:: k1), deos (q., k2)...., deos (g., kNk)))))) )}, (q:
(5)
where deos (u, v)
| T [0, 2) is cosine distance function, and NB the training batch size. shown Fig. b), the distance between two samples the embedded space becomes larger as the category gap between them increases. For example, when the relationship between sample pairs the same physical object, different objects of the same kind, different species of the same genus, and different genera of the same family, the sample pairs become less and less
cos
T

---

similar. To simplify the model, we set the distances between sample pairs with the same sample, the same fine- ane-grained category, the same coarse- coarse-graines grained category, and different coarse-grained categories as do. d1, d2, and d3. respectively, where do < d1 < d2 < dg. We obtain the target semantic distance between each feature {q: )
No in Q and all features (ke { j= in K for each sample pairs by
Mi [d k1), (q., k2). ... de KN )
(6)
do
if u, v are same instance- labels, else u, v are same fineggraine pseudo labels, To ensure that the
di
where d)
d2 else u, v are same coarse-grained labels,
d3 otherwise.
feature distance distribution of samples in the embedding space aligns with the hierarchical class semantic similarity, we approximate the distributions of the pairwise feature cosine distances of samples and targets using the Kullback-Leible (KI.) divergence
1
Nn Σ
Lhem
NB == KL,,,, W.).
Finally, we combine Eq. (4) and Eq. to obtain the overall loss during the training process
C = Lets + aCheem
(8)
where a is a trade-off hyperparameter.
During testing, utilize the trained (embed to embed the support set and query set samples into hyperbolic space. In the following, we classify the query samples by computing their distance to the support samples through the k-nearest neighbors method.
Adaptive Hierarchical Cosine Distance Eq. (6). we use [do, d1, d2, dz] in de(-) as the target values of the feature cosine distance distribution, which should be determined by the intrinsic hierarchy in different datasets, so we propose an Adaptive Hierarchical Cosine Distance (AHCD) strategy to update the target values during the training process according to the training data. Specifically, set the target value dg of samples with the same instance-level label to and the target value d3 of samples with different coarse- coarse-grained labels to which correspond to angles of 90% We initialize dj and d2 to 0.134 and 0.5 which correspond to angles of 30 and 60%, respectively, and using dg and d3 anchor values to uniformly distribute them. During training, calculate the average feature cosine distance between different examples with the same grained label each batch data
de No Σ Kyek deoa(gi, kt)),
where (1,2) 2} respectively represent fine-gratd ane-ggrined label hierarchy and coarse coarse-grained hierarchy (i.e., di and d2 respectively represent average distance of sample pairs with same fine-grained label and same coarse-grained label), S2 represents a set the sample pairs that belong to same grained labels, N₁ is the size of Ωg. Then, we update the target semantic distance by momentum update
dj = 3dg (1 3)d,
(10)
We the momentum update method to dynamically update the target semantic cosine distance of sample pairs with the same fine-grained label and the same searse-grained label, so that d₁ and da can adjust values adaptively according to the distribution of data in different datasets.
4 Experiments
Benchmark Datasets Table [ summa-
Table I: Summaries of the benchmark datasets. L-17, NL-26,
rizes the benchmark datasets CIFAR-
E-13, E30 are the LIVING-17, NONLIVING-26, ENTITY- 13. ENTITY-30 sub-datasets from BREEDS 23
100 [16] and BREEDS (23]. The
BREEDS dataset includes four subsets de- rived from ImageNet, namely LIVING- 17, NONLIVING-26. ENTITY-13, and ENTITY-30,, each of which has a class hier- archy calibrated to ensure that classes at the same level have similar visual granularity.
Datasets
L-17
NL-26
E-13
E-30
CIFAR- 100
Course classes
17
26
13
30
Fine classes
68
104
260
240
100
# Train images
88K
132K
334K
307K
50K
Test images
3.4K
5.2K
13K
12K
Image resolution
224
224
224
224
32
7

---

Table 2: Comparisons on BREEDS 23
Red and blue bold numbers the top two best results.
LIVING-17
NONL.VVING-26
ENTITY-13
ENTITY-30
Methods
5-way
all-way
5-way
all-way
S-way
all-way
S-way
all-way
Fine upper-bound
90.7510.44
62.6510.15
90,33
60.6810.14
94.721033
65.181009
94,0210.36
63.72.1010
MoCo-v2
56.66120.70
18.57:0.11
2075
21.07:0.11
82.002007
33.0620077
80.3720.60
28.62:0006
MoCo-12-ImageNet
62.2110.73
40 10.29
77.07.1076
34.78.10:33
85.2410000
35.62.1008
83.06.0.02
31.73_0008
SWAV-InugeNet
79.83
38. 38.79
76,26:0771
33.94
81.15=0055
33.57:00.7
79.91
31.15=0007
ANCOR
89. 59.23340.55
45.14=0.12
86.23++0.4
43.10=0.11
90.58=0.54
42.29::0.08
88.12+0.54
41.79=008
ANCOR- fo
90.41
46. 46.19=0.16
88,77=00.44
45.34:2013
89.05=008
38.52:4:008
91 91.84400.49
42.33=0000 to
SCGM-G
89.72+0.54
48.74=0.15
89.87=
49.25++0.33
90.15=0051
40.00
92.90=00.46
42.17=005
SCGM-A
90.9740.55
49.31
58.78=00.55
46.93+0.13
88.48=0059
41,07:00.99
91. 20.55
44.14=0.19
Ours
90.94:00.3
53.09
89.97=0422
50.12=0.11
91.24::0.55
41.64::0.00
92.95
44.53=
Table 3: Comparisons on CIFAR- 100 [16]
Table 4: Comparisons of intra-class (all fine-grained class recogni-
Red and blue bold numbers are the top two best results.
tion within random coarse class) on the BREEDS 23 Red and blue bold numbers the top two best results.
Methods
5-way
all-way
Methods
Living17
Nonliving26
Entiy13
3n
Fine upper-boune bound
75.53=008
31.35-011
Fine upper-bound
70.72.10.91
74.02.0.91
72.2410.00
73.8610.57
ANCOR
74.562070
29.84=011
ANCOR
48.77:0.31
49.6420.88
42.0020.47
45.17:10.99
ANCOR- fc
74.73=073
27.32-0010
ANCOR-16 fe SCGM-G
51.07+0.22
53.51+1000
41,83+0.45
47,82+00.5
SCGM-G
76.19
29.92-0111
53.48
$7.32
43.89
46.80
SCGM-A
17.37
25.91=0.10
SCGM-A
53.88=0.90
55.12+1000
45.09:
50.02=0.71
Ours
81.42...69
36.28 0.12
Ours
57.11
58.63+1.01
44.85=a.54
51.24:0.65
Baselines We compare PE-HCM with the most relevant state-of-the-art models on embedding learning: (1) MoCo-v2 (5). trained on the above datasets; (2) MoCo-v2-ImageNet [5), pre-trained on the official full ImageNet; (3) SWAV-ImageNet, pre-trained by Caron 3] (4) ANCOR [2]. combine supervised and self-supervised contrastive learning with their proposed Angular normalization module; (5) SCGM [18), based on the auperclass-conditional Gaussian mixture model. (6) 'Fine upper-bound'
natualy trained on the ane-ggained labels.
Implementations For fair comparisons with ANCOR 22 and SCGM [8] we use ResNet-12 12 [2] and ResNet-50 [12 as the backbone network for CIFAR- 100 [16] and BREEDS 223. We use a three-layer MLP the projector, and the exponential mapping follows it embed the features in the hyperbolic space. The input, hidden, and output layer dimensions of the MLP are a + a + b, where is 640 and 2048 for ResNet-12 12 and ResNet-50, respectively, and the output layer dimension b is 128, following ANCOR and SCGM. For the hyperparameters, we set = 0.001 in Eq. ① a 800 in Eq. (8) 8 0.999 in Eq. TT). We used the Adam optimizer to train the model on GeForce RTX 3090 Ti GPUs, and training a total of 200 epochs. For CIFAR-100 and BREEDS, the batch size were 1024 and 256, the initial learning rates were x 10 and 1.25 x 10 the learning rates were reduced by 10 times when the epoch was 120 epoch and 160 epoch. In order to prevent the distance distribution from getting stuck in local optima, we reinitialize d 0.134 and d2 0.5 the 120-th epoch and 160-th epoch. We followed ANCOR (2) to implement random data enhancement using random resized crop, random horizontal flipping, random color Jitter, random grayscale, and random Gaussian smoothing during training. We evaluate the performance using 5-way and all-way 1-shot settings during testing. The evaluation is conducted on 1000 random episodes, and we report the mean accuracy along with the 95% confidence interval.
Main Results Table and Table 3 present the average accuracy rates for fine-grained learning from coarse labels on BREEDS and CIFAR-100. For each dataset, we report both 5-way and all-way fine-grained recognition results during testing. As shown in these tables, our proposed model is significantly better than other baseline methods on the above datasets. Especially on CIFAR- 100, we achieve 4.05% and 6.36% improvements over state-of-the-ar methods, even surpassing the fine upper-bound baseline significantly. Tableeevaluates evaluates an intra-class case when all fine-grained classes of random coarse- gaarsee classes were sampled in each episode. The results demonstrate that our method can better distinguish fine-graince categorics in the embedding space.
Retrieval Task Results In addition to evaluating the ability of learning fine- greengr embeddings from coarse labels in few-shot learning tasks, Grafit 29 and MaskCon (8 also investigated the effectiveness of the 'coarse to fine' approach by using retrieval tasks for validation. As shown in Table 5 we conducted fair comparisons of our method with these otate-of-the-art approaches, using
8

---

Table 5: Comparisons CIFAR-100. Red and blue bold numbers the top best results.
Methods
5-way
All-way
Recall@
Recall@5
mAP
Grafit (Reported in puper)
60.57
82.32
MaskCon (Reported in paper)
65.52
83,64
MaskCon (Our re- re-implementation)
81.05
40.81
65.35
83.57
60.27
Ours
85,45
47.76
62.11
80.68
62.04
the same backbone and epoch settings as MaskCon. Our method still achieves superior results over Grafit and MaskCon (almost 7% improvement on all-way FSFG) under FSFG, and achieves a mean Average Precision (mAP) of 62.04%, surpassing MaskCon's ' 60.27%. This also reinforces the efficacy of our approach. Additionally, it is worth noting that our method is lower than MaskCon when we evaluation metric (by following MaskCon). The reason might be that image
used Recall @k retrieval involves identifying relevant images from gallery of diverse images, usually by utilizing query image as a reference. In this scenario, the task entails finding any images belonging the same category as the query image from the gallery, which might contain multiple images per category. On the other hand, few-shot recognition, exemplified by the one-shot scenario, presents a more challenging scenario. In this setting, the goal is to recognize objects/classes with very limited training examples, often relying on only a single image per category. This constraint inherently magnifies the difficulty of the task.
Proposals in Our Method As shown in Ta- ble 6. we explored the improvement of each module in our method on LIVING-17 and CIFAR-100. The results of the first two rows show that the hyperbolic space is more suitable for the fine- ane-grained embedding from coarse la- bels with a hierarchical structure, which is con- sistent with the conclusion shown in [T] [5]. The results of the last three rows show that the introduction of our HCM manner and AHCD strategy the hyperbolic space can bring great improvement. This indicates that all proposals of our method can better learn the discriminative patterns between finer-grained categories from coarse labels.
Table 6: Comparisons of our proposals on LIVING- 17 23 and CIFAR-100 100 [66. E: Coarse-grained su- pervised lcarning in the Euclidcan space. H: Coarse- grained supervised learning in the hyperbolic space. HCM: Our proposed Hierarchical Cosine Margin man- ner. AHCD: Our proposed Adaptive Hierarchical Cosine Distance strategy.
Methods
LIVING-17
CIFAR- 100
E
H
HCM AHCD
S-way
all-way
5-way
all-way
87.72
32.71
76.34
24.92
88.06
33.78
76.62
26.19
90.02
46.53
77.64
33.22
90.94
53,09
81.42
36.28
Visualization As shown in Fig. it can be observed that compared training the model using only coarse labels, our method leads to more concentrated distribution of sample points for each category, while also accentuating the distinctions between different classes. The visualiza- tion validates the effectiveness of our method of contribut- ing the discriminative power from the qualitative aspect.
Figure 4: Visualization by t-SNE 13 on
CIFAR- 100. Left: training the model using only coarse labels. Right: method.
Adaptive Ability We extensively explored the effective- ness of the AHCD strategy. As shown in Fig. 5 we tracked the update processes of dN and dy during training on CIFAR-100 and LIVING-17 datasets. In general, there are the same trends and different details. In the early stages of training, as the model lacks discriminative ability, the values of d, and da gradually decrease. As the model continues to train, it gradually acquires discriminative ability, resulting in an increase in the values of dj and d2 followed by stabilization. From the final stable results, we observe that the margin between coarse-grained -reiier categories (d2) is larger than the margin between fine- fine-grained categories (dy) ) and the margin between fine- aiee-grained categories on CIFAR-IO0 is larger than the margin between fine- iine-grained categories on LIVING-17, 17, which aligns with the true category relationships in the data distribution. This demonstrates that our proposed adaptive strategy can adjust d1 and d2 during training according to the actual data distribution.
As shown in Fig. 6 we changed the trade-off hyperparameter a of the hierarchi- cal cosine loss Eq. (8) on the CIFAR-100 and L.VING-17 datasets to observe its impact for the final result. It can be seen that, as increases exponentially, the performance of the model increases
Hyperparameters

---

208
CIFAR-IM
CIFAR- LIVING-17
9077
LIVING-I?
209
088
0.3
0.6
0.4
0.
0.3
0.4
LIVING-17
0.3
0.3
0.1
002
01
0.1
0
50
100 150
200
50
100 150 200
50
100
200
401
500
1600
spoch
epoch
iepersamnnter
Figure 5: The update processes of dj (left) and d2 (right) in adaptive hicrarchi- Figure 6: Comparisons with dif- cal cosine distance strategy on CIFAR- 100 and LIVING-17. d1 and d2 are the ferent value of a on CIFAR- target distance for the same fine-grained and coarse- coarse-grained sample pairs.
100 and LIVING-17.
first and then decreases. The value of a is also sensitive, with smaller value of a limiting the discriminative hierarchical embedding ability the HCM manner, and larger value of a suppressing the generalization ability to fine-grained categories.
As shown in Fig. 7 we change the number of k-means clusters for each -oarse-grained class in the HCM manner on CIFAR- 100 and LIVING- 17, and report the recognition results for all fine-grained categories (all-class) and intra- coarse class fine- grained categories (intra-class). From the experimental results, it can be observed that the number of clusters has a significant impact on the fine-grained recog- nition performance. When the number of clusters is small, the model performs poorly tne-ggrined recognition tasks. Having a larger number of clusters than the actual number of fine- grained categories leads to better results. Additionally, in our experiments across five datasets, we found a recurring pattern that offers a guideline. Typically, when clustering coarse-graine gaasseg categories, the number of clusters that yields optimal results is roughly twice the actual number of fine-grained subclasses within that coarse category. While this heuristic emerged consistently across our tested datasets, we acknowledge that it may not be universally optimal. However, it does provide a good starting point for users. In practice, fine-tuning based on validation performance is always recommended to ascertain the best hyperparameter for a specific task.
CIFAR-100
145
2858
57
+ intra class all--class
39
37
35
51
intra-clas clll-class
50
33
49
12 14
clussers
clusters
Figure Comparison results 1-shot learn- ing with different numbers of clusters on CIFAR-IO 100 and LIVING-17.
5 Conclusions
We proposed a novel method (PE-HCM) for fine-grained learning from coarse labels. We used the hyperbolic space embed the samples and enhanced the discriminative ability with hierarchical cosine margins manner. On the one hand, we performed supervised learning using coarse-graines labels to distinguish coarse-grained category samples initially. On the other hand, we constructed instance-level labels and fine-grained pseudo-labels using data enhancement and clustering methods, for the hierarchical cosine margins manner constraining the distance between sample pairs of four relationships, which resulted in a more refined hierarchical division distances for fine-graid une-grained learning. By optimizing the hierarchical cosine distance using the hierarchical cosine margins manner, the learned embedding could be well generalized to fine-grained visual recognition tasks. Experiments on five benchmark datasets demonstrated the effectiveness of the proposed method.
Throughout our experiments, we have occasionally observed particular scenario where our method yields comparatively lower results. This typically occurs when classes within the same hierarchy level exhibit substantial variations in their scopes. For instance, when some classes possess a significantly broader scope while others have notably narrower scope. We believe this phenomenon arises from the fact that our method employs a uniform margin across all levels. In the future, more refined eitegory-specifee semantic hierarchical distance constraints constraining the distances between sample pairs of different categorics within the same grain of granularity make them more consistent with the real data distribution.
worth further study, that is, further
10

---

References
[1] Mohammad H. Alohaidi, Mohamed A. Meguid, and Tarek Zayed. Semi-supervised learning framework for and gas pipeline failure detection. Scientific Reports. 12(13758), 2022.
[2] Guy Bukchin, Eli Schwartz, Kate Sacnko, Ori Shahar, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. Fine-grained angular contrastive learning with coarse labels. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8730-8740, 2021.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski. and Armand Joulin. Unsuper-
vised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, pages 9912-9924, 2020.
[4] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine
Learning, pages 1597-1607, 2020.
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422: 2020.
[6] Hyunghoon Cho, Benjamin DeMeo, Jian Peng. and Bonnie Berger. Large-margin classification in hyperbolic space. arXiv preprint 1r6. 00437, 2018.
[7] Yawen Cui, Qing Liao, Dewen Hu, Wei An, and Li Liu. Coarse-to- fine pscudo supervision guided meta-task optimization for few-shot object classification. Pattern Recognition, 122: 108296, 2022.
[8] Chen Feng and Ioannis Patras. MaskCon: Masked contrastive learning for coarse-lahelled dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19913-
19922. 2023.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariing for fast adaptation of deep networks. arXiv preprint 1oXiv:1733.0322
2017.
[10] Dimitris Fotakis, Alkis Kalavasis, Vasilis Kontonis, and Christos Tzamos. Efficient algorithms for learning
from coarse labels. In Proceedings of Conference on Learning Theory, pages 2060 2079, 2021.
[11] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in Neural Information Processing Systems, pages 5345-5355, 2018
[12] Kaiming He, Xiangyu Zhang. Shaoqing In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778,
and Jian Sun. Deep residual learning for image recognition.
2016.
[13] Geoffrey E. Hinton and Sam Roweis. Stochastic neighbor embedding. In Advances in Neural Information Processing Systems, pages 857-864, 2002.
[14] Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with self-supervisee hyperbolic representations. In Advances in Neural Information
Processing Systems, pages 51 5112-5123, 2021.
[15] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova. Ivan Oseledets, and Victor Lempitsky. Proceedings of the IEEE/CVF Conference on Computer Vision and Hyperbolic image embeddings.
Pattern Recognition, pages 6418 8-6428, 2020.
[16] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[17] Weiyang Liu, Yandong Wen. Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. arXiv preprint 161iv:1612.02290 2016.
[18] Jingchao Ni, Wei Cheng, Zhengzhang Chen, Takayoshi Asakura, Tomoya Soma, Sho Kato, and Haifeng gaussian mixture model for learning fine-grained embeddings. In Proceedings of International Conference on Learning Representations, pages 1-12, 2022.
Chen. Superclass-conditonn
[19] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Advonces in Neural Information Processing Systems, pages 6338-6347, 2017
[20] Archit Parnami and Minwoo Lee. Learning from few examples: A summary of approaches few-shot learning. arXiv preprint vrXiv:2007.073355 2022.
[21] Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10023-
10044, 2021.
[22] Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoffss for hyperbolic embeddings. In Proceedings of the International Conference on Machine Learning, pages 4460 4469, 2018.
[23] Shibani Santurkar, Dimitris Tsipras. and Aleksander Madry. BREEDS: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008. 00859,, 2020.
[24] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077-4087, 2017.
[25] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus. Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse- coarse-grained classification problems. In Advances in Neural Information 2020.
Processing Systems, pages
[26] Otilia Streteu, Emmanouil Antonios Platanios, Tom M Mitchell, and Barnabás Póczos. Coarsc-to- corsc-to-finte fine curriculum learning. arXiv preprint 106.v:2106.0400) 2021.
[27] Yifan Sun, Yuke Zhu, Yuhan Zhang. Pengkun Zheng, Xi Qiu, Chi Zhang, and Yichen Wei. Dynamic metric learning: Towards a scalable metric space to accommodate multiple semantic scales. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5393-5402, 2021.
11

---

[28] Flood Sung, Yongxin Yang, Li Zhang. Tao Xiang, Philip HS Torr and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1199-1208, 2018.
[29] Hugo Touvron, Alexandre Sablayrolles, Matchijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning fine-grained image representations with coarse labels. In Proceedings of the IEEF/CVE International
Conference on Computer Vision, pages 8 874-884, 2021.
[30] Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation lcarning for natural world image collections. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12884-12893, 2021.
[31] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot leamning In Advances in Neural Information Processing Systems, pages 3630-3638,
2016.
[32] Hao Wang. Yitong Wang, Zilei Zhou, Xingang Ji, Dihong Gong, Jingdong Zhou, Zhifeng Li, and Wei Liu. CosFace: Large margin cosine loss for deep face recognition. Proceedings of the IFEF//VE Conference on Computer Vision and Pattern Recognition, page 5265-5274, 2018.
[33] Peng Wang. Lingqiao Liu, Chunhua Shen, Zi Huang, Anton van den Hengel, and Heng Tao Shen. Multi- attention network for one shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2721-2729 2017.
[34] Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya K Menon, and Sanjiv Kumar. Robust large- margin learning in hyperbolic space. In Advances in Neural Information Processing Systems, pages 17873,17873, 2020.
[35] Xiu-Shen Wei, Yang Shen, Xuhao Sun, Peng Wang, and Yuxin Peng. Attribute-aware deep hashing with self-consistency for large-scale fine-grained image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3(14)113904113900 2023.
[36] Xiu-Shen Wei, Yi-Zhe Song. Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang. and Serge Belongie. Fine-grained image analysis with deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):8927 8948, 2022.
[37] Xiu-Shen Wei, Peng Wang, Lingqiao Liu, Chunhua Shen, and Jianxin Wu. Piecewise classifier mappings: Learning fine-grain fine-grained learners for novel categories with few examples. IEEE Transactions Image Processing. 28( (8(12):611 16 16-61125 2019.
[38] Xiu-Shen Wei, Hc- Yang Xu, Faen Zhang, Yuxin Peng, and Wei Zhou, An embarrassingly simple approach to semi-supervised few-shot learning. In Advances Neural Information Processing Systems, pages 14489-14500, 2022.
[39] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5542-5551, 2018.
[40] Xiang Xiang. Yuwen Tan, Qian Wan, Jing Ma. Alan Yuille, and Gregory D Hager. Coarse-to-fine incremental few-shot learning. arXiv preprint arXiv:2111. 11..160,6, 2021.
[41] Shu-Lin Xu, Faen Zhang, Xiu-Shen Wei, and Jianhua Wang. Dual attention networks for few-shot fine- grained recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 291 1-2919, 2022.
[42] Yuanhong Xu, Qi Qian, Hao Li, Rong Jin, and Juhua Hu. Weakly supervised representation learning with coarse labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
10593-10601, 2021.
[43] Yi Xu, Dongsheng Wang, Bo Chen, Ruiying Lu, Zhibin Duan, Mingyuan Zhou, et al. HyperMiner: Topic taxonomy mining with hyperbolic embedding. In Advances in Neural Information Processing Systems,
pages 31557-31570, 2022.
[44] Han-Jia Ye, Lu Han, and De-Chuan Zhan. Revisiting unsupervised meta-learning via the characteristics of few-shot tasks. IEFF Transactions on Pattern Anolysis and Machine Intelligence, 45(3):371133737, 2023.
[45] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 8808-8817, 2020.
[46] Yaohui Zhu, Chenlong Liu, and Shuqiang Jiang. Multi-attention meta learning for few-shot fine-grained image recognition. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1090-1096, 2020.
12


Reviews:
[{"review_id": "whKcMoxd6D", "reviewer": "Reviewer_zkc6", "summary": "This paper addresses the challenge of learning fine-grained embeddings from coarse labels, where detailed distinctions required for fine-grained tasks are often lacking. The authors propose a novel method that embeds visual embeddings into a hyperbolic space and enhances their discriminative ability using a hierarchical cosine margins approach. The hyperbolic space offers advantages such as capturing hierarchical relationships and increased expressive power, making it suitable for modeling fine-grained objects. By enforcing relatively large/small similarity margins between coarse/fine classes in the hyperbolic space, the proposed hierarchical cosine margins approach is introduced. While enforcing similarity margins in the regular Euclidean space is popular for deep embedding learning, extending it to the hyperbolic space is non-trivial and valuable for coarse-to-fine generalization. Experiments are conducted on five benchmark datasets.", "questions": "I am curious about how does the proposed method perform when the available coarse-grained labels are noisy or contain inaccuracies?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tNovel Method: The paper introduces a novel method, Poincare embedding with hierarchical cosine margins (PE-HCM), which addresses the challenging task of fine-grained learning from coarse-grained labels. This innovative approach bridges the gap between coarse-grained and fine-grained labels, enabling the transfer of knowledge from coarse-grained categories to fine-grained recognition tasks.\n2.\tTechnical Contribution: The proposed method incorporates key technical contributions. It utilizes the hyperbolic space to capture hierarchical relationships and provide increased expressive power. Additionally, the hierarchical cosine margins enforce proximity relationships among sample pairs, enabling the model to learn fine-grained features and distinctions. The adaptive strategy for updating target distances further enhances the model's ability to capture underlying relationships and adapt to the fine-grained characteristics of the data.\n3.\tPaper Writing: The paper is well-written and effectively communicates the motivations, challenges, and proposed solutions. The introduction provides a clear background and problem statement, leading to a comprehensive description of the proposed method. The technical details, including the use of hyperbolic space and hierarchical cosine margins, are explained concisely, making the paper accessible to readers.\n4.\tExperimental Performance: The proposed method is thoroughly evaluated on five benchmark datasets, demonstrating its superior performance compared to competing solutions. By achieving state-of-the-art recognition accuracy, the experimental results highlight the effectiveness and potential practical applications of the proposed method in fine-grained recognition tasks.\n", "weaknesses": "1.\tLimited Analysis of Failure Cases: The paper could include a discussion of the failure cases of the proposed method. Analyzing scenarios where the method does not perform well would provide insights into potential weaknesses and opportunities for future improvements.\n\n2.\tMinor Issues: There are several typos throughout this paper, e.g., in Table 1, Coarase should be Coarse.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for the positive comments. Below please find our responses to some specific comments.\n\n### Comment_1: \n\n>  *Limited analysis of failure cases*\n\nThanks for this suggestion. We fully concur that a deep dive into the scenarios where our model underperforms can provide valuable insights and pave the way for future refinements.\n\nDuring our experiments, we used to observe a typical case: if the classes in the same level have very different scopes (e.g., some classes have very large scope and some classes have very small scope), our method tends to achieve relatively low results. We infer it is because our method uses the same margin for each level. We will add the discussion into the manuscript. We also believe that such investigation will help our future research.\n\n\n### Comment_2: \n\n> *Several typos*\n\nThank you for pointing out the typographical errors. We will meticulously review the manuscript to correct these typos, including the one you've highlighted in Table 1, and ensure the clarity and professionalism of the revised submission. \n\n### Comment_3: \n\n> *How does the proposed method perform when the available coarse-grained labels are noisy or contain inaccuracies*\n\nThank you for this good question. Adding noises to the coarse-grained labels considerably compromises the accuracy. During rebuttal, we randomly add 20% noises to the ground-truth labels on CIFAR100 dataset, and observe a decrease of 10.28% 5-way accuracy. Investigating the noise problem is of realistic value and may point the new direction for our future work. \n\n\n"}, {"review_id": "KDX3SK54nt", "reviewer": "Reviewer_FRH5", "summary": "The paper presents a novel method, PE-HCM, for fine-grained learning from coarse labels. The authors propose the use of the hyperbolic space for sample embedding and introduce a hierarchical cosine margins manner to enhance discriminative ability. The method combines supervised learning with coarse-grained labels and instance-level labels obtained through data enhancement and clustering. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed approach.", "questions": "The paper introduces the use of the hyperbolic space for embedding samples and applies the hierarchical cosine margins to enhance the discriminative ability. While the results demonstrate improved performance in fine-grained recognition, I would like to inquire about the interpretability of the learned embeddings. Can the authors provide more insights into the interpretability of the hyperbolic embeddings and how the hierarchical cosine margins contribute to the separability of fine-grained categories? It would be helpful to include visualizations or case studies that illustrate how the learned embeddings capture meaningful hierarchical relationships and contribute to the discriminative power.", "limitations": "The paper concludes by discussing potential directions for future research, highlighting the importance of refining category-specific semantic hierarchical distance constraints. This suggestion indicates the authors' awareness of the limitations of the proposed method and opens up opportunities for further exploration and improvement in fine-grained learning from coarse labels.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper addresses an important problem in the field of fine-grained learning, which is the challenge of learning from coarse labels. Fine-grained recognition is crucial in various applications, and the proposed method offers a solution to improve the performance in this context. The significance of the problem adds value to the research presented in the paper.\n* The paper introduces several novel concepts and methods. The use of the hyperbolic space for sample embedding is a unique approach that has shown promise in other related works. Additionally, the introduction of the hierarchical cosine margins manner to enhance discriminative ability is an innovative strategy that contributes to refining the hierarchical divisions for fine-grained recognition. The combination of these concepts and methods in the proposed PE-HCM method demonstrates the originality of the research.\n* The paper exhibits good writing quality, characterized by clarity and soundness. The paper effectively communicates the motivation, methodology, and experimental results to the readers. The descriptions of the proposed method and the experimental setup are clear and well-presented. The writing style contributes to the overall understanding of the research and enhances the readability of the paper.\n* The paper demonstrates a high level of experimental rigor. The authors provide detailed descriptions of the benchmark datasets used, including their characteristics and sizes. They also compare the proposed method with several state-of-the-art models, covering a wide range of relevant approaches in the field of embedding learning. By conducting extensive experiments on multiple benchmark datasets, the authors validate the effectiveness of their proposed method and show its superiority over existing techniques. The inclusion of statistical measures such as mean accuracy and confidence intervals further enhances the reliability of the experimental results.\n* The paper showcases the adaptive ability of the proposed method through the introduction of the Adaptive Hierarchical Cosine Distance (AHCD) strategy. The authors track the update processes of the distance parameters and show that the method adjusts these parameters during training to align with the actual data distribution. Furthermore, the paper provides a thorough analysis of the trade-off hyperparameter α, demonstrating the sensitivity of the method to its value and offering insights into finding an appropriate balance between discriminative hierarchical embedding and generalization ability.\n", "weaknesses": "* While the paper includes theoretical illustrations of sample pair distances, it would be beneficial for the authors to provide empirical evidence by visualizing the feature distances between real samples.\n* The paper focuses on the evaluation of the proposed method on specific benchmark datasets but lacks a broader discussion on the generalizability of the results. Providing insights into the transferability of the proposed approach to other fine-grained recognition tasks or datasets would strengthen the overall impact of the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for the positive comments. Below please find our point-to-point responses.\n\n### Comment_1: \n\n> *Visualization and interpretability*\n\nThank you for your suggestion. We have provided the visualization by t-SNE on CIFAR-100 in the global rebuttal file. As shown, it can be observed that compared to training the model using only coarse labels, our method leads to a more concentrated distribution of sample points for each category, while also accentuating the distinctions between different classes. The visualization validates the effectiveness of our method of contributing to the discriminative power from the qualitative aspect. We will also add it in the final version.\n\n### Comment_2: \n\n> *Broader discussion*\n\nThanks for this suggestion. Our proposed approach, rooted in the principles of hyperbolic embeddings and hierarchical cosine margins, is intrinsically designed to capture hierarchical relationships. Such hierarchies are ubiquitous in nature and span various tasks beyond fine-grained recognition, encompassing a broad spectrum of datasets and domains.\n\nWhile this study focuses on specific benchmarks, we acknowledge the reviewer's point regarding the potential applicability of our method to a diverse set of tasks influenced by natural hierarchies. In our future work, we intend to explore the versatility of our method in tasks characterized by hierarchical structures. By expanding our experimentation horizon, we will delve into the broader potentials and possible limitations of our approach.\n\nWe genuinely appreciate your feedback and believe that broadening the applicability of our method in subsequent research will shed light on its capabilities in handling tasks with inherent hierarchical relationships, thereby increasing its impact and significance in the broader scientific community."}, {"review_id": "6uNH25ZOEN", "reviewer": "Reviewer_EUtT", "summary": "This paper proposes a method developed in the hyperbolic space to embed visual embeddings to deal with the task of fine-grained learning from coarse labels. By discovering the advantages of hyperbolic space, the authors design the hierarchical cosine margin manner and an adaptive hierarchical cosine distance, which favour modelling fine-grained objects. Experiments are conducted on five benchmark datasets. Evaluations show that the proposed method achieves superior recognition accuracy over competing solutions on these datasets.", "questions": "- As stated in the section of Hyperparameters, different numbers of clusters lead to different accuracy. How to choose an appropriate hyperparameter for a specific task for users?\n\n- Some minor issues of paper writing should be fixed, e.g., “Fig” should be fixed as “Fig.” or “Figure”.\n", "limitations": "This paper does not explicitly discuss its limitations.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ The proposed method is novel and makes sense, which has great technical contributions, especially for the introduction of hyperbolic space into the fine-grained learning from coarse labels.\n\n+ The fine-grained learning from coarse labels task has good practical potential, which also is a challenging problem in computer vision and machine learning. Such a task deserves further exploration.\n\n+ The hierarchical cosine margins and the adaptive hierarchical cosine distance are interesting. These modules are crucial for the whole framework of the proposed method and they are also tailored designs for the task studied in this paper.\n\n+ The experiments are comprehensive and convincing. Overall, they can validate the effectiveness of the proposed method, as well as the proposed modules.\n\n+ The paper is well-written and easy to follow.\n", "weaknesses": "Although the motivation of Figure 1 is relatively clear, it can be further explained. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for the positive comments. Below please find our point-to-point responses.\n\n### Comment_1: \n\n> *Explain the motivation of Figure 1 in more details*\n\nThanks for your suggestion. Figure 1 primarily illustrates the properties of the Poincaré disk space and the impact of cosine distance constraints on the mapped space. One intrinsic characteristic of this mapping, critical for our work, is the Conformality property derived from Riemannian geometry. It ensures that angles between curves or vectors in the Euclidean space remain consistent when projected onto the Poincaré disk.\n\nBy emphasizing this property in Figure 1, we intend to provide a clearer comprehension of the principles underpinning our method. Detailing this geometric phenomenon underscores the robust theoretical base of our approach and lends deeper insight into our motivation. In our revised manuscript, we will ensure this aspect is delineated more prominently.\n\n### Comment_2: \n\n> *Selection of appropriate hyperparameter*\n\nThank you for pointing out the importance of choosing appropriate hyperparameters. In our experiments across five datasets, we found a recurring pattern that offers a guideline. Typically, when clustering coarse-grained categories, the number of clusters that yields optimal results is roughly twice the actual number of fine-grained subclasses within that coarse category.\n\nWhile this heuristic emerged consistently across our tested datasets, we acknowledge that it may not be universally optimal. However, it does provide a good starting point for users. In practice, fine-tuning based on validation performance is always recommended to ascertain the best hyperparameter for a specific task. We will clarify this heuristic in our revised manuscript to guide potential users of our method.\n\n### Comment_3: \n\n> *Minor issues in paper writing*\n\nThank you for highlighting this inconsistency. We will align the use of \"Fig.\" and \"Figure\"  throughout the paper in the revised version. We appreciate your attention to detail and will carefully proofread the manuscript."}, {"review_id": "kAsb1Jmgp6", "reviewer": "Reviewer_tnZ4", "summary": "The paper adresses the issue of coarse to fine learning, _i.e._ coarse labels are present during training but the metric are computed on fine-grained labels. The paper introduces a method Poincaré embeddings with hierarchical cosine margins (PE-HCM). The paper suggests embedding the image representation in the hyperbolic space for coarse label classification to take advantage of its natural hierarchical organisation. It has a second branch that clusters with k-means instances that share the same coarse labels to simulate fine-grained pseudo-labels. It uses a hierarchical loss that aims at making the cosine distance between pairs match their relation, i.e. same instance label (same instance with different data augmentation), same cluster label (assigned by k-means), same coarse label (supervision), different coarse label (supervision). The paper validates experimentally the method on several benchmarks, CIFAR-100 and BREEDS - a collection of 4 datasets. The paper conducts ablation studies of the different components of the method.", "questions": "Q1. Line 145: The paper gives notations for coarse and fine-grained labels. Could the method take advantage of a hierarchy of classes rather than solely a coarse label?\n\nQ2. What is the impact of the granularity of the coarse labels, _i.e._ how much will the performance deteriorate when the coarse labels become more and more broad and on the contrary what happens when given the fine-grained labels?\n\nQ3. Line 227: the paper defines a distance that is used in the HCM module. It uses the Euclidian cosine distance, this is surprising as the paper states that it compares instances in the hyperbolic space. The paper should elaborate this design choice.\n\nQ4. Table 2: reports results on the BREEDS benchmark. I am not sure to understand what are the results indicated as “5-way” and “all-way”.\n\nTypos:\n\n- Table 1: Coarase —> Coarse\n- Table 5: European —> Euclidian", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The method is well justified. The adaptive hierarchical distance is interesting and works well experimentally.\n\nS2. Experimental results are encouraging.\n\nS3. Ablation studies are present.", "weaknesses": "W1. Some works are missing in the related works: MaskCon (MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset, CVPR 2023) and Grafit (Grafit: Learning fine-grained image representations with coarse labels, ICCV 2021) that both tackle the coarse to fine task.  And UnHyp (Unsupervised Hyperbolic Metric Learning, CVPR 2021) that uses hyperbolic space to create a hierarchy.\n\nW2. The paper lacks comparison to state-of-the-art methods such as MaskCon and Grafit. It could conduct experiments on the large scale iNaturalist-18 dataset.\n\nW3. The paper lacks comparison to unsupervised baselines, such as UnHyp or more recent state-of-the-art self-supervised-learning methods, such as Dino (Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021), iBot (iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022) etc.\n\nMinor:\n\nW4. The paper could remind the reader, in the benchmark datasets section, what classes are present in the different datasets, and what are the coarse classes on the different datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for the positive comments. Below please find our responses to some specific comments.\n\n### Comment_1: \n\n>  *Additional related works*\n\nThanks for your kind reminder. These works do share some connections with our method. We did not notice them because these works have different task-of-interest from ours and are seldom discussed in prior few-shot fine-grained literature. We will cite the recommend works and add the discussions (as detailed in the following responses).\n\n\n### Comment_2: \n\n>  *Comparison to SOTA methods such as MaskCon and Grafit*\n\nThank you for the suggestion. Since MaskCon and Grafit are proposed for a different task (i.e., retrieval), we conduct a comparison on both our few-shot fine-grained (FSFG) task and the retrieval task with CIFAR-100, as shown in the table below.\n\n\n| Methods                     | 5-way (FSFG) | All-way (FSFG)|Recall@1 (Retrieval) | Recall@5 (Retrieval) | \n|-|-|-|-|-|\n| Grafit (Reported in paper)  | -     |-   |60.57    | 82.32    | \n| MaskCon (Reported in paper) | -     |-   |**65.52**    | **83.64**    | \n| MaskCon (Our Replication)   | 81.05     |40.81   | 65.35    | 83.57    | \n| Ours                         | **85.45**     |**47.76**   |62.11    | 80.68    | \n\n**Analysis:** As shown, under FSFG, our method still achieves superior results over MaskCon (almost 7% improvement on all-way FSFG). Under the retrieval scenario (their specialized task), our method already surpasses Grafit but is lower than MaskCon. The reason might be: MaskCon's self-supervised design emphasizes the finest granularity and thus is favored by instance-based retrieval. In contrast, our approach harmonizes embeddings across multiple granularities and thus is favored by FSFG. \n\nGiven the time constraints, we couldn't finish the experiment on the large-scale iNaturalist-18 dataset in 7 days, but we pledge to incorporate results on this dataset in the final version.\n\n### Comment_3: \n\n>  *Comparison to unsupervised baselines and more recent SOTA SSL methods*\n\nThank you. We compare the two most recent unsupervised methods (DINO and iBot with their publicly released model) in the table below. It is observed that our method consistently outperforms both DINO and iBot across different metrics and datasets by a large margin.\n\n| Methods| CIFAR-100 || Living17|| Non-Living26||\n|-|-|-|-|-|-|-|\n|| 5-way| All-way| 5-way| All-way| 5-way| All-way|\n| Dino-ResNet50 | 51.23 | 12.67 | 83.62  | 44.31 | 72.58 | 33.34 |\n| iBOT-ViT-Base16  | 46.72   | 10.68 | 75.41   | 31.38  | 74.89 | 32.95 |\n| Ours-ResNet50    | **81.42**    | **36.28** | **90.94**   | **53.09** | **89.97** | **50.12** |\n\n### Comment_4: \n\n>  *The detailed classes in different datasets*\n\nThanks. We adopt the protocol dataset split in their original papers. Briefly:\n- Living-17, rooted at \"living thing\" in WordNet with coarse-grained classes at a depth of 5, includes salamander, turtle, dog, bear, and monkey.\n- NonLiving-26, rooted at \"non-living thing\" and also at a depth of 5, encompasses bag, boat, car, digital computer, and ship.\n- Entity-13, originating from \"entity\" at a depth of 3, features bird, reptile, and mammal.\n- Entity-30, also rooted in \"entity\" but at depth 4, includes classes like serpentes, building, footwear, and fruit.\n\nSince the number of classes is large, we do not have enough space to enumerate all the class names. Please kindly refer to the original paper for details. \n\n\n### Comment_5: \n\n> *Could the method take advantage of the hierarchy rather than solely a coarse label?*\n\n\nOur method already utilizes an estimated hierarchy. Specifically, given the coarse labels, we estimate the finer-grained classes through unsupervised clustering (Ln. 49) and form an estimated hierarchy (with inevitable noises). If we have access to the ground truth of fine-grained labels, the results are expected to be better.\n\n### Comment_6\n\n> *Impact of the granularity of the coarse labels*\n\nThanks for this good question. Generally, if the given labels become more fine-grained, the results become better. During rebuttal, we adjust the granularity by splitting the dataset into different classes (the larger number has finer granularity).\nThe results are summarized below.\n\n|  | Living17 | | | NonLiving26 |||\n|-|-|-|-|-|-|-|\n| Num of Splitted Classes | 9        | 17    | 34    | 13          | 26    | 52    |\n| 5-way                         | 82.84    | 90.94 | 90.58 | 82.53       | 89.97 | 88.91 |\n| All-way                       | 43.65    | 53.09 | 63.94 | 42.51       | 50.12 | 54.78 |\n| Intra-class                   | 50.74    | 57.11 | 68.46 | 53.06       | 58.63 | 62.68 |\n\nThank you for pointing out this new aspect. We believe this investigation provides valuable insight into our method, as well as the FSFG task itself. \n\n### Comment_7\n\n> *Using Euclidean cosine distance in the hyperbolic space*\n\nThe property of Conformality, as discussed in Ln. 181, plays a pivotal role by preserving the angles between feature vectors of instances within the hyperbolic space. This fundamental characteristic ensures that regardless of the choice of angular measurement, two feature vectors with the same angle will maintain that angle in the hyperbolic space. Consequently, employing the Euclidean cosine distance for this purpose is a valid approach. More importantly, during our experimental process, we observed that optimizing with the Euclidean cosine distance provided more stability in gradient computations compared to the hyperbolic cosine distance. This observation in the empirical results guided our choice to use the Euclidean cosine distance, as it aligned well with our objectives and offered more robust training. In the final version, we will follow your suggestion to include a more detailed explanation. \n\n### Comment_8\n\n> *The meaning of \"5-way\" and \"all-way\"*\n\n\"5-way\" refers to tasks that involve 5 randomly selected fine-grained categories. \"All-way\", encompasses all available fine-grained categories in the dataset. "}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We provide point-to-point responses to each reviewer, as well as a supplmentary PDF for some visualization results.

Meta-review: This paper proposes a method that leverages hyperbolic space to embed visual representations for addressing the task of fine-grained learning based on coarse labels. The effectiveness of the proposed approach is demonstrated through experiments on benchmark datasets. The majority of the reviewers acknowledge the novelty of this paper and the comprehensiveness of the experiments. Detailed responses and discussions were provided regarding some of the shortcomings pointed out by the reviewers, such as the absence of comparisons with certain baseline methods.   
Overall, the proposal is technically reasonable and has a certain enlightening significance for the utilization of label relationships. The experiment was sufficient, including multiple fine-grained learning benchmark datasets and existing SOTA methods, and the proposed method achieved consistent improvements. Therefore, I am inclined to recommend accepting this paper.
Decision: Accept (poster)