Superclass-Conditional Gaussian Mixture Model For Learning Fine-Grained Embeddings
Authors: Jingchao Ni
Wei Cheng
Zhengzhang Chen
Takayoshi Asakura
Tomoya Soma
Sho Kato
Haifeng Chen
Venue: ICLR 2022 Spotlight (2022)
Keywords: Deep learning
represenation learning
generative model

Abstract: Learning fine-grained embeddings is essential for extending the generalizability of models pre-trained on "coarse" labels (e.g., animals). It is crucial to fields for which fine-grained labeling (e.g., breeds of animals) is expensive, but fine-grained prediction is desirable, such as medicine. The dilemma necessitates adaptation of a "coarsely" pre-trained model to new tasks with a few "finer-grained" training labels. However, coarsely supervised pre-training tends to suppress intra-class variation, which is vital for cross-granularity adaptation. In this paper, we develop a training framework underlain by a novel superclass-conditional Gaussian mixture model (SCGM). SCGM imitates the generative process of samples from hierarchies of classes through latent variable modeling of the fine-grained subclasses. The framework is agnostic to the encoders and only adds a few distribution related parameters, thus is efficient, and flexible to different domains. The model parameters are learned end-to-end by maximum-likelihood estimation via a principled Expectation-Maximization algorithm. Extensive experiments on benchmark datasets and a real-life medical dataset indicate the effectiveness of our method.

Published as a conference paper at ICLR 2022
SUPERCLASS-CONDITIONN GAUSSIAN MIXTURE MODEL FOR LEARNING FINE-GRAINED EMBEDDINGS
Jingchao Ni', Wei Cheng', Zhengzhang Chen', Takayoshi Asakura', Tomoya Soma", Sho Kato', Haifeng Chen'
NEC Laboratories America, 'NEC Corporation,
R 'Renascience, Inc.
[jni, weicheng, zchen, haifeng) inec-labs, com.
(takayoshi asakura, tomoya- @nec. com, *kato@@reasciennce jp
ABSTRACT
Learning fine-graine mne-grained embeddings is essential for extending the generalizability of models pre- rre-trained "coarse" labels (e.g., animals). It is crucial to fields for which fine-grained labeling (e.g., breeds of animals) is expensive, but fine- fine-grained prediction is desirable, such medicine. The dilemma necessitates adaptation of a "coarsely" pre-trained model to new tasks with few "finer-grained training la- bels. However, coarsely supervised pre-training tends to suppress intra-class vari- ation, which is vital for cross- goons--gaullartty adaptation. In this paper, we develop a training framework underlain by novel pnrrelass-conditiona Gaussian mixture model (SCGM). SCGM imitates the generative process of samples from hierar- chies of classes through latent variable modeling of the fine-grained subclasses. The framework is agnostic to the encoders and only adds a few distribution re- lated parameters, thus is efficient, and flexible to different domains. The model parameters are learned end-to-end by maximum-likelihood estimation via princi- pled Expectation- MaaitionnMMiiiiiiiinnnn algorithm. Extensive experiments on benchmark datasets and real-life medical dataset indicate the effectiveness of our method.
INTRODUCTION
Training deep models with sufficient generalizability is of fundamental importance, which demands immense training data with fine-grained annotations (Krizhevsky et al., 2012; Brown et al., 2020). In many fields, however, data labeling requires domain-specing knowledge, such as medicine (Sohoni al., 2020), thus prohibitive, and infeasible to be exhaustive. In this case, data for model training may only be "coarsely" labeled, while later the model is tested finer-grained classification task (Bukchin et al., 2021). For example, consider an event prediction task for dialysis patients (Inaguma et al., 2019). Hemodialysis is major renal replacement therapy for patients with end-stage renal failure. These patients have to take hemodialysis thrice a week, each lasts for 4-5 hours. During the treatment, unexpected events, such as muscle cramp, perspiration, and dizziness, may happen result of lowering blood pressure, which need intensive medical care, thus should always be avoided. It is therefore an important medical issue to predict such events before initiation of hemodialysis. In this task, binary labels, which mark the incidence of an event, can be collected. In contrast, finer- grained labels that annotate different subtypes of events are seldom recorded. Since distinguishing different subtypes facilitates precise diagnoses, and helps physicians assess the risk for deciding whether to perform a hemodialysis (with certain precautions), it is desirable that model trained with coarse (binary) labels can perform well on a finer-grained multi-class (subtypes) task.
To fill the gap of granularity between the training and testing scenarios, a practical way to collect a few new records for a patient, with their fine-grained annotations. These data constitute a support set for fine-tuning a pre-trained model to the specific data distribution induced by the annotations of the target patient, for whom the adapted model is used for future predictions. Although massive fine- grained annotation is impractical, annotating a few-shot set is feasible. In this work, we are interested in such a Cross- Granularity Few-Shot (CGFS) learning problem, where model pre- trained of coarse classes (denoted as superclasses), needs to adapt to an unseen set of fine-graines target classes (denoted subclasses). The target subclasses could be descendants of the superclasses (as in
set

---

Published as a conference paper at ICLR 2022
the aforementioned example), or could descend from other superclasses that unobserved during pre-training. To be practical, the adaptation should only use a few samples from the subclasses.
The CGFS problem is not limited to the above application. It occurs in model's ' lifespan when an application requires separating some subclasses from the superclasses yet when the training dataset was created these subclasses were unannotated. For example, it could occur in detecting rare pathol- ogy or variants using medical images that were coarsely described (Oakden-Rayna et al., 2020), or personalizing a generic model trained on all historical users a specific customer (Luo et al., 2020).
Despite its significance, CGFS cannot be trivially solved by regularly training models with coarse labels, because typical losses for supervised learning aim maximize inter-class boundaries but ne- glect intra-class variation. Thus, subclasses may arbitrarily and unevenly spread in every superclass. Recently, Bukchin et al. (2021) proposed to integrate coarse supervision and contrastive learning (within superclasses) for solving CGFS. However, their approach cannot be readily used for medi- cal records which typically contain static profiles and time series (Che et al., 2016). This is due to the absence of standard augmentation method for generating constrastive pairs some data other than images. Also, since contrastive learning does not model subclass explicitly, their solution could be suboptimal (as evaluated in Sec. 4). Moreover, as their model was built upon MoCo (He et al., 2020), and maintains many dictionaries for data sampling, its computational costs are high (Sec. 4).
In this work, propose novel Superclass-Condeee Conditional Gaussian Mixture model (SCGM) to learn fine-grained embeddings for the CGFS problem. Our contributions are summarized as follows.
SCGM is agnostic to the encoder, thus is flexible to different applications. It models the generation of samples from hierarchical classes, and explicitly represents the unobserved subclasses by latent variables, without assuming their identities. It dynamically computes a Gaussian mixture for every sample conditioned on its superclass, and the model forms a hierarchy of Gaussian mixtures.
SCGM only adds small overhead to encoder, for parameterizing its distributions, thus is ef- ficient. The model parameters are learned end-to-end by maximum likelihood estimation via a principled
(EM) algorithm. We also theoretically linked our loss func- tion to InfoNCE (Oord et al., 2018), explaining its effectiveness from a contrastive perspective.
In the experiments. we evaluated SCGM on both benchmark image datasets and a real-life medical dataset. Since SCGM is compatible with contrastive learning, also tested it with momentum encoder (He et al., 2020). The results demonstrate SCGM on generic encoders has already out- performed the state-of-the-arr (SOTA) baselines, with less computational costs, and achieves boosted performance in some cases with momentum contrast integrated.
2 RELATED WORK
To our best knowledge, this is the first work to develop an SCGM model for underlying a framework that enables tackling CGFS across domains. The most relevant work (Bukchin et al., 2021) combines superclass-win contrastive learning with coarse classification for preserving intra-class variation. Another work (Yang et al., 2021) used a three-step approach that pseudo-labets the embeddings (pre- trained by coarse classification and batch-wise contrastive learning) by clustering every superclass. were used to re-train the encoder. A similar three-step method (Sohoni et al.,
The pseudo-fine-labee 2020) used a different loss for maximizing the worst-case expected accuracy over the pseudo-labeled subclasses. As discussed before, the former two methods require non-trivial searches for a suitable data augmentation method, which may be unavailable for some non-image data. By splitting training steps, the latter two methods could lead to suboptimal pseudo-labeling and misleading labels could confuse the downstream steps. In contrast, our model is end-to-end, of the subclass during coarse training. Moreover, it can achieve better performance without using as many computational resources as the model of (Bukchin et al., 2021) (as evaluated in See. 4).
explicitly infers the posterior
Learning with coarse supervision. Several works deal with coarse/fine labels from the perspective of weakly supervised learning (Zhou, 2018) than to tackle the CGFS problem, including training methods that take advantage of a mixture of either balanced (Ristin et al., 2015; Guo et al., 2018; Taherkhani et al., 2019) or unbalanced (Hsieh et al., 2019; Liu et al., 2019; Robinson et al., 2020) coarse and fine labels. Among them, Liu et al. (2019) addressed few-shot learning problem, but the model training assumes access to some fine labels and a graph of class hierarchy, which are unavailable in problem. Thus it cannot be adapted to solve our problem.
2

---

Published as a conference paper at ICLR 2022
Few-shot learning. Meta-learning has become popular idea to handle the few-shot learning prob- lem, which derives metric-based (Vinyals et al., 2016; Snell et al., 2017) and optimization- based methods (Finn et al., 2017; Nichol et al., 2018). The idea has been extended to semi-supervised (Ren et al., 2018), unsupervised (Hsu et al., 2018) and semantics-augmmnnted (Xing et al., 2019) 9) sce- narios, when labels scarce. However, none of them explores coarse labels for cross-granularity learning. Recently, many works observed that learning embeddings on all classes (without episodic training), followed by simple fine-tuning, is superior SOTA meta-learning methods (Wang al., 2019; Dhillon et al., 2020; Tian et al., 2020). In this work, similar to (Bukchin et al., 2021), we focus on this paradigm to learn useful embeddings, and do not use meta-learning for pre-training.
Embedding methods. The emerging self self- supervised methods, such as contrastive methods (Oord et al., 2018; Chen et al., 2020a; He et al., 2020), are appealing in their ability to attain comparable embeddings to the supervised counterparts, and even surpass them when transferring to other tasks (He et al., 2020; Tian et al., 2020). Beyond instance-win -wise contrast, recent methods have explored instance-cluster contrast to boost performance (Caron et al., 2020; Li et al., 2020). In unsupervised methods, joint embedding and clustering has been found beneficial (Caron et al., 2018; Asano et al., 2020). Whereas, these methods never exploited coarse supervision. Thus, their embeddings/clusterss do not necessarily reflect intra-class variation, which is important to the CGFS task.
Vanilla Gaussian mixture (GM) model is unfit for the CGFS scenario. A recent work (Manduchi et al., 2021) extended GM to constrained clustering by conditioning every sample with a prior clus- tering preference. It is neither supervised by coarse classes nor hierarchically structured. Conven- tional hierarchical GM is used for hierarchical clustering (Goldberger & Roweis, 2005; Olech & Paradowski, 2016; Athey et al., 2019) by applying GM agglomeratively or divisively. These unsu- pervised methods only infer clusters, but do not pre-train embedding models for task adaptation.
3
GAUSSIAN MIXTURE MODEL
Firstly, a word about some notations. Let Duran {x;; y:)) be n sample-label training pairs, where E V super = {1...., ...., c} is superclass label. Each x; is associated with a latent (unobserved) subclass label Y/ E Vunh = (l....,). Vauh relates to Veuper by hierarchical structure, i.e., Vauh can be partitioned into disjoint sets Vo Vauh.c. such that if 9; Junh.j. then y; = (1 S j S c). Let fe be an encoder (i.e., backbone network) that is trained on Derain (without knowing Vob.). maps x; to a d-dimensional feature fx(x.). At test time, given k-shot support set for subset Duo Vuub of m subclasses, i.e., D Psupport = {(x,, 0:)10 E yi ymk i=b the task is to train a classifier C: Rd + sub with optimal accuracy on a test set of ymmb subclasses. In our experiments, we also explored the case when the subclasses belong to superclasses that not used for pre-traie pre-training As discussed in Sec. 2, our focus is to train So for good embeddings without modifying So during the adaptation-a paradigm with SOTA few-shot performance (Dhillon et al., 2020; Tian al., 2020).
Formally, let (x,) = v,, our goal is to find the model parameter 0 that maximizes the likelihood of the posterior distribution pe (y: vv) on the observed data in Duain for classification tasks (Grathwohl et al., 2019). To model the unobserved subclasses, we associate every v; with a latent variable z to indicate to which subclass v, belongs. Suppose there are possible subclasses, the log- likelihoodd to maximize can be rewritten by marginalizing out the latent variables.
((Dinin 0)
log ppllvvv)) =
Σ log Σ plus plunls))po=sl)
where the distribution (2 vv)) specifics the subclass membership of v,. and p(y)22) associates 2; with subclass partition Vuub-n. Unlike some previous works (Sohoni et al., 2020), which searched the number of subclasses for every superclass using a quality metric, we only assume a total number of subclasses, and seek to infer their relationship with superclasses, iee, p()))=)). without any prior on subclass partitions, so that the model is more generalizable.
3.1 PARAMETERIZATION OF PROBABILITIES
In Eq. (1), both p()) and po(2 |v) are not specified, and involve non non-maraettiic variables. To make it solvable, we introduce our SCGM that models the generative process of the embeddings.
3

---

Published as a conference paper at ICLR 2022
(a)
Suppose y
(b)
Supoose
(c)
Σ
Generic
Gaussian
Vi
mixture
π
Dynamic misture probosiity
SCGM
Σ
Σ
elubi
Polo My Fee
ploty
y
πt'
Z
Conditional distribution vfv, Ri are meens superclesses (lee, yi M2. Aa are means subclasses (ie.2))
Conditional distribution vf
μ
Figure I:
An illustration of (a) embedding distribution tuned by SCGM via dynamic mixture prob- abilities n², when y = 1, and (b) when y = 2. where (c) summarizes SCGM's graphical model.
Let 1i and Σ (1 و S r)) be the mean and variance of the j-th mixture component, and тy be its corresponding mixture probability. a generic GM model, the generation of v, involves two steps: (1) draw a latent variable from categorical distribution on all mixture components, and (2) draw V; from the Gaussian distribution (H,,,) >,,) (Bishop, 2006), as illustrated by Fig. I(c).
In step (I), the categorical distribution is usually defined on |- Tr_ E RT, i.e., the static mixture probabilities, which cannot reflect different samples' superclasses. In light this, for each V,. we propose to define x' = | .... mH) to be conditional on ys. which dynamically changes w.r.. different samples. To this end, we introduce py and Ey, as the mean and variance of superclass yıs and compute T| softmax (- (22)) where Dy (-) measures the Mahalanobis distance between data point and the superclass distribution (( E..) Basically, xÿ evaluates the density ratio the mixture component mean M4 by regarding it as a sample from y: s distribution.
exp
(1,, P ".. (1,, p..))
(N Pe Ey,)
exp 1/2 (M,, P,.)) Σ (My - (2,)))
E PN (Hy P Ev.)
where (.) represents the density function of multivariate Gaussian distribution.
Our generative process can be summarized as
1. for each sample index é:
(a) draw superclass label ys Categorical(p(s:)
(b) for j = I,...,?"
using Mahalanobis distance Dy: (-) (c) draw latent subclass variable for the i-th sample 2, ~ CCtteooriaal(mii.... x[]]
i. compute πj = softmax(-D,,,
(22) (1,,
(d) draw feature vector v. ~ ~N(, E.,)
where p(ı)) is prior superclasses, which can be drawn from Dirichlet distribution Dir(a:) (the conjugate of categorical distribution). and a can be estimated by the ratios of different labels in D Dirain- p(u)) can also be
uniform prior, for unknown datasets and better generalization.
As can be seen, steps (a)(b) specify a (static) mixture of superclasses, and steps (c)(d) specify a (dynamic) mixture of subclasses. The whole process establishes hierarchical GM distribution. Fig. (a)(b) illustrate the process using 2 superclasses and 3 subclasses, where the embedding distribution p(ve(B)) dynamically changes w.r.t. v,'s superclass as a result of updated r'. For example, when = 1 (Fig. (()),, V; is more likely to be generated from the first two subclasses. This yı -daaussteed Gaussian mixture distribution highlights the first two subclasses, and will be used to fit data with = 1 during model training. Fig. ((c) bottom illustrates the graphical model of SCGM, which distinguishes itself from the generic GM in Fig. I(c) by (1) the condition ye, and (2) the variable T..
Now, if we consider Ty, and Pz as the surrogate representations of yi and z; in the embedding space, we can specify two probabilities p(z; y:)) and p( (1)))))) using their corresponding densities: p(2]l))) = pN (H,, Myy. ). p(v./2.) = pN(v//p., Ex,)
from which, the probabilities p(y: (2;) and p(z= vv)) in Eq. can be derived by using Bayesian rules as (the detailed derivation deferred to Appendix A.
PN (N P.,. Er, )p(())
(v Ms Ex, br
(v(l)) Ei,=1 P0::iPN(C (1,, My E. P((()
po.((2lv)) CN Eg )
4

---

Published as a conference paper at ICLR 2022
where 0 are the parameters of encoder fe. & includes our added distribution parameters. To reduce complexity, in this work, we investigate the feasibility of using isotoopic Gaussian with tied variance, i.., Σ.. I and y, I which turned out to be efficient in experiments. Here, I is the identity matrix, a and a are hyperparameters. Empirically, it is sufficient to set ā = and tune o to adjust the relativity between super- and subclasses. Also, tied variance is commonly used trick in Gaussian discriminate analysis (GDA) for generative classifiers (Lee et al., 2018).
Finally, substituting the factors in Eq. (1) with Eq. (4), whose probabilities are specified and param- eterized, we get our SCGM induced loss &SCOM (Dinin; 0, Φ). where & (1 {
3.2 MODEL OPTIMIZATION VIA EXPECTATION-MAAIMMIZATOOO
It is hard to directly optimize (ecam((rrann 0, Φ)), because exact posterior inference intractable (due to an exponential searching space). To solve it, we resort to variational methods, and introduce an approximated posterior (zz M/). where y is included so that the inference also conditioned on superclass. Then, lower-bound of Eq. (1) is derived as (the details are in Appendix A.2)
EsCOM (Duain: O, Φ)
Pylts,veedd) (ogg 22)) + oggro.oallv,, - log ((2)) (vll))
(5)
which can be maximized by alternately inferring the posterior (<2//ve ys) and solving the model parameters {0, }} through an ExpectationMMaximizattooi (EM) algorithm.
E-step. This step is to infer ((2:) vi, y:) when fixing model parameters. A straightforward way is to apply k-means the embeddings Vi, similar to (Caron et al., 2018). However, as discussed in (Asano et al., 2020), alternately applying k-means and learning embeddings without any constraints will lead degenerate solution, i.e., all samples are assigned a single (arbitrary) subclass. Also, k-means is unaware of the superclass Yi as it doesn't 't optimize Eq. (5). To address this problem, inspired by (Asano et al., 2020), let Qz two r X matrices, we enforce an equal partition of subclasses by constraining Q to belong to the transportation polytope (Appendix A.3), and rewrite Eq. (5) as (the derivation is in Appendix A.4)
q(25/Ve. y/ and P zu. P p (2: v:) be
min - (TT(Q" T log P) x H(Q)) where @ = fo E "I QEQ where H(Q) = Ei Qij log Qı is the entropy function, 1,, is the vector of ones in dimension n, and À 2 1 controls the regularization (it also further relaxes the lower-bound in Eq. (5)).
- 1
QL, =
(6)
n
The problem in Eq. (6) is regularized instance of the optimal transport problem, whose minimizer can be written as Q* = Diag(u))PDiaa((w) where E IR"" and v E IR" are renormalization vectors These vectors can be efficiently solved by the iterative Laikhorn-Koopp algorithm (Cuturi, 2013). Once Q" is found, rounding its values and using discrete codes adds more algorithmic stability.
M-step. Fixing q(2; V y/),, the model parameters {0, Φ} can be efficiently solved by stochastic gradient descent (SGD). Combining Eq. (4) and Eq. (5), the loss to minimize is (Appendix A.5)
1 Σ (( vi) log E exp (m! Thy, 05::07 (80) p(v)) log Σ exp (v. 11/ /00)
exp exp(p (M.,, P,,, //"))(())))))))))))))))())
exp exp (v. M4,, /,,//²?) /0²)m²,
60.0
It is noteworthy that the two terms in the bracket have similar form to InfoNCE (Oord et al., 2018). Therefore, Eq. (7) can be interpreted from a contrastive perspective, where the positive pairs in both terms are specified by g(>.| Vs, 34) from E-step, i.e., (z;, V/) and (z;, v:) that induce the maximal ((2:/V,. yt), and the variances d² and σ² resemble temperature scaling. Thus, the M-step seeks to push the embedding v; to its closest subclass M2 which should meanwhile be close to its superclass
p.. in contrast to the distances of the negative pairs. In the second term, Tं, (Eq. (2)) serves weight to select μt, that is close to n. As such, our problem can be seen as a generalized InfoNCE for subclass- and superclass-level contrastive learning.
In practice, we found it is beneficial to integrate Eq. (7) with cross-entropy loss tcE (on superclass)
{(Pra O Φ) Log (Prani 0 (ū, 1=)) + 7ls. (Pumii 0 ())
(8)
where fer uses superclass means { 14_ as the classification weights, 7 is a trade-off parameter. The detailed algorithm for training SCGM can be found in Algorithm 1 in Appendix A.6.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, we first evaluate SCGM on benchmark datasets and compare it with SOTA methods. Then evaluate SCGM for a model personalization task real-life medical dataset.
The table below summarizes the benchmark datasets: (1) BREEDS (Santurkar et al., 2020) includes four datasets [Living17, Nonliving26, Entity 13, Entity30) derived from ImageNet with class hierar- chy calibrated so that classes on the same level are of similar visual granularity (i.e. the granularity of images for distinguishing classes); (2) CIFAR- 100 (Krizhevsky, 2009); and (3) mageed
(Ren et al., 2018), which has 34 superclasses as per the ImageNet hierarchy. and was divided into 20/6/8 splits for (disjoint) train/val/test sets, For BREEDS and CIFAR- 100, the val set 10% of the train set. These datasets have been used in (Bukchin et al., 2021).
Dataset
Living17
Nonliving26
En
Entity:0
CIFAR-100
Tiered
# Coarase classes
17
26
13
30
20
206888
# Fine classes
104
260
240
100
351/9//100
* Train images
88K
132K
334K
307K
50K
448K
# Test inuges
3.4K
5.2K
13K
12K
10K
206K
Image resolution
224
224x224
224x
224x224
32x32
84
4.1 EXPERIMENTAL SETUP
Baselines. We compare SCGM with the most relevant SOTA models on embedding learning, includ- ing self-supervised models: (1) MoCo-v2 (Chen et al., 2020b) trained on each of the above dataset;
(2) MoCo-v2-ImageNNet pretrained on ImageNet by Chen et al. (2020b); (3) SwAV-ImageNet is a pretrained model by (Caron et al., 2020), which takes advantage of cluster-level contrastive learning. Note that pretrained models on full ImageNet saw more data during training than did SCGM. (4) ANCOR (Bukchin et al., 2021), introduced in Sec.1 and 2: (5) GEORGE (Sohoni et al., 2020), a three-step framework that uses a separate clustering step to pseudo-fine-labe the dataset; and
(6) SeLa (Asano et al., 2020), an unsupervised algorithm for joint clustering and embedding. The detailed setup of these models was deferred to Appendix C.I.
Similar to (Bukchin et al., 2021), we include natural baselines trained on superclasses: (7) "Coarse", uses encoder fe followed by classifier C: (8) "Coarse+", uses fe - E + C, which adds an embedder E. Also, baselines trained on subclass labels represent performance upper-bounds: (9) "Fine", uses fe - C; and (10) "Fine+". uses fe - E + C. For our method, since it is a flexible framework, we tested with a generic encoder, and momentum-basee wise dictionaries similar to ANCOR (Bukchin et al., 2021). We denote these two variants by SCGM- and SCGM-A, where the training loss (i.e., Eq. (8)) of the latter adds an InfoNCE term with angular normalization. The detailed implementation of SCGM-A can be found in Appendix C.2.
encoder with superclass-
Implementation. The encoder is ResNet-50 (He et al., 2016) for the 224x224 images in BREEDS, and 12 for the small resolution images in CIFAR- 100 and tieredImageNet, in few-shot learning works. The output dimension of these networks is 2048 and 640, respectively. Similar to (Chen et al., 2020b), two types of embedder E were tested: (1) fc layer: d + e, and (2) MLP: d + - e, with e = 128. At test time, some models performed better with E dropped better otherwise. For each model, the better case was reported. For training, used cosine annealing with warm restarts schedule (Loshchilov & Hutter, 2017) with 20 epochs per cycle. The batch size was 256 for BREEDS, 1024 for CIFAR-I00, and 512 for tieredImageNet. The learning rate was 0.03 for BREEDs, and 0.12 for CIFAR- 100 and tieredImageNet. The weight decay was 14-4, All models were trained with 200 epochs. The best training practices of (Tian al., 2020) were also applied. For SCGM, we set 7 0.5, の² = 0.1, and X = 25 (A follows (Asano et al., 2020)). The number of latent variables r varies w.r.t. different datasets, and was grid-searched from 100 to 500 with step size 50. All hyperparameters were set according to validation sets.
is common
while some
4.2 EXPERIMENTAL RESULTS
Following (Tian et al., 2020), we report the mean accuracy and the 95% confidence interval of 1000 random episodes with 5- 5-way/all-war way s-oho,, 15-query tests. Unless otherwise stated, k = 1. Effects of more shots are in Appendix D.I. For each episode, 5 augmented copies were created per support sample. A logistic regression classifier was trained on the support set embedded by each model with

---

Published as a conference paper at ICLR 2022
Living
Nonliving
Ennryy3
Entity:0
Method
5-way
All-way
5-way
wal-way
S-way
wal-way
waway
All-way
(upper-bounds)
91.10:10.47
58.9520.16
85.25:00.49
47.6810.13
90.0
50.1910.08
91.6500441
56.54:0099
Finet (upper bounds)
90.75:1:0.88
62.65:10.18
90,33:10.47
60.68:10.14
94.72.10.33
65.18
94.02.
63.72:10.10
MoCo-V2
50.66=20.70
18.57#0.11
63.51=0075
21.07=0.11
82,00=00667
33.06=00007
80.37=0.22
28.62=0.06
83.85:4:0.65
40.59:10.15
77.02::0.72
35.13:1.111
85,07:10.61
35.78:1000
03,44:0.64
32.00:10,08
SwAVV-ImgeNet
81.13:00.63
39,57=00.15
75.67=+0.70
35.28:+0.12
81.76:+0.61
34,53+00.07
80.76=0.63
31.92=00008
GEORGE
85.50:1:0.63
32.10:10.11
80.75:10.62
31.07:10.10
77.68:00.88
0..5
83.41=0.59
21.12::0.06
Sel.a
52.46.100.68
14.87:10.100
54.08.10.72
14.23.10.07
63.29.10.73
13.83
0.0.7
12.20
Coarse
85.12:40074
13.83-10.10
83.53-40.64
13.524-0.11
82.33:40..1
17,49:40.04
87.03-40.54
24.01 40.06
Coarsc+
79.29==0.65
37,44=0.112
75.91=0.66
36,80++0111
83,23+00.66
31.15=00077
8481+=0.11
33 33.22=00.08
ANCOR
89.23:+:0555
45.14:20.12
06,23=+0.54
43.10=20111
90.58:00.54
42.29::0.08
88. 08.1
41.79=0.08
ANCOR.fe
90.41.10.57
46.19.10.16
08.77.10.54
45.34.1..33
89.05.10.58
18.52.10.08
91,8410.49
00.33.10.10
SOGM-G G
89.72:1:0.54
48.74:1:0.15
89.87
49.25.1..13
90.15
40.00:10.008
92.90 0.46
42.17:10.08
SCGM-A
90.97:1:0.55
0.13
88.78:40.55
46.93+0013
08.48:0.559
41.07
91.22
4..14:0.099
Table 1: Comparison on BREEDS datasets. Bold and underlined numbers are the best and second best results.
Method
5-way
All-way
Method
Living17
Nonliving26
Entity13
Entity30
Fine
74.36.10.68
28.82.10.11
Fine (upper-bounds)
66.60:10.89
66.3110992
04.20::0.63
65.74:0068
Fine+
75.53:+00.68
31.35:t0.11
Finc+ (upper-bounds)
70.72=00.2
74.02=0.91
72.24=0.60
73.86=0.67
MoCo-v2
48.07=00.08
10.61=0..06
McCo-v2
41.38:10.62
45.21
31.7811.49
41.67=0.599
GEORGE
70.64+0070
24.54:00110
McCo-v2-Iman 22ImmaeNNet
$1.73:10.79
53.69110.4
42.78:10.54
45.13110.60
ScLa
38.52:1:.60
5.87:1:005
SwAV- ImageNet
52.26-10.73
54.77:1..00
42.9511.554
45.78-10.00
Course
74.40:10.70
27.37.10.11
GEORGE
38.17=00.52
43.55=00.0
21.42:0.300
30,79=00.47
Coarse+
70.69:::0.99
16:1010
SeLa
40.58=0.61
42.56=0..0
23.90=:0322
31.02=0.46
ANCOR
74.56.1.0.70
29.84
Course
38.141:0.53
40.3810.62
16.01::0.22
28.2010.44
ANCOR-fe
74.73:+0073
27.32:40.10
Courset
40.70.10.59
50,21110.3
37.1310.488
13.10.
SCGM G
76.19.10.73
29.92.10.11
ANCOR
48,77=00.71
49.64=0.88
42,00=00.47
45. 07
SCGM-A
77.37:400777
25.91+00.10
ANCOR.fe
51.07=00.82
13.51+11000
41.83+0.48
47.82=0.65
SCOM-G
53,48:0881
57.32:1004
43,8910.58
06.80
SCGM-A
53,88+00900
15.12=1.00
05.09
50.02=0.71
Table 2: Comparison on CIFAR- datasets. Bold and underlined numbers are the best and second best results.
Table Intra-class few-shot learning results of the compared methods on BREEDS datasets.
L2 norm The model with the resulting classifier was used to classify the query samples. The test classes of each episode were random subset of all subclasses (or all of them for all-way tests). Two cases were evaluated: (1) unseen subclasses of seen superclasses (on BREEDS, CIFAR-100), and (2) unseen subclasses of unseen superclasses (on tieredImageNett, On CIFAR- 100 and teeredllageeeet ImageNet-pratraingg models were excluded since pretrained ResNet- 12 models are unavailable.
Evaluation case (1). We first evaluate when test classes are unseen subclasses of the training super- classes. Table and 2 summarize the results. For ANCOR, its variants when the embedder & is fc (with "-fé") or ML. (without '-f') were included as either variant does not consistently outperform another. The embedder setup of all models are in Appendix C.I. From the tables, we have several observations. First, models that cannot leverage superclasses, such as MoCo-v2 and SeLa, are infe- rior to Coarse(+). Although full ImageNet- models used larger training datasets, they are unable to fill the gap of superclass supervision, compared with Coarse(+). Second, the limitation of GEORGE indicates separate clustering step may induce misleading pseudo-labels that degener- ate embedding quality. Third, ANCOR and SCGM consistently outperform Coarse(+), and SCGM significantly outperforms ANCOR(-fe) in most all-way cases. With generic encoder, SCGM-G is suprior or competitive to ANCOR(-fe). Using momentum contrast, SCGM-A further gains on some datasets. Note that 5-way has less room to improve than the all-way cases, because the random sub- classes are more likely to be from different superclasses, which may degrade the evaluation to coarse classification. The results suggest SCGM could perform better when more subclasses are from the same superclass. To investigate it, we evaluated an "intra-class" case when all subclasses of ran- dom superclass were sampled in each episode. Table indicates SCGM got more improvements in this challenging task, implying its embeddings distinguish subclasses better than other models.
Moreover, Table compares the computational costs of ANCOR and SCGM-G, from which we can see training ANCOR re- quires much more resources and time than SCGM-G as it inherits MoCo-v2 (Chen et al., 2020b). Also, its memory costs vary lot
ANCOR
SCGM.G
Dataset
memory
time (h)
size
memory
time (h)
size
Living77
$1,98G
16.43
0.74G
29.63G
8.09
0.09G
Nonliving26
55.27G
25.42
1.02G
29.63G
12.94
0.09G
Em
50.35G
65.45
0.62G
29.63G
43.94
0.09G
Emiiyy
56.91G
1.15G
29.63G
34.56
0.09G
Table 4: Computational costs on 4 Quadro RTX6000 24G GPUs.
7

---

Published as a conference paper at ICLR 2022
Method
b-way-1-
5-way-S-
All-way-
(a) ANCOR
(b) SCGM-G
shol
shol
1-shot
70.15=0.70
84,96=0.47
15.42=0.06
Fine+
6..33
84.43:10:47
15.22.00.66
MoCo-V2
53.19=0.68
70.90=0..58
8.35::0.04
GEORGE
60,97:00770
77.72:00.55
10.24=0.05
SeLa
47.96=20.33
66.39=0,37
7.17::0.04
C2F VSME
60.54:0.79
75.22-00.63
N/A
Cuarse
61,61
75.72.10.56
8.42.1.0.04
Crarse+
64.07:4:0.75
78.41-00.58
10.70=0.005
ANCOR
02,86=00,70
79,93=0.52
11.97=0.06
ANCOR-fe
60.43-100.69
78. 1853
01.04=0.05
SCGM-G
6..64::0.71
0.52
12.32-0.05 0.05
SCGM-A
64.47:10.71
0..58
11.75
Figure 2: The tSNE visualization of the emheddings Icarned
by ANCOR and SCGM-G, Star represents superclass means.
Table 5: Comparison on tieredImageNet..
"+" marker represents subclass means of SCGM-G.
by datasets since its dictionaries are superclass wise whose size changes w.rt. the number of super- classes. In contrast, SCGM-G had lower and stable costs, with better performance, thus is more favorable practical deployments.
Evaluation case (2). Since tieredImageNet has distinct train/val/test sets, it was used to evaluate the case when the testing subclasses is out of any training superclass. All models were trained with the superclasses of the train set, and tested on the subclass labels of the test set. Table 5 presents the results. Another three-step method C2F-VSME (Yang et al., 2021) was included as the same evaluation has been performed (but their code is unavailable). First, from the smaller gaps between Fine(+) and Coarse(+), this task is more difficult than the previous case. Similar to Table 1, MoCo- v2 and Sel.a inferior to Coarse(+), underlining the benefit of superclass supervision, test subclasses are not their descendants. Compared to GEORGE and C2F-VSME, ANCOR and SCGM can better leverage superclasses, by using coherent learning frameworks. Both SCGM-G and SCGM-A outperform ANCOR(-fc) in most cases, demonstrating the effectiveness of explicit modeling of latent subclasses, in both evaluation case (1) and (2).
when
Visualization of embedding. To understand how SCGM uncovers the subclass structures, we vi- sualize a random batch of embeddings using tSNE (Maaten & Hinton, 2008). Fig. 2 presents the visualization results of ANCOR and SCGM-G on Living dataset, where colors mark different su- perclasses, "4" markers are the learned subclass means by SCGM-G, and stars represent superclass means. ANCOR's 's embeddings resemble those of Coarse+ (Appendix D.5) as it distinguishes sub- classes from an angular perspective, which induces intra-class variation that is suboptimal. SCGM-G explicitly detects the boundaries of both super- and subclasses, properly positions Gaussian means, and associates subclasses with their corresponding superclasses, whereby establishing a hierarchical structure. We evaluated SCGM-G's detected subclasses using the subclass labels in the val set. It got an NMI of 71 .58% and purity accuracy of 74.79%, much higher than its unsupervised counterpart SeLa (NMI 36.50% purity 38. 14%). This validates SCGM's effectiveness in leveraging superclass guidance for subclass inference. Several other model's 's visualization can be found in Appendix D.5.
Performance analysis. We have evaluated SCGM in terms of the effects of more shots, the impacts of the number of latent variables r, the impacts of the variance o², and the convergence, which are in Appendix D.1, D.2, D.3, D.4.
4.3 CASE STUDY: PERSONALIZED PREDICTION ON MEDICAL RECORDS
Next, we evaluate SCGM on a real Dialysis- dataset collected by several hospitals in Japan. This dataset consists of one-year medical records of 673 hemodialysis patients. contains 40 tempo- ral features (e.g., blood flow, venous pressure, etc.) that were monitored during each hemodialysis session (thrice/week). Each session was annotated indicate whether certain subtypes of events have occurred. A subtype corresponds to certain unstable behaviors of blood pressure during the dialysis, as categorized by experts. As introduced Sec. 1, these events are risky and may develop complications, thus should be avoided. Formally, our task is as follows. Given X(- E dx a length-u segment including the i-th patient's records starting at time step t, where d = 40 is the dimensionality, the task outcome is label y {0, 1} indicating event occurs (y 1) or not (y 0), at time step t + w. When y = 1, a fine-grained label ğ E {1...., s) is to be predicted to mark the subtype of the event.
8

---

Published as a conference paper at ICLR 2022
(a)
Personalized prediction comparison
New patients
seggmentt
dot one potient's model
1.0
Historical medical records
Pretrained model
X
0.8
Training
0.4
Protraining stage
0.0
time
DOCN
GEORGE
Sela
SCGM
Support set model personalization
methods
Figure 3: (a) An illustration of the model personalization framework for fine-grained event predic- tion for hemodialysis patients. (b) The distributions of patient-wise prediction accuracy on the query set by different methods. Each dot represents the accuracy of one patient's personalized model.
Since annotating fine-grained labels is costly, training a model on them at a large scale is infeasible in practice. To predict event subtypes, we use a model personalization framework as illustrated in Fig. 3(a). At the pre-training stage, only binary labels are used. For every new patient, a short period of (a few) medical records are collected as the support set for the fine ime-grained labels w.rt. that specific patient. It is used to train personalized classifier based on the pre-trained model.
To evaluate how SCGM fits this paradigm, we randomly split the 673 patients into 50%/50% train/test set, and reserved 10% of the train set for validation. For each testing patient, the first 50% records (i.e,, half year) form the support set, and the remaining form the query set. It is likely some event types didn't 't appear in the support set will appear in the query set, which are out-of- distribution (OOD) cases, thus is out of our scope. Hence, we only selected patients without such cases, which constitute major part (>65%) of the test set. The dataset has two event subtypes, i.e.
8 The segments x() Combiner Network (DCCN), similar to (Che et al., 2016). DCCN has two channels (e.g., RNNs and MLP) for encoding temporal and static features, respectively. The outputs them are integrated by a classification head. Its architecture can be found in Appendix C.3. In this application, each patient has 28 static features such as demographic information and infrequent blood test results.
- were sampled with w = 5, stride 1. The encoder is a Dual-Channel
Table 6 presents the average accuracy of all testing patients over 10 random train/test splits for the applicable methods. As is con- sistent with Sec. 4.2, unsupervised method Selaa and three-step framework GEORGE cannot improve DCCN's performance due to (1) ignorance of superclass supervision, and (2) separate pseudo- labeling step, respectively. In contrast, SCGM outperforms the original DCCN significantly. Fig. 3(b) compares different models by their distributions of personalized prediction accuracy of differ- ent testing patients in random train/test split. Each dot represents the mean accuracy of one patient's personalized predictions in the query set (220 dots in total). From the figure, SCGM has more dots close to the top (e.g., above 0.8) than other models, meaning more patients received fairly accurate predictions on the fine- ane-graieed labels. Although current performance could be limited by the data size, the results demonstrate the flexibility of SCGM, and its potential in tackling the CGFS problem in real practice.
Method DCCN GEORGE SeLa SCGM
Accuracy (%)
45.59-10.99
43.63:10.63
42.32:10.80 80
49.89110.90
Table 6: The comparison results on Dialysis- Event dataset.
CONCLUSION
In this paper, we proposed a new method, superclass-conditional Gaussian mixture model (SCGM), that underlies a general framework for tackling CGFS across domains. SCGM imitates the genera- tive process of samples from hierarchies of classes, and explicitly models unobserved subclasses by latent variables. It dynamically computes a Gaussian mixture for every sample given its superclass, with small overhead to encoder. We learned the model parameters by maximum likelihood framework, and also theoretically linked SCGM
timation in a principled aopecctaiio-Maxxiizzzn with contrastive learning. The extensive experiments on various datasets demonstrated the effective- ness, efficiency, and flexibility of our proposed method.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
In the case study in Sec. 4.3, we evaluated our SCGM on datasets related to human subjects that are hemodialysis patients. We clarify that got the permission perform the experiments on the dataset as described in Sec. 4.3, There is no ethics issue nor harmful insights in our current experiments. The dataset was properly used under the guidance of the data providers and domain experts, Sensitive information about the patients was well protected. The final description about the experiments was checked and approved by the data providers and relevant authorities.
REPRODUCIBILITY STATEMENT
The code of SCGM available at https: //ggt///iiitt togg//hh.cmmm/ii Ingchho/SCGt for repro- ducibility study. The benchmark datasets are large and cannot be upoloaded. The repository in- cludes an instruction on how to obtain the datasets and process them. Relevant instructions were provided how to the code train SCGM-G and SCGM-A, and evaluate their performance for CGFS task in 5-way and all-way cases (Table 1, 2, 5), and in the intra-class case (Table 3).
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. ICLR, 2020.
Thomas L Athey, Benjamin D Pedigo, Tingshan Liu, and Joshua Vogelstein. Autogmm: Auto- matic and hierarchical gaussian mixture modeling in python. arXiv preprint arXiv: arXiv:1909.000 1909.0268, 2019.
M. Christopher Bishop. Pattern recognition and machine learning. Springer, 2006.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Guy Bukchin, Eli Schwartz, Kate Saenko, Ori Shahar, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. Fine-grained angular contrastive learning with coarse labels. In CVPR, pp. 8730- 8740, 2021.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering unsu- pervised learning of visual features. In ECCV, pp. 132-149, 2018
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Zhengping Che, Sanjay Purushotham, Robinder Khemani, and Yan Liu. Interpretable deep models for icu outcome prediction. AMIA annual symposium proceedings, volume 2016, pp. 371.
American Medical Informatics Association, 2016.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0429) 2020b.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurlPS, pp- 2292-2300, 2013.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. /CLR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In ICMI., pp. 1126-1133 113.. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2022
Jacob Goldberger and Sam T Roweis. Hierarchical clustering of mixture model. In NeurlPS, pp- 505-512, 2005.
Will Grathwohl, Kuan-Chieh Wang, fôrn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint (eXiv:/9/2.03200 2019.
Yanming Guo, Yu Liu, Erwin M Bakker, Yuanhao Guo, and Michael Lew. Cnn-rmn: a large-scale hierarchical image classification framework. Multimedia tools and applications, 77(8):10251 10271, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Cheng-Yu Hsieh, Miao Xu, Gang Niu, Hsuan-Tien Lin, and Masashi Sugiyama. A pseudo-label method for coarse-to-fine multi-label learning with limited supervision. In ICLR, 2019.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. Interna- tional Conference on Learning Representations, 2018.
Daijo Inaguma, Daichi Morii, Daijiro Kabata, Hiroyuki Yoshida, Akihito Tanaka, Eri Koshi-Ito, Kazuo Takahashi, Hiroki Hayashi, Shigehisa Koide, Naotake Tsuboi, et al. Prediction model for cardiovascular events or all-cause mortality in incident dialysis patients. PloS one, 14(8):
e0221352, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. NeurlPS, 25:1097-1105, 2012.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper-
vised representations. In International Conference on Learning Representations, 2020.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Prototype propa- gation networks (ppn) for 3015-3022, 2019.
few-shot learning on category graph. In IJCAI, pp.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Mi Luo, Fei Chen, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Jiashi Feng, and Zhenguo Metaselector: Meta-learning for recommendation with user-level adaptive model selection. In The Web Conference, pp. 2507-2513, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 1), 2008
Laura Manduchi, Kieran Chin ShinCCheong, Holger Michel, Sven Wellmann, and Julia E Vogt. Deep conditional gaussian mixture model for constrained clustering. arXiv preprint arXiv:1106.06385 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint wrXiv:1803.02999 2018.
Luke Oakden- Dakden-Rayner, Jared Dunnmon, Gustavo Carneir,, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In CHIL, pp. 151- 159, 2020.
11

---

Published as a conference paper at ICLR 2022
Lukasz P Olech and Mariusz Paradowski. Hierarchical gaussian mixture model with objects at- tached to terminal and non-terminal dendrogram nodes. In Proceedings of the 9th International Conference Computer Recognition Systems CORES 2015, pp. 191-201 Springer, 2016.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini 1rXXiv:807.0331 2018.
Boris N Oreshkin, Negar Rostamzadeh, Pedro O Pinheiro, and Christopher Pal. Clarel: Classifica- tion via retrieval loss for zero-shot learning. CVPR Workshops, pp. 916-917, 2020.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In ICLR, 2018.
Marko Ristin, Juergen Gall, Matthieu Guillaumin, and Luc Van Gool. From categories to subcate- gories: large-scale image classification with partial class label refinement. In CVPR, pp. 231-239, 2015.
Joshua Robinson, Stefanie Jegelka, and Suvrit Sra. Strength from weakness: Fast learning using weak supervision. ICMI,, pp. 8127-8136. PMLR, 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04885 04855, 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, pp. 4077-4087, 2017.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In NeurIPS, 2020.
Fariborz Taherkhani, Hadi Kazemi, Ali Dabouci, Jeremy Dawson, and Nasser M Nasrabadi. A weakly supervised fine label classifier enhanced by coarse supervision. In ICCV, pp. 6469-6468, 2019.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: good embedding all you need? In ECCV, pp. 266-282. Springer, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurlPS, pp. 3630-3638, 2016.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting neirest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.0462. 2019.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In CVPR, pp. 5542-5551, 2018.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O Pinheiro. Adaptive cross-modal few-shot learning. In NeurIPS, pp. 4847-4857, 2019.
Jinhai Yang, Hua Yang, and Lin Chen. Towards cross-granularity few-shot learning: coarse-to-fine pseudo-labeling with visual-semantic meta-embedding. MM, 2021.
Zhi- Hua Zhou. A brief introduction to weakly supervised learning. National science review, 5(1): 44-53, 2018.
12

---

Published as a conference paper at ICLR 2022
A
APPENDIX FOR DETAILS OF DERIVING SCGM
A.1 DERIVATION OF PARAMETERIZED PROBABILITIES
In this section, we provide the details of the probabilities in Eq. (4). First, using Bayesian rules and marginal property, derive
p(y: p(zilu))p((s)) p(z;)
p(za pp(p) - (E(
p(z:lv.) p(v.lz)m p(v:)
plva 2
(9)
Eil= P(vilz))
Since M2, and Byy are surrogate representations of 2; and y: in the embedding space, substituting p(z)lys) and p(valzz) with the equality in Eq. (3), Eq. (4) can be achieved.
A.2 THE LOWER-BOUNT OF THE LIKELIHOOD FUNCTION
In this section, we provide the details of the bower-bound in Eq. (5). By introducing the approximated posterior (= Vi. yi), the likelihood in Eq. (1) becomes
9 8c) (2.. vi.
E( erDn 0)
log
plu (2o
2
p(us (oo ( (1. vi)
log
{(z vi.
(10)
g(2s/vi, wi) logp(ui12;) + log logro(zilvi (s) log log
(q(ealvaan)) log po (v)))) log (z: log g(2s///, v:]]
where the third step uses Jensen's inequality. This completes the derivation of Eq. (5).
A.3 DETAILS ON TRANSPORTATION POLYTOPE
Let a be a probability vector in the simplex A, = {x E R = 1} and b be probability vector in the simplex An (x E B x 1), where 1, and 1, are the vectors of ones in dimension and n, respectively. The transportation polytope of a and b, iee, the polyhedral set of X matrices, is defined as (Cuturi, 2013)
2 2= 2=f2 Rf* = a 1,
(11)
which means Q a set of non-negative r x n matrices with row and column sums a and b, re- spectively. From probabilistic perspective, any matrix Q E Q represents a joint probability of two multinomial random variables X and Y that take values in (1....,r) r) and {I....,?} n) with marginal distributions and b, respectively.
The transportation polytope defines a space of transport matrix Q for mapping a to b. Given a cost matrix M E RT*", the cost of the mapping can be quantified Tr(Q "M), and the problem
min Tr(Q M) QEQ
(12)
called an optimal transport (OT) problem between a and b, given cost M. Since the traditional algorithms to solve the OT problem scale inefficiently to large datasets, (Cuturi, 2013) introduced fast Sinkhorn-Kan -noop algorithm, which amounts to solve regularized problem
1 H() min Tr M) - QEO
(13)
where ((Q) = -ij Qij log Qij is the entropy function, and À is trade-off parameter. If À is large, optimizing Eq. (13) approximates optimizing Eq. (12).
13

---

Published as a conference paper at ICLR 2022
In our case of Eq. (6), M - a 1, and b 1, In addition, similar to (Asano al,, 2020), Q= ((2a) Vi, yi ! can be considered joint probability of 2; and the i-th sample (note the condition (v:, y4) corresponds the incidence of the i-th sample) with uniform prior 1. For x
the details of the Sinkhorn Knopp algorithm, we refer the readers to (Cuturi, 2013).
It is noteworthy that our optimization problem in Eq. (6) the E-step is derived from our likelihood function in Eq. (5), which turns out to be form of the regularized OT problem in Eq. (13), rather than directly formulated using Eq. (13). The derivation of the optimization problem in Eq. (6) can be found in next section (Appendix A.4).
A.4 THE OPTIMIZATION PROBLEM IN E-STEP
In this section, we claborate the optimization problem for inferring the posterior (22,|Vi, y4) in Eq. (6). Substituting ((2g) V., 2/) with n nQ is and substituting (2 with P the third step in Eq. (10) can be rewritten by
1 (()) log
Ozeii log P.,. Q log Qz,, Q logn) 2 Tr(Q" Pog +
(14)
i=1 51=1
where Tr(.)) is the trace function, H(-) is the entropy function, the last constant term is because Ei Ex =1 Qz 1, and the inequality stems from À 2 1, which is introduced to control the strength of the regularization (Cuturi, 2013). This is the objective function in Eq. (6), where the constant log n is omitted because it has no effects on the optimization.
The right hand side of Eq. (14) is valid lower bound of the likelihood function in Eq. (5) according to Eq. (10). Therefore, optimizing it (i.e., the objective function in Eq. (6)) provides a reasonably approximated solution the likelihood function in Eq. (5). In addition, the transport polytope Q in Eq. (6) is introduced as an equal partition constraint on subclasses (Asano et al., 2020), so that the optimization problem is feasible. is noteworthy that although subclasses have the equal partition constraint, superclasses can have different sizes since they can have different numbers of subclasses.
A.5 THE OPTIMIZATION PROBLEM IN M-STEP
In this section, we derive the optimization problem for learning model parameters in Eq. (7). Intro- ducing the probabilities specification in Eq. (4) to the negative of the lower-bounded likelihood in Eq. (5), and removing terms that are irrelevant to model parameters (0, Φ). ) we have
px (Hz Pv. ((())())
pn (vilms, Ex )m
la.0
a(2a) V1.8)) log ES PN (M., Bug Ev; (((()
+ log Σ Σz PN (v. M4g E
(15)
Considering isotropic Gaussian with Ey, IT2 we have
(f1
Fe )² = exp (M) Fy, ) exp (1L P /0²)
(N Ey,,) exp
(16)
2
exp (1/72)
where the second equivalence is because we apply L2 norm to Us. and By respectively. Similarly, considering Σr. I and applying T2 norm to v,, we have
pn (v E..) = exp (vv Me 1l,,)² = exp (v. Me,
1)
exp (v. /0, exp (1/o2)
(17)
20²
g²
Substituting the corresponding terms in Eq. (15) with Eq. (16) and Eq. (17), and removing the constants exp (1/6²) and exp (1/a2)in the fractions, we obtain the loss function in Eq. (7). In Eq. (7), the prior p(v)) can be set as uniform, i.e., p(v.) = 1. If using a prior from Dirichlet distribution, p(y: can be instantiated by estimating the parameters a (of the Dirichlet distribution) by the ratio of different superclass labels in Durain the training dataset. Also, nं in the second term of Eq. (7) can be calculated using Eq. (2).
A.6 THE TRAINING ALGORITHM OF SCGM
The training algorithm of SCGM is summarized in Algorithm 1.
14

---

Published as a conference paper at ICLR 2022
Algorithm 1: Superclass-condiiionn Gaussian mixture model (SCGM)
Input: encoder fe. training dataset Diin (number classes is c), hyperparameters r, X, 7 Output: model parameters (0. Φ)
Initialization(c,,
// initiaiiee Gaussian means Φ = =111,) (1,)))=)((,,)))111
for i <- 1 to MaxEpach do /= E-stop */
for (x, in Dataloader(PPnnll do
/+ Load a minibatch (x, y)
v= [x)
// get embeddings
Compute for Eq. (6) based on Eq. (4) using v and øb end
Concatenate all batch-wise probabilities p to get for Drnin
Q = SinkhornKnopp(PP X) // Inferring posterior using Sinkhorn-Knopp
a algorithm for optimizing (c)
/- M-step */
for (x, y, q) in Dataloader(Dmma)
/+ Load a minibatch (x, v. q) where q is obtained from E-step +/
10
v v= fo(x)
// forward pass through the encoder Calculate ". // calculate loss in (8) using ls.0 in Bq.(7)
12
Φ SGD(E,
// update model parameters
13
end
end
B APPENDIX FOR FURTHER DISCUSSION
B.I DISCUSSION ON THE BQUAL PARTITION CONSTRAINT
The equal partition constraint on subclasses in Eq. (6) theoretically assumes subclasses are of equal or similar sizes, through using Q1, 1,, (i.c., a = 1, in Eq. (11)) as a prior distribution of the subclass sizes (superclasses can have different sizes because they can have different numbers of subclasses). It is possible to replace 1, by other prior distributions to reflect uneven distribution of the subclasses sizes. In this work, we used uniform prior because we don't assume such prior knowledge (since the subclasses are unobserved during training). This prior is inspired by (Asano et al., 2020). If we don't have such a prior, the searching space subclasses will be exponential and the optimization is intractable. To make the optimization feasible, included this constraint to help reduce the searching space. In this way, we found the solution is efficient and effective to our CGFS problem, which does not necessitate an exact identification of every subclass. Compared to other baseline methods, the performance improvements indicate that reasonable boundaries between subclasses can be learned by SCGM with this constraint for model adaptation, though the boundaries may not be ideal. However, it is worth to note that this uniform prior may not always be reasonable in different applications. For example, if identifying rare subclass important in an application (Sohoni et al., 2020), more reasonable prior may be considered. In our experiments, the evaluations on the benchmark datasets (which were used for consistent setup with the existing works (Bukchin et al., 2021)) )) do not reflect the case of substantially skew distributions of subclass sizes. We leave the extension of the proposed method to this scenario in our future work.
B.2 DISCUSSION ON THE RVALUATION CASES IN SECTION 4.2
In Section 4.2, we evaluated two fundamental cases of (1) unseen subclasses of seen superclasses; and (2) unseen subclasses of unseen superclasses. Another case that considers the mixture of the subclasses of seen and unseen superclasses worth discussion because is a practical case. As discussed in (Xian et al., 2018; Oreshkin et al., 2020), mixture of samples from seen and un- seen classes during testing may result in an imbalance prediction performance between seen and unseen classes, and the inductive bias may lead to misclassification of unseen classes towards the seen classes. It is noteworthy that (Xian et al., 2018; Oreshkin et al., 2020) focus on a generalized zero-shot learning problem, which is not for rross-grauulartty adaptation and don't 't consider class hierarchy. Thus their task is remarkably different from our CGFS task. Despite the difference, we inspired be aware of the challenge that there could be similar imbalance in the prediction of
15

---

Published as a conference paper at ICLR 2022
the subclasses of seen and unseen superclasses if they mixed. This could be open challenge in the CGFS task which is rarely discussed in the existing works. To alleviate this problem, some specific technical extensions may be designed, as discussed in (Xian et al., 2018; Oreshkin et al., 2020), which is out our scope. We leave this scenario our future work.
C APPENDIX FOR IMPLEMENTATION DETAILS
C.1 THE SETUP OF THE COMPARED MODELS
following (Chen al., 2020b), we its queue size 65536, momentum 0.999 (0.99 for CIFAR-100), temperature 0.2, embedder & as MLP. For SwAV- ImageNet, following (Caron et al., 2020), its number of prototypes was 3000, embedder & as MLP. For ANCOR, following (Bukchin et al., 2021), we set its queue size 65536, momentum 0.999 (0.99 for CIFAR-100), temperature 0.2, embedder & as fc and MLP, with the multi-class queue, and an- gular norm on. For GEORGE (Sohoni et al., 2020), we set its robust learning rate le & as fc, (dim 16, 10 neighbors, 0 distance) for its dimension reduction step, the number of clusters per su- perclasses was grid-searched within {5, 10, 15, 20). For SeLa, following (Asano et al., 2020), set its X = 25, & as fc, and number clusters was grid-searched from 100 to 500 with step size 50. For Coarse+ and Fine+,
For MoCo-v2 and MoCo-v2-Imagee
set & as MLP. For our SCGM, set ? 0.5, o² 0.1, and À 25 (A follows (Asano et al., 2020)); as fc, the number of latent variables r was grid-searched from 100 500 with step size 50.
Remarks on embedder. For the compared methods, adding an embedder follows the practice of the existing works such as MoCo-v2 (Chen al., 2020b) and SimCLR (Chen al., 2020a), which was also used in the different compared methods in (Bukchin et al., 2021). In these works, adding certain types of embedder (e.g., MLP) is empirically found helpful for improving performance. We have the same observation in our experiments, and think the effectiveness could come from (1) more capacity; and (2) better generalization in some cases. The latter because after training, the embedder may be specific to the pre-training tasks, which enables more generalizable representations to be learned by the backbone networks. Thus dropping the embedder at testing stage may help alleviate overfitting to some extent. For comprehensiveness, we evaluated both of the cases when an embedder is added or not in our experiments, as described in Section 4.1.
C.2 THE ARCHITECTURE OF SCGM-A
E-step: infer q(z, (v)
{
P" SCOM
M-step: backgropagation
x
Backbone
Embedder
[
=
& C
Momentum updates
updates
SCGM classification head
k
Momentum
Mamentum embedder
IIfONCE
Backbone
queues
different
k
Figure 4: An illustration the architecture of SCGM-A
Fig. 4 illustrates the architecture of SCGM-A, which was implemented generic encoder (e.g., ResNet-50) with momentum encoders. The query v and positive key k4 are computed from two random augmentations of the input images through the encoder and embedder, and their momentum updates. There is a superclass wise dictionary maintains c queues for c different superclasses so that superclass wise negative keys k_ can be generated for within- within-superccae superclass contrastive learning, which facilitates preserving intra-class variation. At the SCGM classification head, v. k, and k_ are ap- plied with angular normalization, followed by the InfoNCE loss, where the angular normalization
16

---

Published as a conference paper at ICLR 2022
is defined by (Bukchin et al., 2021)
Vi
Ply
Bly P.y.
(18)
|
v.
which enables contrasts from an angular perspective, and induces better synergy with cross-entropy loss.
The top part of the SCGM classification head are similar to its counterpart using a generic encoder, where interacts with superclass means { 1 and subclass means {H 1 for computing SCGM loss lo and a cross-entropy loss leE as in Eq. (8). Thus the loss for training SCGM-A is
((Dmiin 0, +) (Demiiiii 0, ( 1==1) LIdNNcE (Peninax la; 0) + 7 (Dmini 0, 0,4)
(19)
where the model parameters are learned by our proposed EM-based algorithm that alternates be- tween the SGD backpropagation (M-step) for computing model parameters {0, Φ} and the posterior inference (B-step) for computing subclass distributions.
C.3 THE ARCHITECTURE OF DCCN
Combination layer
ho
LSTM
LSTM
Static input
Temporal input
Figure 5: An illustration the architecture of DCCN
Fig. 5 illustrates the architecture of the Dual-Channel Combiner Network (DCCN). DCCN is effec- tive to process heterogeneous medical records data that usually consists of static profiles and time series (i.e., records) (Che et al., 2016). It has static channel realized by MLPs to encode static features (e.g., demographic information, infrequent blood test results, efc.) and temporal channel realized by RNNs (e.g., LSTM) to encode temporal feature (e.g., blood flow, venous pressure, etc.). The hidden representations output by the two channels are concatenated (or using certain pooling method) and further projected to a compact embedding that is used for prediction by the classifi- cation head, i.e., the combination layer in Fig. 5, which is realized by MLPs. Attention layer may be added before the combination layer for weighted combination of the hidden representations from different channels and different time steps.
D APPENDIX FOR MORE EXPERIMENTAL RESULTS
D.1 EFFECTS OF MORE SHOTS
Method
I-shot
S-shot
10-shot
20-sbot
Courset
37,44.1.112
50.51.10.31
$7.35
63.23:10.25
45.14:1..12
59.60-1-0.35
04
68.8744:026
ANCOR-fe
16.19:0..16
60.64-10.30
64.31=0.27
66.85
SCGM-G
48.74:10.15
60.39:1:0.29
63.42=0.31
65.67440.25
SOGM-A
49.31:10.16
63.82-10.32
67.43-00330
69.99-1-..25
Table 7: Effects of more shots (all-way)
Table 7 and summarize the performance of SCGM-G and SCGM-A, with the strongest baseline ANCOR and ANCOR-fc, together with the coarse baseline Coarse+, Living dataset. From
17

---

Published as a conference paper at ICLR 2022
Method
F-shot
S-shot
10-shot
20-4hor
Coarse+
79.29:00.6
92.99
94,33=0,36
95.
ANCOR
89.23=0.55
94,44-10.36
95.33=0.31
95.90:10.227
ANCOR.fe
90.41:0.557
93.84.100.66
94,52-0.32
95.09:10..29
SOGM-G
89.72.10.54
94.38:10.37
0.309
95.52.4-0.31
SOGM-A
90.97=0.55
94,41=0.34
95. 95.10
95,43-t:0.28
Table 8: Effects of more shots (5-way)
Table 7, we observe SCGM-A continuously improves performance as more shots are added, consis- tently outperforms other models, which validate the effectiveness of SCGM framework with varying number of shots. SCGM-G's performance becomes comparable to ANCOR(-fe) as more shots are added, due its simpler architecture, which however is more computationally efficient. Thus pro- vides a good balance between performance and efficiency. From Table 8, we observe ANCOR and SCGM perform similarly. This is because the task close the regular superclass classification, and its performance limit can be easily reached by adding a few more shots. This can be seen from Coarse+'s results, which become close to other models when there are 20 shots in the support set.
D.2 IMPACTS OF THE NUMBER OF SUBCLASSES
Seting
P=
1=100
1=0
2=200
P=250
7=300
r=400
7=500
5-way 89.45500000
54
2655
88.05
00556
86.5220.56
55
All-way
42.98=0.144
48.74+0.155
46.21=0.15
47.81+0.115
0..5
16.91=
45,23=0.15
44.63+00.15
Table 9: Impacts of the number of latent variables the performance of SCGM-G
Table presents the results of SCGM-G on Living 17 by varying the number of latent variables r from 50 to 500. As can be seen, the performance of SCGM-G is generally stable and it consistently outperforms Coarse(+) (Table 1) w.rt. different r. By looking into the details, we observe too small (e.g., 50) or too big (e.g., 500) may degenerate the performance in the all-way case, which may be caused by slight underfitting and overfitting, respectively. In the 5-way case, small r does not impact performance obviously, this because the 5-way case close to evaluate coarse classification, which does not need exact inference of subclasses. Thus a small number of r is fine (which may not distinguish all possible subclasses).
D.3 IMPACTS OF THE VARIANCE
Setting
=2-005
a020.1
=0.55
20222
=00225
7²=0.
5-way
90.49:10.56
0.57
90.09:1..57
00.55
90.56:E0.59
90.69:10.9
way
46.47.10.15
18.74.1..15
47.48.10.15
16.03
42.21
38.38.10.13
Table 10: Impacts of the variance parameter o² on the performance of SCGM-G
Table presents the results SCGM-G on Living by varying the variance o² of subclasses, which resembles the temperature scaling in Eq. (7). As described in Sec. 3.1, in our experiments, we set σ² and tune o² to adjust the relativity between super- and subclasses, which we found empirically effective. From Table 10, we observe o² is better to be selected from 0.1 to 0.2, and a too small (e.g., 0.05) or too big (e.g. 0.3) value may degenerate the performance the all-way case. This observation is similar to the evaluation of the temperature scaling parameter in MoCo-v2 (Chen et al., 2020b). In the 5-way case, the performance is relatively stable w.r.t. different values of g², Similar the observation in Appendix D.2, this because the 5-way case is close evaluate coarse classification, which is less impacted by the learning of subclasses.
D.4 CONVERGENCE ANALYSIS
Fig. 6 presents the training loss E(Drain 0, Φ)) in Eq. (8) of SCGM-G, and ((Drain: 0, ②) in Eq. (19) of SCGM-A, during the training process the four BREEDS datasets. The training alternates between E-step and M-step, where E-step runs every 5 epochs. As can be seen, although there are some periodic fluctuations, the loss function values generally decrease and converge to small value across different datasets. The periodic fluctuations are not synchronized with the alternate E-step and M-step since its period is not 5 epoch can be observed. In fact, the fluctuations come from the
18

---

Published as a conference paper at ICLR 2022
(a) SCGM-G
(b) SCGM-A
37
delasel
Uving17
16
Noriiving2
Enety13
15
l0ss calue
75
100
125
200
75
1.00
125
150
200
opach
esoch
Figure 6: The training loss w.r.t. the number of epochs of SCGM on BREEDS datasets
cosine annealing with warm restarts schedule for the learning rate (as discussed in Sec. 4.1), which has 20 epochs per cycle. This is consistent with Fig. 6. In practice, observed both E-step and M-step can decrease the loss function values since both steps are theoretically optimizing the same loss function. There rounding step (as discussed Sec. 3.2) for generating discrete code from the posterior, which follows the E-step. We found this rounding step is beneficial, since without it, too small values may be generated in the posterior inference, which unstable in computation. Generally, the current setup provides a stable training framework and we don't 't need to interfere in the program during the training process on all of our experimental datasets.
D.5 VISUALIZATION EMBEDDING
(a) Coarse+
(b) MoCo-v2
(c) ANCOR-fc
(d) SCGM-A
Figure 7: The tSNE visualization of the embeddings learned by different methods. Star represents
superclass means. "4" marker represents subclass means.
19

---

Published as a conference paper at ICLR 2022
In addition the visualization results of ANCOR and SCGM-G provided in Fig. 2, Fig. 7 presents the tSNE visualization of the embeddings of several most relevant methods in comparison using Liv- ing dataset, including Coarset, MoCo-v2, ANCOR-fe and SCGM-A. In the figure, colors mark different superclasses, "+" markers are the learned subelass means by SCGM-G, and stars represent superclass means. From Fig. 7a))c), can observe the embeddings of both ANCOR and ANCOR- fo resemble Coarse+. Although ANCOR-fe slightly distinguish several small groups of embeddings in some superclasses, their intra-class variation is suboptimal. From Fig. 7(b), MoCo-v2 has the ability to detect superclasses to some extent, but cannot fill the gap of superclass supervision, by comparing with Coarse+, in this dataset. The reason may be the Living17 dataset is not sufficiently large and comprehensive for MoCo-v2 to learn embeddings that are comparable to supervised meth- ods. In contrast to them, SCGM-A (and SCGM-G Fig. 2) explicitly detects subclasses, with proper position of subclass means and superclass means. It also associates every subclass to its corresponding superclass, which establishes a hicrarchical class structure. This explains the effee- tiveness of SCGM in goss-granularity learning tasks.
20


Reviews:
[{"review_id": "wrEP3PlFvT", "reviewer": "Reviewer_s1rd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- paper is clearly written and well motivated\n- the approach is novel\n- empirical results are convincing and very solid\n- authors provide code and instructions to recreate datasets for reproducibility, datasets are public\n\nCons:\n- Please provide bullet point list of contributions. I advise to replace the last paragraph in the intro, which is too wordy and is not to the point\n- There have been some work studying the relationship between coarse and fine-grained level supervision in the zero-shot classification domain, for example https://arxiv.org/pdf/1906.11892.pdf. Please discuss how your work is different.\n- Additional experiments on well established fine-grain classification datasets such as CUB and Flowers would significantly strengthen the results of the study. I beleive those datasets are much more relevant to the task than CIFAR100, for example.\n- Testing only on seen / unseen superclasses is limited in that is overlooks the problems that arise in the practical setting when both seen and unseen classes are expected to be mixed. This has been pointed out in relation to generalized zero-shot learning https://arxiv.org/pdf/1712.00981.pdf. ", "summary_of_the_paper": "The paper addresses the problem of transferring knowledge from abundant data with low granularity labels to the fine grained data with few labels. The model uses gaussian mixture approach to link the structures in the low granularity data with the few fine grained labels.", "summary_of_the_review": "I recommend accept since there is clear novelty in the paper, problem is important and well motivated, empirical results are convincing", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "tbMzKlgx40U", "reviewer": "Reviewer_FxzL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper include the following:\n\n(1) The problem addressed is an important and understudied one.  While previous work has identified the CGFS problem (though not necessarily called it by that name), methods to address it in a technically principles way remain in their infancy.\n\n(2) The method proposed both naturally follows from the existing treatment of the problem in the literature and presents a novel approach to address it.  The generative modeling framework presented by the authors is well-motivated, described with a reasonably high degree of precision, and its relationship to other approaches (e.g. GEORGE, contrastive representation learning) is analyzed in a compelling manner.\n\n(3) Empirical results are generally compelling, and comparison between the proposed method and existing work (a) takes into account relatively current approaches to the problem and (b) evaluates performance along several important axes, including both classification performance and systems-level performance.\n\nWeaknesses of the paper include the following:\n\n(1) Though the presentation and analysis of the proposed method is reasonable, comparisons to other methods are at some points overstated in the opinion of this reviewer.  Several statements (mentioned in the “detailed comments” below) should be either refined, repositioned, or omitted to accurately reflect the contribution of this work.\n\n(2) Empirical performance gains with respect to ANCOR in particular are not always convincing.  There exist a number of cases where the two are very close, and it is not clear how evaluations of superiority/non-inferiority between different methods was performed.  The authors should make their procedure for computing confidence intervals and determining superiority/noninferiority clear, and should ideally provide appropriate statistical tests/analysis for these claims.\n\n(3) It would be desirable to have some analysis of theoretical guarantees about worst-case subclass performance.  The paper does not provide any such analysis in its current state.  It also appears (at least as far as this reviewer understands it) that the current formulation imposes a constraint that subclasses be of equal size.  If true, this should be discussed in substantially more detail, and would make the worst-case subclass performance analysis even more important.", "summary_of_the_paper": "The paper presents SCGM, a new technique for solving the Cross-Granularity Few-Shot learning (CGFS) problem.  CGFS is defined as the problem of adapting a classification model trained on coarse (“superclass”) labels to perform well on fine-grained labels, which consist of multiple “subclass” labels within each superclass.  The paper presents a new generative modeling approach that enables end-to-end classifier training in a manner that (a) incorporates information about superclass-subclass hierarchies (b) does not rely on explicit subclass enumeration (c) provides improved empirical performance on CGFS and (d) provides systems benefits with respect to existing baselines.  The authors provide detailed description of their approach and characterize its relationship with existing methods, and present empirical results that support their methodological choices.", "summary_of_the_review": "Abstract:\n\n”It is crucial to fields where fine-grained labeling (e.g., breeds) requires strong domain expertise thus is prohibitive, such as medicine, but predicting them is desirable.” --> I found this sentence confusing, consider rephrasing.\n\nIntroduction:\n\n“During the treatments…task” --> I understand the general point here, but the example seems a bit strange.  For instance, if I could predict an event before performing dialysis, would I not consider delaying the procedure?  If I delay the procedure, the events never happen.  If it's useful to be able to say \"we predict event type A vs. event type B\" to help physicians assess the risk of performing the procedure, that would make sense...but that doesn't seem to be how it's presented here.  Consider reframing this to make it reflect the situation you're trying to describe more precisely (or consider providing a slightly different example).  This is a minor issue, but the framing could throw some readers off.\n\n“The data constitute a support set” --> In the sense that they provide support for the different fine-grained annotation schema?  Or for training/fine-tuning?  Or for both?  Additional precision would help here.\n\n“Unseen siblings of the superclasses” --> What is meant by this phrase?\n\n“Subclasses may arbitrarily spread within each superclass” --> Maybe point out \"and in real datasets, it is often the case that they are unevenly represented in a way that causes subclass performance gaps.\"\n\n“However, their approach is confined to images” --> Why is this true?  Certainly the referenced work focuses on images, but it is not obvious to me why it could not be directly extended to the type of data in medical records.  I am also not sure I agree that there are not reasonable/standard approaches for obtaining positive/negative pairs on other types of data.  Even if this were true, however, I would argue that it's not a downfall of Bukchin et al.  Recommend rephrasing or cutting this piece.\n\n“Also, since contrastive learning does not model subclasses…Sec .4).” --> At this point in the paper, it's not clear at this point how you support these statements in Sec. 4, and the claims may seem unwarranted.  Recommend either deferring this discussion or providing some quantitative previews of your results here to mitigate this.\n\n“It explicitly represents the unobserved subclasses as latent variables…” --> It may be worth clarifying here that you do not assume knowledge of the subclass identities.\n\nRelated Work:\n“…cannot trivially be extended to non-image data…” --> See comment above, I don’t think this is a great argument.  Recommend cutting or rephrasing.\n\n“…cannot leverage coarse training to guide their pseudo-labeling…” --> This claim is unclear. Coarse training explicitly does guide the pseudo-labeling in e.g. GEORGE (Sohoni et al. 2020), as clusters w/in the coarsely trained representation define the pseudolabels.  The argument about end-to-end optimization of the pseudo-labels/model parameters seems reasonable, but not sure this statement is accurate.  Maybe rephrase to indicate that a two-step approach could lead to suboptimal psuedolabeling rather than arguing that the coarse representation doesn’t contribute at all in these two-step methods.\n\n“These methods are unsupervised, and cannot be used…” --> Why does the unsupervised nature of these approaches mean they are not capable of application so this problem?  I found this part unclear…I suspect it may be a writing issue rather than a technical issue, but clarification would be appreciated.\n\nSuperclass-conditional Gaussian Mixture Model:\n\n“unseen superclass” --> What does an unseen superclass mean?  Is it an unseen subgroup that spans existing superclasses?  Or an OOD example?  Need a more precise description of what this means, recommend providing mathematical definition.\n\n“Unlike some previous works (Sohoni et al., 2020), which allocated a predefined number of subclasses to every superclass…” --> This isn't true, as far as I understand it.  Sohoni et al. perform a search over the number of subclasses w/in each superclass and optimize an unsupervised cluster quality metric.  One could argue that fewer searches are required in the method this paper proposes – because all of the searching over number of subclasses is baked into the r parameter – but I don’t think that argument is particularly compelling from a practical standpoint.  I would suggest cutting or softening this claim. \n\nFigure 1 --> Appears to be a typo in the caption, all of the subclass means have the same subscript (“1”)\n\n“…and tune \\sigma to adjust the relativity between super- and sub-classes” --> What range does this usually fall in?  Worth reporting this.\n\n“transport polytope” --> Many readers may not be familiar with this.  Recommend providing a brief description/background on this in the Appendix (and reference this in the text)\n\n“It is noteworthy…” --> This is an interesting set of observations.  Nit: typo in “contrasitive”\nExperiments:\n\n“visual granularity” --> What exactly does this mean?  A comment would be helpful.\n\n“20/6/8 splits” --> So the superclasses are disjoint in train/test/val?  That seems a little confusing.  Suggest clarifying\n\n“which adds an embedder” --> What exactly does this embedder do/why is it added? More representational capacity?  More detail would be welcome.\n\n“dim” --> “dimension”\n\nTable 1: Was there a statistical test done here to support improvements over ANCOR?  Seems to be overlapping confidence intervals in a couple of cases.  Please provide detail on how confidence intervals were computed and what statistical approach was used to determine superiority/non-inferiority.  Also, what's similar about the cases where the method outperforms and doesn't outperform ANCOR?  Is there any additional explanation that can be given here?\n\n“baselines on contrastive learning cannot be applied…\" --> see above comments, I don’t think this is a particularly strong argument.  I would cut it, or at least acknowledge that this is not a fundamental limitation of the contrastive approaches.  Augmentation schemes certainly exist for non-image data types.  The strongest response would run these baselines using data augmentation approaches for these other modalities.\n\nAppendix A:\n\nEq. (10) --> I am not following exactly how Jensen’s inequality results in the move from line 1 to line 2 here.  I could be missing something, but it would certainly help if the authors provided more explicit detail in the intermediate steps (e.g. laying out the terms in the inequality clearly – i.e. a_i and x_i here https://en.wikipedia.org/wiki/Jensen%27s_inequality).\n\n“the second step in Eq. (11)…” --> I think this is a typo…should be Eq. (10) referenced here?\n\n“\\lambda \\ge 1” --> Am I correct in understanding that there is strict equality for \\lambda = 1?\n\n“equal partition constraint” --> If I understand correctly, this statement deserves more discussion.  Does this imply that the constrained transport polytope formulation here effectively assumes that subclasses are of equal size?  This would appear to be consistent with Eq. (6), and if this is indeed true this would represent a substantial limitation of the method that should be further discussed.  On this note, what is the distribution of subclass sizes in the experiments run in this paper?  Are the subclasses all of similar size, or does it vary?\n\nEq. (13) --> the last equality/definition here seems a bit strange.  You remove the 1 constant, and define the term without the 1 as equal to the term without the 1?  Why is this true/necessary?\n\nAppendix B:\n\n“…robust learning rate…” --> Could be worth searching over robust learning rate for this baseline.\n\nAppendix C:\n\nTable 8 --> Suggest adding bolding to this table as well to maintain consistency with other tables.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IwtXhS8E2GR", "reviewer": "Reviewer_w8nz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-motivated and addresses the important problem of few-shot multi-granularity adaptation.\n2. The paper includes comprehensive evaluation across various datasets and demonstrate the effectiveness and stability of the proposed approach.\n3. Visualization confirms the proposed approach learns the superclass-subclass structure.\n\nWeaknesses:\nThe main focus of the proposed approach is learning embeddings, but the evaluations are focused on classification. If the goal is classification, why not directly build a classification model? If the goal is learning embeddings, it would be better to put more focus on embedding based evaluation and clarify the relationship between embedding and classification performance.", "summary_of_the_paper": "The paper introduces a setup with the goal to adapt from a coarse pretrained model to unseen fine-grained labels. This problem is formulated as a superclass-subclass latent model and learned with maximum likelihood via expectation-maximization. The proposed approach, super-class conditional Gaussian mixture (SCGM) model, defines a hierarchical Gaussian distribution on the class hierarchy that models both the superclasses and subclasses. SCGM is evaluated on 6 image-net related datasets and a a real Dialysis-Event dataset collected by hospitals, and demonstrates competitive results under two evaluation setups: generalizing to seen and unseen superclass.", "summary_of_the_review": "The paper introduces the interesting problem of few-shot multi-granularity adaptation through hierarchical Gaussian mixture and demonstrates its effectiveness across multiple datasets. The relation between the goal (embedding learning) and evaluation (classification accuracy) is not very clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents an approach to learning good representations for few-shot learning when supervision is provided at the super-class level and is otherwise missing at the sub-class level.

After some discussion with the authors, all reviewers are supportive of this work being accepted. Two reviewers were even supportive of this work being presented at least as a spotlight.

The approach presented is well motivated, experiments demonstrate its value and include a nice application in the medical domain, making the work stand out relatively to most work in few-shot classification. Therefore, I'm happy to recommend this work be accepted and receive a spotlight presentation.
Decision: Accept (Spotlight)