SUPERCLASS-CONDITIONAL GAUSSIAN MIXTURE MODEL FOR LEARNING FINE-GRAINED EMBEDDINGS
Year: 2022

Abstract:
Learning fine-grained embeddings is essential for extending the generalizability of models pre-trained on “coarse” labels (e.g., animals). It is crucial to fields for which fine-grained labeling (e.g., breeds of animals) is expensive, but fine-grained prediction is desirable, such as medicine. The dilemma necessitates adaptation of a “coarsely” pre-trained model to new tasks with a few “finer-grained” training labels. However, coarsely supervised pre-training tends to suppress intra-class variation, which is vital for cross-granularity adaptation. In this paper, we develop a training framework underlain by a novel superclass-conditional Gaussian mixture model (SCGM). SCGM imitates the generative process of samples from hierarchies of classes through latent variable modeling of the fine-grained subclasses. The framework is agnostic to the encoders and only adds a few distribution related parameters, thus is efficient, and flexible to different domains. The model parameters are learned end-to-end by maximum-likelihood estimation via a principled Expectation-Maximization algorithm. Extensive experiments on benchmark datasets and a real-life medical dataset indicate the effectiveness of our method.

heading: 1 INTRODUCTION
text: Training deep models with sufficient generalizability is of fundamental importance, which demands immense training data with fine-grained annotations (Krizhevsky et al., 2012; Brown et al., 2020). In many fields, however, data labeling requires domain-specific knowledge, such as medicine (Sohoni et al., 2020), thus is prohibitive, and infeasible to be exhaustive. In this case, data for model training may only be “coarsely” labeled, while later the model is tested on a finer-grained classification task (Bukchin et al., 2021). For example, consider an event prediction task for dialysis patients (Inaguma et al., 2019). Hemodialysis is a major renal replacement therapy for patients with end-stage renal failure. These patients have to take hemodialysis thrice a week, each lasts for 4-5 hours. During the treatment, unexpected events, such as muscle cramp, perspiration, and dizziness, may happen as a result of lowering blood pressure, which need intensive medical care, thus should always be avoided. It is therefore an important medical issue to predict such events before an initiation of hemodialysis. In this task, binary labels, which mark the incidence of an event, can be collected. In contrast, finergrained labels that annotate different subtypes of events are seldom recorded. Since distinguishing different subtypes facilitates precise diagnoses, and helps physicians assess the risk for deciding whether to perform a hemodialysis (with certain precautions), it is desirable that a model trained with coarse (binary) labels can perform well on a finer-grained multi-class (subtypes) task.
To fill the gap of granularity between the training and testing scenarios, a practical way is to collect a few new records for a patient, with their fine-grained annotations. These data constitute a support set for fine-tuning a pre-trained model to the specific data distribution induced by the annotations of the target patient, for whom the adapted model is used for future predictions. Although massive finegrained annotation is impractical, annotating a few-shot set is feasible. In this work, we are interested in such a Cross-Granularity Few-Shot (CGFS) learning problem, where a model pre-trained on a set of coarse classes (denoted as superclasses), needs to adapt to an unseen set of fine-grained target classes (denoted as subclasses). The target subclasses could be descendants of the superclasses (as in
the aforementioned example), or could descend from other superclasses that are unobserved during pre-training. To be practical, the adaptation should only use a few samples from the subclasses.
The CGFS problem is not limited to the above application. It occurs in a model’s lifespan when an application requires separating some subclasses from the superclasses yet when the training dataset was created these subclasses were unannotated. For example, it could occur in detecting rare pathology or variants using medical images that were coarsely described (Oakden-Rayner et al., 2020), or personalizing a generic model trained on all historical users to a specific customer (Luo et al., 2020).
Despite its significance, CGFS cannot be trivially solved by regularly training models with coarse labels, because typical losses for supervised learning aim to maximize inter-class boundaries but neglect intra-class variation. Thus, subclasses may arbitrarily and unevenly spread in every superclass. Recently, Bukchin et al. (2021) proposed to integrate coarse supervision and contrastive learning (within superclasses) for solving CGFS. However, their approach cannot be readily used for medical records which typically contain static profiles and time series (Che et al., 2016). This is due to the absence of a standard augmentation method for generating constrastive pairs on some data other than images. Also, since contrastive learning does not model subclass explicitly, their solution could be suboptimal (as evaluated in Sec. 4). Moreover, as their model was built upon MoCo (He et al., 2020), and maintains many dictionaries for data sampling, its computational costs are high (Sec. 4).
In this work, we propose a novel Superclass-Conditional Gaussian Mixture model (SCGM) to learn fine-grained embeddings for the CGFS problem. Our contributions are summarized as follows.
• SCGM is agnostic to the encoder, thus is flexible to different applications. It models the generation of samples from hierarchical classes, and explicitly represents the unobserved subclasses by latent variables, without assuming their identities. It dynamically computes a Gaussian mixture for every sample conditioned on its superclass, and the model forms a hierarchy of Gaussian mixtures.
• SCGM only adds a small overhead to an encoder, for parameterizing its distributions, thus is efficient. The model parameters are learned end-to-end by maximum likelihood estimation via a principled Expectation-Maximization (EM) algorithm. We also theoretically linked our loss function to InfoNCE (Oord et al., 2018), explaining its effectiveness from a contrastive perspective.
• In the experiments, we evaluated SCGM on both benchmark image datasets and a real-life medical dataset. Since SCGM is compatible with contrastive learning, we also tested it with a momentum encoder (He et al., 2020). The results demonstrate SCGM on generic encoders has already outperformed the state-of-the-art (SOTA) baselines, with less computational costs, and it achieves boosted performance in some cases with momentum contrast integrated.

heading: 2 RELATED WORK
text: To our best knowledge, this is the first work to develop an SCGM model for underlying a framework that enables tackling CGFS across domains. The most relevant work (Bukchin et al., 2021) combines superclass-wise contrastive learning with coarse classification for preserving intra-class variation. Another work (Yang et al., 2021) used a three-step approach that pseudo-labels the embeddings (pretrained by coarse classification and batch-wise contrastive learning) by clustering every superclass. The pseudo-fine-labels were used to re-train the encoder. A similar three-step method (Sohoni et al., 2020) used a different loss for maximizing the worst-case expected accuracy over the pseudo-labeled subclasses. As discussed before, the former two methods require non-trivial searches for a suitable data augmentation method, which may be unavailable for some non-image data. By splitting training steps, the latter two methods could lead to suboptimal pseudo-labeling, and misleading labels could confuse the downstream steps. In contrast, our model is end-to-end. It explicitly infers the posterior of the subclass during coarse training. Moreover, it can achieve better performance without using as many computational resources as the model of (Bukchin et al., 2021) (as evaluated in Sec. 4).
Learning with coarse supervision. Several works deal with coarse/fine labels from the perspective of weakly supervised learning (Zhou, 2018) than to tackle the CGFS problem, including training methods that take advantage of a mixture of either balanced (Ristin et al., 2015; Guo et al., 2018; Taherkhani et al., 2019) or unbalanced (Hsieh et al., 2019; Liu et al., 2019; Robinson et al., 2020) coarse and fine labels. Among them, Liu et al. (2019) addressed a few-shot learning problem, but the model training assumes access to some fine labels and a graph of class hierarchy, which are unavailable in our problem. Thus it cannot be adapted to solve our problem.
Few-shot learning. Meta-learning has become a popular idea to handle the few-shot learning problem, which derives metric-based (Vinyals et al., 2016; Snell et al., 2017) and optimization-based methods (Finn et al., 2017; Nichol et al., 2018). The idea has been extended to semi-supervised (Ren et al., 2018), unsupervised (Hsu et al., 2018) and semantics-augmented (Xing et al., 2019) scenarios, when labels are scarce. However, none of them explores coarse labels for cross-granularity learning. Recently, many works observed that learning embeddings on all classes (without episodic training), followed by simple fine-tuning, is superior to SOTA meta-learning methods (Wang et al., 2019; Dhillon et al., 2020; Tian et al., 2020). In this work, similar to (Bukchin et al., 2021), we focus on this paradigm to learn useful embeddings, and do not use meta-learning for pre-training.
Embedding methods. The emerging self-supervised methods, such as contrastive methods (Oord et al., 2018; Chen et al., 2020a; He et al., 2020), are appealing in their ability to attain comparable embeddings to the supervised counterparts, and even surpass them when transferring to other tasks (He et al., 2020; Tian et al., 2020). Beyond instance-wise contrast, recent methods have explored instance-cluster contrast to boost performance (Caron et al., 2020; Li et al., 2020). In unsupervised methods, joint embedding and clustering has been found beneficial (Caron et al., 2018; Asano et al., 2020). Whereas, these methods never exploited coarse supervision. Thus, their embeddings/clusters do not necessarily reflect intra-class variation, which is important to the CGFS task.
Vanilla Gaussian mixture (GM) model is unfit for the CGFS scenario. A recent work (Manduchi et al., 2021) extended GM to constrained clustering by conditioning every sample with a prior clustering preference. It is neither supervised by coarse classes nor hierarchically structured. Conventional hierarchical GM is used for hierarchical clustering (Goldberger & Roweis, 2005; Olech & Paradowski, 2016; Athey et al., 2019) by applying GM agglomeratively or divisively. These unsupervised methods only infer clusters, but do not pre-train embedding models for task adaptation.

heading: 3 SUPERCLASS-CONDITIONAL GAUSSIAN MIXTURE MODEL
text: Firstly, a word about some notations. Let Dtrain = {(xi, yi)}ni=1 be n sample-label training pairs, where yi ∈ Ysuper = {1, ..., c} is a superclass label. Each xi is associated with a latent (unobserved) subclass label ŷi ∈ Ysub = {1, ..., s}. Ysub relates to Ysuper by a hierarchical structure, i.e., Ysub can be partitioned into c disjoint sets Ysub-1, ..., Ysub-c, such that if ŷi ∈ Ysub-j , then yi = j (1 ≤ j ≤ c). Let fθ be an encoder (i.e., backbone network) that is trained on Dtrain (without knowing Ysub). It maps xi to a d-dimensional feature fθ(xi). At test time, given a k-shot support set for a subset Ymsub ⊆ Ysub of m subclasses, i.e., Dsupport = {(xi, ŷi)|ŷi ∈ Ymsub}mki=1, the task is to train a classifier C: Rd → Ymsub with optimal accuracy on a test set of Ymsub subclasses. In our experiments, we also explored the case when the subclasses belong to superclasses that are not used for pre-training. As discussed in Sec. 2, our focus is to train fθ for good embeddings without modifying fθ during the adaptation–a paradigm with SOTA few-shot performance (Dhillon et al., 2020; Tian et al., 2020).
Formally, let fθ(xi) = vi, our goal is to find the model parameter θ that maximizes the likelihood of the posterior distribution pθ(yi|vi) on the observed data in Dtrain for classification tasks (Grathwohl et al., 2019). To model the unobserved subclasses, we associate every vi with a latent variable zi to indicate to which subclass vi belongs. Suppose there are r possible subclasses, the log-likelihood to maximize can be rewritten by marginalizing out the latent variables.
ℓ(Dtrain;θ) = 1
n n∑ i=1 log [ pθ(yi|vi) ] = 1 n n∑ i=1 log [ r∑ zi=1 p(yi|zi)pθ(zi|vi) ]
(1)
where the distribution pθ(zi|vi) specifies the subclass membership of vi, and p(yi|zi) associates zi with a subclass partition Ysub-yi . Unlike some previous works (Sohoni et al., 2020), which searched the number of subclasses for every superclass using a quality metric, we only assume a total number of r subclasses, and seek to infer their relationship with superclasses, i.e., p(yi|zi), without any prior on subclass partitions, so that the model is more generalizable.

heading: 3.1 PARAMETERIZATION OF PROBABILITIES
text: In Eq. (1), both p(yi|zi) and pθ(zi|vi) are not specified, and involve non-parametric variables. To make it solvable, we introduce our SCGM that models the generative process of the embeddings.
Let µj and Σj (1 ≤ j ≤ r) be the mean and variance of the j-th mixture component, and πj be its corresponding mixture probability. In a generic GM model, the generation of vi involves two steps: (1) draw a latent variable zi from a categorical distribution on all mixture components, and (2) draw vi from the Gaussian distribution N (µzi ,Σzi) (Bishop, 2006), as illustrated by Fig. 1(c). In step (1), the categorical distribution is usually defined on π = [π1, ..., πr] ∈ Rr, i.e., the static mixture probabilities, which cannot reflect different samples’ superclasses. In light of this, for each vi, we propose to define πi = [πi1, ..., π i r] to be conditional on yi, which dynamically changes w.r.t. different samples. To this end, we introduce µ̄yi and Σ̄yi as the mean and variance of superclass yi, and compute πij = softmax(−Dyi(µj)2/2), where Dyi(·) measures the Mahalanobis distance between a data point and the superclass distribution N (µ̄yi , Σ̄yi). Basically, π i j evaluates the density ratio of the mixture component mean µj , by regarding it as a sample from yi’s distribution.
πij = exp
( − 1
2 (µj − µ̄yi) ⊤Σ̄ −1 yi (µj − µ̄yi) ) ∑r
j′=1 exp ( − 1 2 (µj′ − µ̄yi)⊤Σ̄ −1 yi (µj′ − µ̄yi) ) = pN (µj |µ̄yi , Σ̄yi)∑r j′=1 pN (µj′ |µ̄yi , Σ̄yi)
(2)
where pN (·|·) represents the density function of a multivariate Gaussian distribution. Our generative process can be summarized as
1. for each sample index i: (a) draw a superclass label yi ∼ Categorical(p(yi)) (b) for j = 1, ..., r:
i. compute πij = softmax(−Dyi(µj)2/2) using Mahalanobis distance Dyi(·) (c) draw a latent subclass variable for the i-th sample zi ∼ Categorical([πi1, ..., πir]) (d) draw a feature vector vi ∼ N (µzi ,Σzi)
where p(yi) is a prior on superclasses, which can be drawn from a Dirichlet distribution Dir(α) (the conjugate of categorical distribution), and α can be estimated by the ratios of different labels in Dtrain. p(yi) can also be set as a uniform prior, for unknown datasets and better generalization. As can be seen, steps (a)(b) specify a (static) mixture of superclasses, and steps (c)(d) specify a (dynamic) mixture of subclasses. The whole process establishes a hierarchical GM distribution. Fig. 1(a)(b) illustrate the process using 2 superclasses and 3 subclasses, where the embedding distribution p(vi|yi) dynamically changes w.r.t. vi’s superclass as a result of updated πi. For example, when yi = 1 (Fig. 1(a)), vi is more likely to be generated from the first two subclasses. This yi-adjusted Gaussian mixture distribution highlights the first two subclasses, and will be used to fit data with yi = 1 during model training. Fig. 1(c) bottom illustrates the graphical model of SCGM, which distinguishes itself from the generic GM in Fig. 1(c) by (1) the condition yi, and (2) the variable πi.
Now, if we consider µ̄yi and µzi as the surrogate representations of yi and zi in the embedding space, we can specify two probabilities p(zi|yi) and p(vi|zi) using their corresponding densities:
p(zi|yi) = pN (µzi |µ̄yi , Σ̄yi), p(vi|zi) = pN (vi|µzi ,Σzi) (3) from which, the probabilities p(yi|zi) and p(zi|vi) in Eq. (1) can be derived by using Bayesian rules as (the detailed derivation is deferred to Appendix A.1)
pϕ(yi|zi) = pN (µzi |µ̄yi , Σ̄yi)p(yi)∑c
y′i=1 pN (µzi |µ̄y′i , Σ̄y′i)p(y
′ i) , pθ,ϕ(zi|vi) =
pN (vi|µzi ,Σzi)π i zi∑r
z′i=1 pN (vi|µz′i ,Σz′i)π i z′i
(4)
where θ are the parameters of encoder fθ, ϕ includes our added distribution parameters. To reduce complexity, in this work, we investigate the feasibility of using isotropic Gaussian with tied variance, i.e., Σzi = Iσ 2 and Σ̄yi = Iσ̄ 2, which turned out to be efficient in our experiments. Here, I is the identity matrix, σ and σ̄ are hyperparameters. Empirically, it is sufficient to set σ̄ = 1 and tune σ to adjust the relativity between super- and subclasses. Also, tied variance is a commonly used trick in Gaussian discriminate analysis (GDA) for generative classifiers (Lee et al., 2018).
Finally, substituting the factors in Eq. (1) with Eq. (4), whose probabilities are specified and parameterized, we get our SCGM induced loss ℓSCGM(Dtrain;θ,ϕ), where ϕ = {{µj}rj=1, {µ̄j}cj=1}.

heading: 3.2 MODEL OPTIMIZATION VIA EXPECTATION-MAXIMIZATION
text: It is hard to directly optimize ℓSCGM(Dtrain;θ,ϕ), because exact posterior inference is intractable (due to an exponential searching space). To solve it, we resort to variational methods, and introduce an approximated posterior q(zi|vi, yi), where yi is included so that the inference is also conditioned on superclass. Then, a lower-bound of Eq. (1) is derived as (the details are in Appendix A.2)
ℓSCGM(Dtrain;θ,ϕ) ≥ 1
n n∑ i=1 Eq(zi|vi,yi) [ log pϕ(yi|zi) + log pθ,ϕ(zi|vi)− log q(zi|vi, yi) ] (5)
which can be maximized by alternately inferring the posterior q(zi|vi, yi) and solving the model parameters {θ,ϕ} through an Expectation-Maximization (EM) algorithm. E-step. This step is to infer q(zi|vi, yi) when fixing model parameters. A straightforward way is to apply k-means on the embeddings vi, similar to (Caron et al., 2018). However, as discussed in (Asano et al., 2020), alternately applying k-means and learning embeddings without any constraints will lead to a degenerate solution, i.e., all samples are assigned to a single (arbitrary) subclass. Also, k-means is unaware of the superclass yi as it doesn’t optimize Eq. (5). To address this problem, inspired by (Asano et al., 2020), let Qzi,i = q(zi|vi, yi) 1n and Pzi,i = pϕ(yi|zi)pθ,ϕ(zi|vi) be two r × n matrices, we enforce an equal partition of subclasses by constraining Q to belong to the transportation polytope (Appendix A.3), and rewrite Eq. (5) as (the derivation is in Appendix A.4)
min Q∈Q
− ( Tr(Q⊤ logP) + 1
λ H(Q)
) , whereQ = { Q ∈ Rr×n+ |Q1n = 1
r 1r,Q
⊤1r = 1
n 1n
} (6)
where H(Q) = − ∑
ij Qij logQij is the entropy function, 1n is the vector of ones in dimension n, and λ ≥ 1 controls the regularization (it also further relaxes the lower-bound in Eq. (5)). The problem in Eq. (6) is a regularized instance of the optimal transport problem, whose minimizer can be written as Q∗ = Diag(u)PλDiag(v), where u ∈ Rr and v ∈ Rn are renormalization vectors. These vectors can be efficiently solved by the iterative Sinkhorn-Knopp algorithm (Cuturi, 2013). Once Q∗ is found, rounding its values and using discrete codes adds more algorithmic stability.
M-step. Fixing q(zi|vi, yi), the model parameters {θ,ϕ} can be efficiently solved by stochastic gradient descent (SGD). Combining Eq. (4) and Eq. (5), the loss to minimize is (Appendix A.5)
ℓϕ,θ = − 1
n n∑ i=1 q(zi|vi, yi) [ log exp (µ⊤zi · µ̄yi/σ̄ 2)p(yi)∑c y′i=1 exp (µ⊤zi · µ̄y′i/σ̄ 2)p(y′i) + log exp (v⊤i · µzi/σ 2)πizi∑c z′i=1 exp (v⊤i · µz′i/σ 2)πi z′i ] (7)
It is noteworthy that the two terms in the bracket have similar form to InfoNCE (Oord et al., 2018). Therefore, Eq. (7) can be interpreted from a contrastive perspective, where the positive pairs in both terms are specified by q(zi|vi, yi) from E-step, i.e., (zi, yi) and (zi,vi) that induce the maximal q(zi|vi, yi), and the variances σ2 and σ̄2 resemble temperature scaling. Thus, the M-step seeks to push the embedding vi to its closest subclass µzi , which should meanwhile be close to its superclass µ̄yi , in contrast to the distances of the negative pairs. In the second term, π i zi (Eq. (2)) serves as a weight to select µzi that is close to µ̄yi . As such, our problem can be seen as a generalized InfoNCE for subclass- and superclass-level contrastive learning.
In practice, we found it is beneficial to integrate Eq. (7) with cross-entropy loss ℓCE (on superclass)
ℓ(Dtrain;θ,ϕ) = ℓCE(Dtrain;θ, {µ̄j} c j=1) + γℓϕ,θ(Dtrain;θ,ϕ) (8)
where ℓCE uses superclass means {µ̄j}cj=1 as the classification weights, γ is a trade-off parameter. The detailed algorithm for training SCGM can be found in Algorithm 1 in Appendix A.6.

heading: 4 EXPERIMENTS
text: In this section, we first evaluate SCGM on benchmark datasets and compare it with SOTA methods. Then we evaluate SCGM for a model personalization task on a real-life medical dataset.
The table below summarizes the benchmark datasets: (1) BREEDS (Santurkar et al., 2020) includes four datasets {Living17, Nonliving26, Entity13, Entity30} derived from ImageNet with class hierarchy calibrated so that classes on the same level are of similar visual granularity (i.e., the granularity of images for distinguishing classes); (2) CIFAR-100 (Krizhevsky, 2009); and (3) tieredImageNet (Ren et al., 2018), which has 34 superclasses as per the ImageNet hierarchy, and was divided into 20/6/8 splits for (disjoint) train/val/test sets. For BREEDS and CIFAR-100, the val set is 10% of the train set. These datasets have been used in (Bukchin et al., 2021).
Dataset Living17 Nonliving26 Entity13 Entity30 CIFAR-100 Tiered # Coarase classes 17 26 13 30 20 20/6/8 # Fine classes 68 104 260 240 100 351/97/160 # Train images 88K 132K 334K 307K 50K 448K # Test images 3.4K 5.2K 13K 12K 10K 206K Image resolution 224×224 224×224 224×224 224×224 32×32 84×84

heading: 4.1 EXPERIMENTAL SETUP
text: Baselines. We compare SCGM with the most relevant SOTA models on embedding learning, including self-supervised models: (1) MoCo-v2 (Chen et al., 2020b) trained on each of the above dataset; (2) MoCo-v2-ImageNet pretrained on ImageNet by Chen et al. (2020b); (3) SwAV-ImageNet is a pretrained model by (Caron et al., 2020), which takes advantage of cluster-level contrastive learning. Note that pretrained models on full ImageNet saw more data during training than did SCGM. (4) ANCOR (Bukchin et al., 2021), as introduced in Sec.1 and 2; (5) GEORGE (Sohoni et al., 2020), a three-step framework that uses a separate clustering step to pseudo-fine-label the dataset; and (6) SeLa (Asano et al., 2020), an unsupervised algorithm for joint clustering and embedding. The detailed setup of these models was deferred to Appendix C.1.
Similar to (Bukchin et al., 2021), we include natural baselines trained on superclasses: (7) “Coarse”, uses encoder fθ followed by a classifier C; (8) “Coarse+”, uses fθ → E → C, which adds an embedder E . Also, baselines trained on subclass labels represent performance upper-bounds: (9) “Fine”, uses fθ → C; and (10) “Fine+”, uses fθ → E → C. For our method, since it is a flexible framework, we tested it with a generic encoder, and a momentum-based encoder with superclasswise dictionaries similar to ANCOR (Bukchin et al., 2021). We denote these two variants by SCGMG and SCGM-A, where the training loss (i.e., Eq. (8)) of the latter adds an InfoNCE term with angular normalization. The detailed implementation of SCGM-A can be found in Appendix C.2.
Implementation. The encoder is ResNet-50 (He et al., 2016) for the 224×224 images in BREEDS, and ResNet-12 for the small resolution images in CIFAR-100 and tieredImageNet, as is common in few-shot learning works. The output dimension of these networks is 2048 and 640, respectively. Similar to (Chen et al., 2020b), two types of embedder E were tested: (1) fc layer: d → e, and (2) MLP: d → d → e, with e = 128. At test time, some models performed better with E dropped while some were better otherwise. For each model, the better case was reported. For training, we used cosine annealing with warm restarts schedule (Loshchilov & Hutter, 2017) with 20 epochs per cycle. The batch size was 256 for BREEDS, 1024 for CIFAR-100, and 512 for tieredImageNet. The learning rate was 0.03 for BREEDs, and 0.12 for CIFAR-100 and tieredImageNet. The weight decay was 1e−4. All models were trained with 200 epochs. The best training practices of (Tian et al., 2020) were also applied. For SCGM, we set γ = 0.5, σ2 = 0.1, and λ = 25 (λ follows (Asano et al., 2020)). The number of latent variables r varies w.r.t. different datasets, and was grid-searched from 100 to 500 with step size 50. All hyperparameters were set according to validation sets.

heading: 4.2 EXPERIMENTAL RESULTS
text: Following (Tian et al., 2020), we report the mean accuracy and the 95% confidence interval of 1000 random episodes with 5-way/all-way k-shot, 15-query tests. Unless otherwise stated, k = 1. Effects of more shots are in Appendix D.1. For each episode, 5 augmented copies were created per support sample. A logistic regression classifier was trained on the support set embedded by each model with
L2 norm. The model with the resulting classifier was used to classify the query samples. The test classes of each episode were a random subset of all subclasses (or all of them for all-way tests). Two cases were evaluated: (1) unseen subclasses of seen superclasses (on BREEDS, CIFAR-100); and (2) unseen subclasses of unseen superclasses (on tieredImageNet). On CIFAR-100 and tieredImageNet, ImageNet-pratrained models were excluded since pretrained ResNet-12 models are unavailable.
Evaluation case (1). We first evaluate when test classes are unseen subclasses of the training superclasses. Table 1 and 2 summarize the results. For ANCOR, its variants when the embedder E is fc (with ‘-fc’) or MLP (without ‘-fc’) were included as either variant does not consistently outperform another. The embedder setup of all models are in Appendix C.1. From the tables, we have several observations. First, models that cannot leverage superclasses, such as MoCo-v2 and SeLa, are inferior to Coarse(+). Although full ImageNet-pretrained models used larger training datasets, they are unable to fill the gap of superclass supervision, compared with Coarse(+). Second, the limitation of GEORGE indicates a separate clustering step may induce misleading pseudo-labels that degenerate embedding quality. Third, ANCOR and SCGM consistently outperform Coarse(+), and SCGM significantly outperforms ANCOR(-fc) in most all-way cases. With a generic encoder, SCGM-G is suprior or competitive to ANCOR(-fc). Using momentum contrast, SCGM-A further gains on some datasets. Note that 5-way has less room to improve than the all-way cases, because the random subclasses are more likely to be from different superclasses, which may degrade the evaluation to coarse classification. The results suggest SCGM could perform better when more subclasses are from the same superclass. To investigate it, we evaluated an “intra-class” case when all subclasses of a random superclass were sampled in each episode. Table 3 indicates SCGM got more improvements in this challenging task, implying its embeddings distinguish subclasses better than other models.
Moreover, Table 4 compares the computational costs of ANCOR and SCGM-G, from which we can see training ANCOR requires much more resources and time than SCGM-G as it inherits MoCo-v2 (Chen et al., 2020b). Also, its memory costs vary a lot
Method 5-way-1shot 5-way-5shot All-way1-shot Fine 70.15±0.70 84.96±0.47 15.42±0.06 Fine+ 67.33±0.69 84.43±0.47 15.22±0.06 MoCo-v2 53.19±0.68 70.90±0.58 8.33±0.04 GEORGE 60.97±0.70 77.72±0.55 10.24±0.05 SeLa 47.96±0.63 66.39±0.57 7.17±0.04 C2F-VSME 60.54±0.79 75.22±0.63 N/A Coarse 61.61±0.74 75.72±0.56 8.42±0.04 Coarse+ 64.07±0.75 78.41±0.58 10.70±0.05 ANCOR 62.86±0.70 79.93±0.52 11.97±0.06 ANCOR-fc 60.43±0.69 78.18±0.53 11.04±0.05 SCGM-G 64.64±0.71 80.83±0.52 12.32±0.05 SCGM-A 64.47±0.71 80.68±0.51 11.75±0.05
Table 5: Comparison on tieredImageNet.
(a) ANCOR (b) SCGM-G
Figure 2: The tSNE visualization of the embeddings learned by ANCOR and SCGM-G. Star represents superclass means. “+” marker represents subclass means of SCGM-G.
by datasets since its dictionaries are superclass-wise whose size changes w.r.t. the number of superclasses. In contrast, SCGM-G had lower and stable costs, with better performance, thus is more favorable in practical deployments.
Evaluation case (2). Since tieredImageNet has distinct train/val/test sets, it was used to evaluate the case when the testing subclasses is out of any training superclass. All models were trained with the superclasses of the train set, and tested on the subclass labels of the test set. Table 5 presents the results. Another three-step method C2F-VSME (Yang et al., 2021) was included as the same evaluation has been performed (but their code is unavailable). First, from the smaller gaps between Fine(+) and Coarse(+), this task is more difficult than the previous case. Similar to Table 1, MoCov2 and SeLa are inferior to Coarse(+), underlining the benefit of superclass supervision, even when test subclasses are not their descendants. Compared to GEORGE and C2F-VSME, ANCOR and SCGM can better leverage superclasses, by using coherent learning frameworks. Both SCGM-G and SCGM-A outperform ANCOR(-fc) in most cases, demonstrating the effectiveness of explicit modeling of latent subclasses, in both evaluation case (1) and (2).
Visualization of embedding. To understand how SCGM uncovers the subclass structures, we visualize a random batch of embeddings using tSNE (Maaten & Hinton, 2008). Fig. 2 presents the visualization results of ANCOR and SCGM-G on Living17 dataset, where colors mark different superclasses, “+” markers are the learned subclass means by SCGM-G, and stars represent superclass means. ANCOR’s embeddings resemble those of Coarse+ (Appendix D.5) as it distinguishes subclasses from an angular perspective, which induces intra-class variation that is suboptimal. SCGM-G explicitly detects the boundaries of both super- and subclasses, properly positions Gaussian means, and associates subclasses with their corresponding superclasses, whereby establishing a hierarchical structure. We evaluated SCGM-G’s detected subclasses using the subclass labels in the val set. It got an NMI of 71.58% and purity accuracy of 74.79%, much higher than its unsupervised counterpart SeLa (NMI 36.50%, purity 38.14%). This validates SCGM’s effectiveness in leveraging superclass guidance for subclass inference. Several other model’s visualization can be found in Appendix D.5.
Performance analysis. We have evaluated SCGM in terms of the effects of more shots, the impacts of the number of latent variables r, the impacts of the variance σ2, and the convergence, which are in Appendix D.1, D.2, D.3, D.4.

heading: 4.3 CASE STUDY: PERSONALIZED PREDICTION ON MEDICAL RECORDS
text: Next, we evaluate SCGM on a real Dialysis-Event dataset collected by several hospitals in Japan. This dataset consists of one-year medical records of 673 hemodialysis patients. It contains 40 temporal features (e.g., blood flow, venous pressure, etc.) that were monitored during each hemodialysis session (thrice/week). Each session was annotated to indicate whether certain subtypes of events have occurred. A subtype corresponds to certain unstable behaviors of blood pressure during the dialysis, as categorized by experts. As introduced in Sec. 1, these events are risky and may develop complications, thus should be avoided. Formally, our task is as follows. Given xit:t+w−1 ∈ Rd×w, a length-w segment including the i-th patient’s records starting at time step t, where d = 40 is the dimensionality, the task outcome is a label y ∈ {0, 1} indicating event occurs (y = 1) or not (y = 0), at time step t + w. When y = 1, a fine-grained label ŷ ∈ {1, ..., s} is to be predicted to mark the subtype of the event.

heading: A segment of * records(a) (b) Personalized prediction comparison
text: Since annotating fine-grained labels is costly, training a model on them at a large scale is infeasible in practice. To predict event subtypes, we use a model personalization framework as illustrated in Fig. 3(a). At the pre-training stage, only binary labels are used. For every new patient, a short period of (a few) medical records are collected as the support set for the fine-grained labels w.r.t. that specific patient. It is used to train a personalized classifier based on the pre-trained model.
To evaluate how SCGM fits this paradigm, we randomly split the 673 patients into 50%/50% train/test set, and reserved 10% of the train set for validation. For each testing patient, the first 50% records (i.e., half year) form the support set, and the remaining form the query set. It is likely some event types didn’t appear in the support set will appear in the query set, which are out-ofdistribution (OOD) cases, thus is out of our scope. Hence, we only selected patients without such cases, which constitute a major part (>65%) of the test set. The dataset has two event subtypes, i.e. s = 2. The segments x(i)t:t+w−1 were sampled with w = 5, stride 1. The encoder is a Dual-Channel Combiner Network (DCCN), similar to (Che et al., 2016). DCCN has two channels (e.g., RNNs and MLP) for encoding temporal and static features, respectively. The outputs of them are integrated by a classification head. Its architecture can be found in Appendix C.3. In this application, each patient has 28 static features such as demographic information and infrequent blood test results.
Table 6 presents the average accuracy of all testing patients over 10 random train/test splits for the applicable methods. As is consistent with Sec. 4.2, unsupervised method SeLa and three-step framework GEORGE cannot improve DCCN’s performance due to (1) ignorance of superclass supervision, and (2) a separate pseudolabeling step, respectively. In contrast, SCGM outperforms the original DCCN significantly. Fig. 3(b) compares different models by their distributions of personalized prediction accuracy of different testing patients in a random train/test split. Each dot represents the mean accuracy of one patient’s personalized predictions in the
query set (220 dots in total). From the figure, SCGM has more dots close to the top (e.g., above 0.8) than other models, meaning more patients received fairly accurate predictions on the fine-grained labels. Although current performance could be limited by the data size, the results demonstrate the flexibility of SCGM, and its potential in tackling the CGFS problem in real practice.

heading: 5 CONCLUSION
text: In this paper, we proposed a new method, superclass-conditional Gaussian mixture model (SCGM), that underlies a general framework for tackling CGFS across domains. SCGM imitates the generative process of samples from hierarchies of classes, and explicitly models unobserved subclasses by latent variables. It dynamically computes a Gaussian mixture for every sample given its superclass, with a small overhead to an encoder. We learned the model parameters by maximum likelihood estimation in a principled Expectation-Maximization framework, and also theoretically linked SCGM with contrastive learning. The extensive experiments on various datasets demonstrated the effectiveness, efficiency, and flexibility of our proposed method.

heading: 6 ETHICS STATEMENT
text: In the case study in Sec. 4.3, we evaluated our SCGM on datasets related to human subjects that are hemodialysis patients. We clarify that we got the permission to perform the experiments on the dataset as described in Sec. 4.3. There is no ethics issue nor harmful insights in our current experiments. The dataset was properly used under the guidance of the data providers and domain experts. Sensitive information about the patients was well protected. The final description about the experiments was checked and approved by the data providers and relevant authorities.

heading: 7 REPRODUCIBILITY STATEMENT
text: The code of SCGM is available at https://github.com/nijingchao/SCGM for reproducibility study. The benchmark datasets are large and cannot be upoloaded. The repository includes an instruction on how to obtain the datasets and process them. Relevant instructions were provided on how to use the code to train SCGM-G and SCGM-A, and evaluate their performance for CGFS task in 5-way and all-way cases (Table 1, 2, 5), and in the intra-class case (Table 3).

heading: A APPENDIX FOR DETAILS OF DERIVING SCGM

heading: A.1 DERIVATION OF PARAMETERIZED PROBABILITIES
text: In this section, we provide the details of the probabilities in Eq. (4). First, using Bayesian rules and marginal property, we can derive
p(yi|zi) = p(zi|yi)p(yi)
p(zi) = p(zi|yi)p(yi)∑c y′i=1 p(zi|y′i)p(y′i) , p(zi|vi) = p(vi|zi)πizi p(vi) = p(vi|zi)πizi∑r z′i=1 p(vi|z′i)πiz′i (9)
Since µzi and µ̄yi are surrogate representations of zi and yi in the embedding space, substituting p(zi|yi) and p(vi|zi) with the equality in Eq. (3), Eq. (4) can be achieved.

heading: A.2 THE LOWER-BOUND OF THE LIKELIHOOD FUNCTION
text: In this section, we provide the details of the lower-bound in Eq. (5). By introducing the approximated posterior q(zi|vi, yi), the likelihood in Eq. (1) becomes
ℓ(Dtrain;θ) = 1
n n∑ i=1 log [ r∑ zi=1 p(yi|zi)pθ(zi|vi) q(zi|vi, yi) q(zi|vi, yi) ]
= 1
n n∑ i=1 log [ r∑ zi=1 q(zi|vi, yi) p(yi|zi)pθ(zi|vi) q(zi|vi, yi) ]
≥ 1 n n∑ i=1 r∑ zi=1 q(zi|vi, yi) [ log p(yi|zi) + log pθ(zi|vi)− log q(zi|vi, yi) ] = 1
n n∑ i=1 Eq(zi|vi,yi) [ log pϕ(yi|zi) + log pθ,ϕ(zi|vi)− log q(zi|vi, yi)
] (10)
where the third step uses Jensen’s inequality. This completes the derivation of Eq. (5).

heading: A.3 DETAILS ON TRANSPORTATION POLYTOPE
text: Let a be a probability vector in the simplex ∆r = {x ∈ Rr+|x⊤1r = 1} and b be a probability vector in the simplex ∆n = {x ∈ Rn+|x⊤1n = 1}, where 1r and 1n are the vectors of ones in dimension r and n, respectively. The transportation polytope of a and b, i.e., the polyhedral set of r × n matrices, is defined as (Cuturi, 2013)
Q = { Q ∈ Rr×n+ |Q1n = a,Q ⊤1r = b }
(11)
which means Q is a set of non-negative r × n matrices with row and column sums a and b, respectively. From a probabilistic perspective, any matrix Q ∈ Q represents a joint probability of two multinomial random variables X and Y that take values in {1, ..., r} and {1, ..., n} with marginal distributions a and b, respectively.
The transportation polytope defines a space of transport matrix Q for mapping a to b. Given a cost matrix M ∈ Rr×n+ , the cost of the mapping can be quantified as Tr(Q⊤M), and the problem
min Q∈Q
Tr(Q⊤M) (12)
is called an optimal transport (OT) problem between a and b, given cost M. Since the traditional algorithms to solve the OT problem scale inefficiently to large datasets, (Cuturi, 2013) introduced a fast Sinkhorn-Knopp algorithm, which amounts to solve a regularized problem
min Q∈Q Tr(Q⊤M)− 1 λ H(Q) (13)
where H(Q) = − ∑
ij Qij logQij is the entropy function, and λ is a trade-off parameter. If λ is large, optimizing Eq. (13) approximates optimizing Eq. (12).
In our case of Eq. (6), M = − logP, a = 1r1r, and b = 1 n1n. In addition, similar to (Asano et al., 2020), Qzi,i = q(zi|vi, yi) 1n can be considered as a joint probability of zi and the i-th sample (note the condition (vi, yi) corresponds to the incidence of the i-th sample) with a uniform prior 1n . For the details of the Sinkhorn-Knopp algorithm, we refer the readers to (Cuturi, 2013).
It is noteworthy that our optimization problem in Eq. (6) in the E-step is derived from our likelihood function in Eq. (5), which turns out to be a form of the regularized OT problem in Eq. (13), rather than directly formulated using Eq. (13). The derivation of the optimization problem in Eq. (6) can be found in next section (Appendix A.4).

heading: A.4 THE OPTIMIZATION PROBLEM IN E-STEP
text: In this section, we elaborate the optimization problem for inferring the posterior q(zi|vi, yi) in Eq. (6). Substituting q(zi|vi, yi) with nQzi,i, and substituting pϕ(yi|zi)pθ,ϕ(zi|vi) with Pzi,i, the third step in Eq. (10) can be rewritten by
n∑ i=1 r∑ zi=1 [ Qzi,i logPzi,i −Qzi,i logQzi,i −Qzi,i logn ] ≥ Tr(Q⊤ logP) + 1 λ H(Q)− logn (14) where Tr(·) is the trace function, H(·) is the entropy function, the last constant term is because∑n i=1 ∑r zi=1
Qzi,i = 1, and the inequality stems from λ ≥ 1, which is introduced to control the strength of the regularization (Cuturi, 2013). This is the objective function in Eq. (6), where the constant log n is omitted because it has no effects on the optimization.
The right hand side of Eq. (14) is a valid lower bound of the likelihood function in Eq. (5) according to Eq. (10). Therefore, optimizing it (i.e., the objective function in Eq. (6)) provides a reasonably approximated solution to the likelihood function in Eq. (5). In addition, the transport polytope Q in Eq. (6) is introduced as an equal partition constraint on subclasses (Asano et al., 2020), so that the optimization problem is feasible. It is noteworthy that although subclasses have the equal partition constraint, superclasses can have different sizes since they can have different numbers of subclasses.

heading: A.5 THE OPTIMIZATION PROBLEM IN M-STEP
text: In this section, we derive the optimization problem for learning model parameters in Eq. (7). Introducing the probabilities specification in Eq. (4) to the negative of the lower-bounded likelihood in Eq. (5), and removing terms that are irrelevant to model parameters {θ,ϕ}, we have
ℓθ,ϕ = 1
n n∑ i=1 r∑ zi=1 q(zi|vi, yi) [ log pN (µzi |µ̄yi , Σ̄yi)p(yi)∑c y′i=1 pN (µzi |µ̄y′i , Σ̄y′i)p(y ′ i) + log pN (vi|µzi ,Σzi)π i zi∑r z′i=1 pN (vi|µz′i ,Σz′i)π i z′i ] (15)
Considering isotropic Gaussian with Σ̄yi = Iσ̄ 2, we have
pN (µzi |µ̄yi , Σ̄yi) = exp (−(µzi − µ̄yi)2
2σ̄2
) = exp ( (µ⊤zi · µ̄yi − 1) σ̄2 ) = exp (µ⊤zi · µ̄yi/σ̄ 2) exp (1/σ̄2) (16)
where the second equivalence is because we apply L2 norm to µzi and µ̄yi , respectively. Similarly, considering Σzi = Iσ 2, and applying L2 norm to vi, we have
pN (vi|µzi ,Σzi) = exp (−(vi − µzi)2
2σ2
) = exp ( (v⊤i · µzi − 1) σ2 ) = exp (v⊤i · µzi/σ 2) exp (1/σ2) (17)
Substituting the corresponding terms in Eq. (15) with Eq. (16) and Eq. (17), and removing the constants exp (1/σ̄2) and exp (1/σ2)in the fractions, we obtain the loss function in Eq. (7). In Eq. (7), the prior p(yi) can be set as uniform, i.e., p(yi) = 1c . If using a prior from Dirichlet distribution, p(yi) can be instantiated by estimating the parameters α (of the Dirichlet distribution) by the ratio of different superclass labels in Dtrain, the training dataset. Also, πizi in the second term of Eq. (7) can be calculated using Eq. (2).

heading: A.6 THE TRAINING ALGORITHM OF SCGM
text: The training algorithm of SCGM is summarized in Algorithm 1.
Algorithm 1: Superclass-conditional Gaussian mixture model (SCGM) Input: encoder fθ , training dataset Dtrain (number of classes is c), hyperparameters r, λ, γ, σ Output: model parameters {θ, ϕ}
1 ϕ = Initialization(c, r) // initialize Gaussian means ϕ = {{µj}rj=1, {µ̄j}cj=1} 2 for i← 1 to MaxEpoch do /* E-step */ 3 for (x, y) in Dataloader(Dtrain) do /* load a minibatch (x, y) */ 4 v = fθ(x) // get embeddings 5 Compute p for Eq. (6) based on Eq. (4) using v and ϕ 6 end 7 Concatenate all batch-wise probabilities p to get P for Dtrain 8 Q = SinkhornKnopp(P, λ) // Inferring posterior using Sinkhorn-Knopp
algorithm for optimizing Eq.(6) /* M-step */
9 for (x, y,q) in Dataloader(Dtrain) do /* load a minibatch (x, y,q) where q is obtained from E-step */
10 v = fθ(x) // forward pass through the encoder 11 Calculate ℓ(v, y,q,ϕ, γ, σ) // calculate loss in Eq.(8) using ℓϕ,θ in Eq.(7) 12 θ,ϕ = SGD(ℓ,θ,ϕ) // update model parameters 13 end 14 end

heading: B APPENDIX FOR FURTHER DISCUSSION

heading: B.1 DISCUSSION ON THE EQUAL PARTITION CONSTRAINT
text: The equal partition constraint on subclasses in Eq. (6) theoretically assumes subclasses are of equal or similar sizes, through using Q1n = 1r1r (i.e., a = 1 r1r in Eq. (11)) as a prior distribution of the subclass sizes (superclasses can have different sizes because they can have different numbers of subclasses). It is possible to replace 1r1r by other prior distributions to reflect an uneven distribution of the subclasses sizes. In this work, we used a uniform prior because we don’t assume such prior knowledge (since the subclasses are unobserved during training). This prior is inspired by (Asano et al., 2020). If we don’t have such a prior, the searching space of subclasses will be exponential and the optimization is intractable. To make the optimization feasible, we included this constraint to help reduce the searching space. In this way, we found the solution is efficient and effective to our CGFS problem, which does not necessitate an exact identification of every subclass. Compared to other baseline methods, the performance improvements indicate that reasonable boundaries between subclasses can be learned by SCGM with this constraint for model adaptation, though the boundaries may not be ideal. However, it is worth to note that this uniform prior may not always be reasonable in different applications. For example, if identifying rare subclass is important in an application (Sohoni et al., 2020), a more reasonable prior may be considered. In our experiments, the evaluations on the benchmark datasets (which were used for a consistent setup with the existing works (Bukchin et al., 2021)) do not reflect the case of substantially skew distributions of subclass sizes. We leave the extension of the proposed method to this scenario in our future work.

heading: B.2 DISCUSSION ON THE EVALUATION CASES IN SECTION 4.2
text: In Section 4.2, we evaluated two fundamental cases of (1) unseen subclasses of seen superclasses; and (2) unseen subclasses of unseen superclasses. Another case that considers the mixture of the subclasses of seen and unseen superclasses is worth discussion because it is a practical case. As discussed in (Xian et al., 2018; Oreshkin et al., 2020), a mixture of samples from seen and unseen classes during testing may result in an imbalance of prediction performance between seen and unseen classes, and the inductive bias may lead to misclassification of unseen classes towards the seen classes. It is noteworthy that (Xian et al., 2018; Oreshkin et al., 2020) focus on a generalized zero-shot learning problem, which is not for cross-granularity adaptation and don’t consider class hierarchy. Thus their task is remarkably different from our CGFS task. Despite the difference, we are inspired to be aware of the challenge that there could be a similar imbalance in the prediction of
the subclasses of seen and unseen superclasses if they are mixed. This could be an open challenge in the CGFS task which is rarely discussed in the existing works. To alleviate this problem, some specific technical extensions may be designed, as discussed in (Xian et al., 2018; Oreshkin et al., 2020), which is out of our scope. We leave this scenario in our future work.

heading: C APPENDIX FOR IMPLEMENTATION DETAILS

heading: C.1 THE SETUP OF THE COMPARED MODELS
text: For MoCo-v2 and MoCo-v2-ImageNet, following (Chen et al., 2020b), we set its queue size 65536, momentum 0.999 (0.99 for CIFAR-100), temperature 0.2, embedder E as MLP. For SwAVImageNet, following (Caron et al., 2020), its number of prototypes was 3000, embedder E as MLP. For ANCOR, following (Bukchin et al., 2021), we set its queue size 65536, momentum 0.999 (0.99 for CIFAR-100), temperature 0.2, embedder E as fc and MLP, with the multi-class queue, and angular norm on. For GEORGE (Sohoni et al., 2020), we set its robust learning rate 1e−4, E as fc, (dim 16, 10 neighbors, 0 distance) for its dimension reduction step, the number of clusters per superclasses was grid-searched within {5, 10, 15, 20}. For SeLa, following (Asano et al., 2020), we set its λ = 25, E as fc, and number of clusters was grid-searched from 100 to 500 with step size 50. For Coarse+ and Fine+, we set E as MLP. For our SCGM, we set γ = 0.5, σ2 = 0.1, and λ = 25 (λ follows (Asano et al., 2020)), E as fc, the number of latent variables r was grid-searched from 100 to 500 with step size 50.
Remarks on embedder. For the compared methods, adding an embedder follows the practice of the existing works such as MoCo-v2 (Chen et al., 2020b) and SimCLR (Chen et al., 2020a), which was also used in the different compared methods in (Bukchin et al., 2021). In these works, adding certain types of embedder (e.g., MLP) is empirically found helpful for improving performance. We have the same observation in our experiments, and think the effectiveness could come from (1) more capacity; and (2) better generalization in some cases. The latter is because after training, the embedder may be specific to the pre-training tasks, which enables more generalizable representations to be learned by the backbone networks. Thus dropping the embedder at testing stage may help alleviate overfitting to some extent. For comprehensiveness, we evaluated both of the cases when an embedder is added or not in our experiments, as described in Section 4.1.

heading: C.2 THE ARCHITECTURE OF SCGM-A
text: Fig. 4 illustrates the architecture of SCGM-A, which was implemented on a generic encoder (e.g., ResNet-50) with momentum encoders. The query v and positive key k+ are computed from two random augmentations of the input images through the encoder and embedder, and their momentum updates. There is a superclass-wise dictionary maintains c queues for c different superclasses so that superclass-wise negative keys k− can be generated for within-superclass contrastive learning, which facilitates preserving intra-class variation. At the SCGM classification head, v, k+ and k− are applied with an angular normalization, followed by the InfoNCE loss, where the angular normalization
is defined by (Bukchin et al., 2021) |vi|A = ( vi ∥vi∥ − µ̄yi ∥µ̄yi∥ ) /∥ vi∥vi∥ − µ̄yi ∥µ̄yi∥ ∥ (18)
which enables contrasts from an angular perspective, and induces better synergy with cross-entropy loss.
The top part of the SCGM classification head are similar to its counterpart using a generic encoder, where v interacts with superclass means {µ̄j}cj=1 and subclass means {µj}rj=1 for computing our SCGM loss ℓθ,ϕ and a cross-entropy loss ℓCE as in Eq. (8). Thus the loss for training SCGM-A is
ℓ(Dtrain;θ,ϕ) = ℓCE(Dtrain;θ, {µ̄j} c j=1) + ℓInfoNCE(|Dtrain|A;θ) + γℓϕ,θ(Dtrain;θ,ϕ) (19)
where the model parameters are learned by our proposed EM-based algorithm that alternates between the SGD backpropagation (M-step) for computing model parameters {θ,ϕ} and the posterior inference (E-step) for computing subclass distributions.

heading: C.3 THE ARCHITECTURE OF DCCN
text: Fig. 5 illustrates the architecture of the Dual-Channel Combiner Network (DCCN). DCCN is effective to process heterogeneous medical records data that usually consists of static profiles and time series (i.e., records) (Che et al., 2016). It has a static channel realized by MLPs to encode static features (e.g., demographic information, infrequent blood test results, etc.) and a temporal channel realized by RNNs (e.g., LSTM) to encode temporal feature (e.g., blood flow, venous pressure, etc.). The hidden representations output by the two channels are concatenated (or using certain pooling method) and further projected to a compact embedding that is used for prediction by the classification head, i.e., the combination layer in Fig. 5, which is realized by MLPs. Attention layer may be added before the combination layer for weighted combination of the hidden representations from different channels and different time steps.

heading: D APPENDIX FOR MORE EXPERIMENTAL RESULTS

heading: D.1 EFFECTS OF MORE SHOTS
text: Table 7 and 8 summarize the performance of SCGM-G and SCGM-A, with the strongest baseline ANCOR and ANCOR-fc, together with the coarse baseline Coarse+, on Living17 dataset. From
Table 7, we observe SCGM-A continuously improves performance as more shots are added, consistently outperforms other models, which validate the effectiveness of SCGM framework with varying number of shots. SCGM-G’s performance becomes comparable to ANCOR(-fc) as more shots are added, due to its simpler architecture, which however is more computationally efficient. Thus it provides a good balance between performance and efficiency. From Table 8, we observe ANCOR and SCGM perform similarly. This is because the task is close to the regular superclass classification, and its performance limit can be easily reached by adding a few more shots. This can be seen from Coarse+’s results, which become close to other models when there are 20 shots in the support set.
D.2 IMPACTS OF THE NUMBER OF SUBCLASSES
Table 9 presents the results of SCGM-G on Living17 by varying the number of latent variables r from 50 to 500. As can be seen, the performance of SCGM-G is generally stable and it consistently outperforms Coarse(+) (Table 1) w.r.t. different r. By looking into the details, we observe too small (e.g., 50) or too big (e.g., 500) r may degenerate the performance in the all-way case, which may be caused by slight underfitting and overfitting, respectively. In the 5-way case, small r does not impact performance obviously, this is because the 5-way case is close to evaluate coarse classification, which does not need exact inference of subclasses. Thus a small number of r is fine (which may not distinguish all possible subclasses).
D.3 IMPACTS OF THE VARIANCE
Table 10 presents the results of SCGM-G on Living17 by varying the variance σ2 of subclasses, which resembles the temperature scaling in Eq. (7). As described in Sec. 3.1, in our experiments, we set σ̄2 = 1 and tune σ2 to adjust the relativity between super- and subclasses, which we found is empirically effective. From Table 10, we observe σ2 is better to be selected from 0.1 to 0.2, and a too small (e.g., 0.05) or too big (e.g., 0.3) value may degenerate the performance in the all-way case. This observation is similar to the evaluation of the temperature scaling parameter in MoCo-v2 (Chen et al., 2020b). In the 5-way case, the performance is relatively stable w.r.t. different values of σ2. Similar to the observation in Appendix D.2, this is because the 5-way case is close to evaluate coarse classification, which is less impacted by the learning of subclasses.

heading: D.4 CONVERGENCE ANALYSIS
text: Fig. 6 presents the training loss ℓ(Dtrain;θ,ϕ) in Eq. (8) of SCGM-G, and ℓ(Dtrain;θ,ϕ) in Eq. (19) of SCGM-A, during the training process on the four BREEDS datasets. The training alternates between E-step and M-step, where E-step runs every 5 epochs. As can be seen, although there are some periodic fluctuations, the loss function values generally decrease and converge to a small value across different datasets. The periodic fluctuations are not synchronized with the alternate E-step and M-step since its period is not 5 epoch as can be observed. In practice, we observed both E-step
and M-step can decrease the loss function values since both steps are theoretically optimizing the same loss function. The instability may come from the rounding step (as discussed in Sec. 3.2) for generating discrete code from the posterior, which follows the E-step. This rounding step may occasionally lead to some jump of loss values since it may make the optimization more aggressive than gradient updates. However, we still found this rounding step is beneficial, since without it, too small values may be generated in the posterior inference, which is unstable in computation. Generally, the current setup provides a stable training framework and we don’t need to interfere in the program during the training process on all of our experimental datasets.
D.5 VISUALIZATION OF EMBEDDING
In addition to the visualization results of ANCOR and SCGM-G provided in Fig. 2, Fig. 7 presents the tSNE visualization of the embeddings of several most relevant methods in comparison using Living17 dataset, including Coarse+, MoCo-v2, ANCOR-fc and SCGM-A. In the figure, colors mark different superclasses, “+” markers are the learned subclass means by SCGM-G, and stars represent superclass means. From Fig. 7(a)(c), we can observe the embeddings of both ANCOR and ANCORfc resemble Coarse+. Although ANCOR-fc slightly distinguish several small groups of embeddings in some superclasses, their intra-class variation is suboptimal. From Fig. 7(b), MoCo-v2 has the ability to detect superclasses to some extent, but cannot fill the gap of superclass supervision, by comparing with Coarse+, in this dataset. The reason may be the Living17 dataset is not sufficiently large and comprehensive for MoCo-v2 to learn embeddings that are comparable to supervised methods. In contrast to them, SCGM-A (and SCGM-G in Fig. 2) explicitly detects subclasses, with proper position of subclass means and superclass means. It also associates every subclass to its corresponding superclass, which establishes a hierarchical class structure. This explains the effectiveness of SCGM in cross-granularity learning tasks.

Review summaries:
The paper presents SCGM, a new technique for solving the Cross-Granularity Few-Shot learning (CGFS) problem.  CGFS is defined as the problem of adapting a classification model trained on coarse (“superclass”) labels to perform well on fine-grained labels, which consist of multiple “subclass” labels within each superclass.  The paper presents a new generative modeling approach that enables end-to-end classifier training in a manner that (a) incorporates information about superclass-subclass hierarchies (b) does not rely on explicit subclass enumeration (c) provides improved empirical performance on CGFS and (d) provides systems benefits with respect to existing baselines.  The authors provide detailed description of their approach and characterize its relationship with existing methods, and present empirical results that support their methodological choices.

The paper introduces a setup with the goal to adapt from a coarse pretrained model to unseen fine-grained labels. This problem is formulated as a superclass-subclass latent model and learned with maximum likelihood via expectation-maximization. The proposed approach, super-class conditional Gaussian mixture (SCGM) model, defines a hierarchical Gaussian distribution on the class hierarchy that models both the superclasses and subclasses. SCGM is evaluated on 6 image-net related datasets and a a real Dialysis-Event dataset collected by hospitals, and demonstrates competitive results under two evaluation setups: generalizing to seen and unseen superclass.