SlowGuess/NeurIPS_2023_OCR huggingface dataset:
B Analysis of results
B.1 Clevr-4 error bars
We show results for the GCD baseline [7], the current state-of-the-art SimGCD [19] and our method,  $\mu$  GCD, in fig. 9. The results are shown for five random seeds for each method, and plotted with the
Color
blue
brown
cyan
gray
orange
pink
purple
red
yellow
Figure 8: Examples of each category from the color and count taxonomies of Clevr-4.
Count
1
standard matplotlib boxplot function, which identifies outliers in colored circles. We also plot the median performance of our method on each taxonomy in dashed lines.
Broadly speaking, the takeaways are the same as the results from Table 4 of the main paper. However, while the mean performance of our method is worse than SimGCD on the shape split, we can see here that the median performance of  $\mu$  GCD is within bounds, or significantly better, than the compared methods on all taxonomies.
B.2 shape failure case
Overall, we find our proposed  $\mu$ GCD outperforms prior state-of-the-art methods on three of the four Clevr-4 splits (as well as on the Semantic Shift Benchmark [20]). We further show in appendix B.1 that, when accounting for outliers in the five random seeds, our method is also roughly equivalent to the SimGCD [19] state-of-the-art on the shape split of Clevr-4.
Nonetheless, we generally find that our method is less stable on the shape split of Clevr-4 than on other taxonomies and datasets. We provide some intuitions for this by visualizing the representations and predictions of our method in fig. 10.
Figure 9: Box plots of results on Clevr-4. We show results for the GCD baseline [7], the current state-of-the-art SimGCD [19] and our method,  $\mu$ GCD. We plot results for five random seeds for the four taxonomies, with outliers shown as colored circles. We also plot the median performance of our method on each taxonomy in dashed lines. On all taxonomies,  $\mu$ GCD is within bounds, or significantly better, than the compared methods.
Preliminaries: In fig. 10, we plot TSNE projections [75] of the feature spaces of two versions of our model, as well as the histograms of the models' predictions on the shape split. Along with the models' image representations (colored scatter points), we also plot the class vectors of the cosine classifiers (colored stars). On the left, we show our trained model when we randomly initialize the cosine classifier, while on the right we initialize the class vectors in the classifier with  $k$ -means centroids. We derive these centroids by running standard  $k$ -means on the image embeddings of the backbone, which is pre-trained with the GCD-style representation learning step (see appendix D).
Observations: In the plot on the left, we find that though the feature space is very well separated (there is little overlap between clusters of different categories), the performance of the classifier is still only around  $90\%$ . The histogram of model predictions demonstrates that this is due to no images being assigned to the 'star' category – this vector in the classifier is completely unused. Instead, too many instances are assigned to 'gear'. In the TSNE plot, we can see that the 'gear' class vector is between clusters for both 'gear' and 'star' images, while the 'star' vector is pushed far away from both. We suggest that this is due to the optimization falling into a local optimum early on in training, as a result of the feature-space initialization already being so strong.
On the right, we find we can largely alleviate this problem by initializing the classification head carefully – with  $k$ -means centroids from the pre-trained backbone. We see that the problem is nearly perfectly solved, and the histogram of predictions reflects the true class distribution of the labels.
Takeaway: We find that when the initialization of the model's backbone – from the GCD-style representation learning step, see appendix D – is already very strong, random initialization of the classification head in  $\mu$ GCD can result in local optima in the model's optimization process. This can be alleviated by initializing the classification head carefully with  $k$ -means centroids – resulting in almost perfect performance – but the issue can persist with some random seeds (see appendix B.1).
B.3 Semi-supervised  $k$ -means with pre-trained backbones
In fig. 11, we probe the effect of running semi-supervised  $k$ -means [7] on top of different pre-trained backbones. This is a simple mechanism by which models can leverage the information from the 'Old' class labels. We find that while this improves clustering performance on some taxonomies, it is insufficient to overcome the biases learned during the models' pretraining, corroborating our findings from table 2 of the main paper.
B.4 Clustering with sub-spaces of pre-trained features
In table 2 and fig. 11, we demonstrate that all pre-trained models have a clear bias towards one of the Clevr-4 taxonomies. Specifically, we find that clustering in pre-trained feature spaces preferentially aligns with a single attribute (e.g. shape or color).
Figure 10: Analysis of the shape failure mode, showing TSNE plots [75] and prediction histograms for two models, trained without (left) and with (right) initialization of the classification head with  $k$ -means centroids. Left: When the backbone initialization (from the GCD representation learning step [7]) is already very strong, the classification head gets stuck in a local optimum, with one class vector unused. Right: We find we can alleviate this by initializing the class vectors with  $k$ -means centroids, almost perfectly solving the problem, but the issue can persist with some random seeds.
Figure 11: Effect of semi-supervised  $k$ -means on representative pre-trained backbones. We find semi-supervised  $k$ -means is insufficient to overcome the bias learned during pre-training.
Here, we investigate whether these clusters have any sub-structures. To do this, we perform PCA analysis on features extracted with two backbones: DINOv2 [60] and MAE [31]. Intuitively, we wish to probe whether the omission of dominant features from the backbones (e.g. the shape direction with DINOv2 features) allows  $k$ -means clustering to identify other taxonomies. Specifically, we: (i) extract features for all images using a given backbone,  $\mathbf{X} \in \mathbb{R}^{N \times D}$ ; (ii) identify the principal components of the features, sorted by their component scores,  $\mathbf{W} \in \mathbb{R}^{D \times D}$ ; (iii) re-project the features onto the components, omitting those with the  $p$  highest scores,  $\hat{\mathbf{X}} = (\mathbf{X} - \mu) \cdot \mathbf{W}(:, p:)$ ; (iv) cluster the resulting features,  $\hat{\mathbf{X}} \in \mathbb{R}^{N \times D - p}$ , with  $k$ -means. Here,  $\mu$  is the average of the features  $\mathbf{X}$ , and the results are shown in figs. 12 and 13.
Overall, we find that by removing the dominant features from the backbones, performance on other taxonomies can be improved (at the expense of performance on the 'dominant' taxonomy). The effect is particularly striking with MAE, where we see an almost seven-fold increase in shape performance after the three most dominant principal components are removed.
This aligns with the reported performance characteristics of DINOv2 and MAE. The object-centric recognition datasets on which these models are evaluated benefit from shape-biased representations (see section 6). We find here that both MAE and DINOv2 encode shape information, but that more work is required to extract this from MAE features. This is reflected by the strong linear probe and kNN performance of DINOv2, while MAE requires full fine-tuning to achieve optimal performance.
Figure 12: Re-clustering DINOv2 [60] features after removing dominant principal components.
Figure 13: Re-clustering MAE [31] features after removing dominant principal components.
Finally, we note that decoding the desired information from pre-trained features is not always trivial, and we demonstrate in section 4 that even in the (partially) supervised, fine-tuning setting in GCD, both of these backbones underperform a randomly initialized ResNet18 on the count taxonomy.
B.5 Design of data augmentation and mis-aligned augmentations
In our SSB experiments, the teacher is passed a weaker augmentation, comprising only RandomCrop and RandomHorizontalFlip. We find this stabilizes the pseudo-labels produced by the teacher. However, the self-supervised literature consistently finds that strong augmentations are beneficial for representation learning [9, 14, 21]. As such, we experiment with gradually increasing the strength of the augmentation passed to the student model in table 7.
Specifically, we experiment along two axes: the strength of the base augmentation ('Strong Base Aug' column); and how aggressive the cropping augmentation is ('Aggressive Crop' column). To make the base augmentation stronger, we add Solarization and Gaussian blurring [21]. For cropping, we experiment with a light RandomResizeCrop (cropping within a range of 0.9 and 1.0) and a more aggressive variant (within a range of 0.3 and 1.0). Overall, we find that an aggressive cropping strategy, as well as a strong base augmentation, is critical for strong performance. We generally found weaker variants to overfit. Though they also have lower peak clustering accuracy, the accuracy falls sharply later in training without the regularization from strong augmentation.
Table 7: Design of student augmentation.
Aggressive Crop	Strong Base Aug	CUB
All	Old	New
X	X	38.6	54.6	30.6
X	✓	41.6	58.8	33.0
✓	X	52.7	69.4	44.7
✓	✓	65.7	68.0	64.6
Table 8: Effect of mis-aligned augmentations on the GCD Baseline.
Color	Count
Aligned Augmentation	84.5	65.2
Misaligned Augmentation	26.1	46.6
Figure 14: Effect of hyper-parameter,  $\lambda_{1}$ . We investigate the effect of  $\lambda_{1}$  (which balances the supervised and un-supervised losses), training  $\mu$ GCD models on the texture split.
Mis-aligned augmentations on Clevr-4 A benefit of Clevr-4 is that each taxonomy has a simple semantic axes. As such, we are able to conduct controlled experiments on the effect of targeting these axes with different data augmentations. Specifically, in table 8, we demonstrate the effect of having 'misaligned' augmentations on two splits of Clevr-4. We train the GCD baseline with ColorJitter on the color split and CutOut for the count split. The augmentations destroy semantic information for the respective taxonomies, resulting in substantial degradation of performance. The 'aligned' augmentations are light cropping and flipping for color, and light rotation for count. The results highlight the importance of data augmentation in injecting inductive biases into deep representations.
B.6 Effect of  $\lambda_{1}$
In fig. 14, we investigate the effect of the hyper-parameter  $\lambda_{1}$ , which controls the tradeoff between the supervised and unsupervised losses in  $\mu \mathrm{GCD}$ . We find that with  $0.1 < = \lambda_{1} < = 0.4$ , the 'All' clustering accuracy is robust, while at  $\lambda_{1} = 0$  (only unsupervised loss) and  $\lambda_{1} = 1$  (only supervised loss), performance degrades. We note that the Hungarian assignment in evaluation results in imperfect 'Old' performance even at  $\lambda_{1}$  close to 1 (more weight on the supervised loss). As such, we also show an 'Upper Bound' ('cheating') clustering performance in gray, which allows re-use of clusters in the 'Old' and 'New' accuracy computation.
C.1 Results with estimated number of classes
In the main paper, we followed standard practise in category discovery [7, 19, 43, 46, 50, 52] and assumed knowledge of the number of categories in the dataset,  $k$ . Here, we provide experiments when this assumption is removed. Specifically, we train our model using an estimated number of categories in the dataset, where the number of categories is predicted using an off-the-shelf method from [7]. We use estimates of  $k = 231$  for CUB and  $k = 230$  for Stanford Cars, while these datasets have a ground truth number of  $k = 200$  and  $k = 196$  classes respectively.
We compare against figures from SimGCD [19] as well as the GCD baseline [7]. As expected, we find our method performs worse on these datasets when an estimated number of categories is used, though we note that the performance of SimGCD [19] improves somewhat on CUB, and the gap between our methods is reduced on this dataset. Nonetheless, the proposed  $\mu$ GCD still performs marginally better on CUB, and further outperforms the SoTA by nearly  $7\%$  on Stanford Cars in this setting.
Table 9: Results on the SSB with estimated number of categories. We use the method from [7] to estimate the number of categories as  $k = 231$  for CUB, and  $k = 230$  for Stanford Cars. We run our method with this many vectors in the classification head, comparing against baselines evaluated with the same estimates of  $k$ . Results from baselines are reported from [19].
Pre-training	CUB	Stanford Cars	Average
All	Old	New	All	Old	New	All
GCD [7]	DINO [14]	47.1	55.1	44.8	35.0	56.0	24.8	41.1
SimGCD [19]	DINO [14]	61.5	66.4	59.1	49.1	65.1	41.3	55.3
μGCD (Ours)	DINO [14]	62.0	60.3	62.8	56.3	66.8	51.1	59.2
Figure 15: Results when varying the proportion of 'Old' category images reserved reserved for  $\mathcal{D}_{\mathcal{L}}$ . We find our  $\mu$ GCD method substantially outperforms the GCD baseline [7] across all settings.
C.2 Results with varying proportion of labelled examples
In the main paper, we follow standard practise in the GCD setting [7, 19, 50, 52] and sample a fixed proportion of images,  $p = 0.5$ , from the labelled categories and use them in the labeled set,  $\mathcal{D}_{\mathcal{L}}$ . Here, we experiment with our method if this proportion changes, showing results in fig. 15. We find our proposed  $\mu$  GCD substantially outperforms the GCD baseline [7] across all tested values of  $p$ .
C.3 Results on Herbarium19
We evaluate our method on the Herbarium19 dataset [66]. We use the 'Old'/'New' class splits from [7] which are randomly sampled rather than being curated as they are in the SSB. Nonetheless, the dataset is highly challenging, being long-tailed and containing 683 classes in total. 341 of these classes are reserved as 'Old', and the dataset contains a total of  $34K$  images. It further contains a clear taxonomy (herbarium species), making it a suitable evaluation for GCD. We compare  $\mu\text{GCD}$  against prior work in table 10, again finding that we set a new state-of-the-art.
D Description of baselines and  $\mu$ GCD algorithms
In this section we provide step-by-step outlines of: the GCD baseline [7]; the SimGCD [19] baseline; and our method,  $\mu$  GCD. Full motivation of the design decisions in  $\mu$  GCD can be found in section 4.
Task definition and notation: Given a dataset with labelled  $(\mathcal{D}_{\mathcal{L}})$  and unlabelled  $(\mathcal{D}_{\mathcal{U}})$  subsets, a model must classify all images in  $\mathcal{D}_{\mathcal{U}}$  into one of  $k$  possible categories.  $\mathcal{D}_{\mathcal{L}}$  contains only a subset of the categories in  $\mathcal{D}_{\mathcal{U}}$ , and prior knowledge of  $k$  is assumed. During training, batches  $(\mathcal{B})$ , are sampled with both labelled images  $(\mathcal{B}_{\mathcal{L}} \in \mathcal{D}_{\mathcal{L}})$  and unlabelled images  $(\mathcal{B}_{\mathcal{U}} \in \mathcal{D}_{\mathcal{U}})$ . The performance metric is the clustering (classification) accuracy on  $\mathcal{D}_{\mathcal{U}}$ .
GCD [7]. Train a backbone,  $\Phi$ , and perform classification by clustering in its feature space.
(1) Train  $\Phi$  using an unsupervised InfoNCE loss [23] on all the data, as well as a supervised contrastive loss [61] on the labeled data. Letting  $\pmb{x}_i$  and  $\pmb{x}_i'$  represent two augmentations of the same image in a batch  $\mathcal{B}$ , the unsupervised and supervised losses are defined as:
Table 10: Results on Herbarium19 [66], which is a long-tailed recognition dataset.
Pre-training	Herbarium19
All	Old	New
k-means [57]	DINO [14]	13.0	12.2	13.4
RankStats+ [46]	DINO [14]	27.9	55.8	12.8
UNO+ [43]	DINO [14]	28.3	53.7	12.8
GCD [7]	DINO [14]	35.4	51.0	27.0
ORCA [8]	DINO [14]	20.9	30.9	15.5
OpenCon [52]	DINO [14]	39.3	58.9	28.6
PromptCAL [50]	DINO [14]	37.0	52.0	28.9
MIB [51]	DINO [14]	42.3	56.1	34.8
SimGCD [19]	DINO [14]	43.3	57.9	35.3
μGCD (Ours)	DINO [14]	45.8	61.9	37.2
$$
\mathcal {L} _ {f e a t, i} ^ {u} = - \log \frac {\exp \left\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {i} ^ {\prime} \right\rangle / \tau}{\sum_ {n} ^ {n \neq i} \exp \left\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {n} \right\rangle / \tau}, \quad \mathcal {L} _ {f e a t, i} ^ {s} = - \frac {1}{| \mathcal {N} (i) |} \sum_ {q \in \mathcal {N} (i)} \log \frac {\exp \left\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {q} \right\rangle / \tau}{\sum_ {n} ^ {n \neq i} \exp \left\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {n} \right\rangle / \tau}.
$$
where:  $\pmb{z}_i = h\circ \Phi (\pmb{x}_i)$ ;  $h$  is a projection head, which is used during training and discarded afterwards; and  $\tau$  is a temperature value.  $\mathcal{N}(i)$  represents the indices of images in the labeled subset of the batch,  $\mathcal{B}_{\mathcal{L}}\in \mathcal{B}$ , which belong to the same category as  $\pmb{x}_i$ . Given a weighting coefficient,  $\lambda_{1}$ , the total contrastive loss on the model's features is given as:
$$
\mathcal {L} _ {\text {f e a t}} = \left(1 - \lambda_ {1}\right) \sum_ {i \in \mathcal {B}} \mathcal {L} _ {\text {f e a t}, i} ^ {u} + \lambda_ {1} \sum_ {i \in \mathcal {B} _ {\mathcal {L}}} \mathcal {L} _ {\text {f e a t}, i} ^ {s} \tag {4}
$$
(2) Perform classification by embedding all images with the trained backbone,  $\Phi$ , and apply semi-supervised  $k$ -means (SS- $k$ -means) clustering on the entire dataset,  $\mathcal{D}_{\mathcal{U}} \bigcup \mathcal{D}_{\mathcal{L}}$ . SS- $k$ -means is identical to unsupervised  $k$ -means [57] but, at each iteration, instances from  $\mathcal{D}_{\mathcal{L}}$  are always assigned to the 'correct' cluster using their labels, before being used in the centroid update. In this way, the cluster centroid updates for labelled classes are guided by the labels in  $\mathcal{D}_{\mathcal{L}}$ .
SimGCD [19]. Train a backbone representation,  $\Phi$ , and a linear head,  $g$ , to classify images amongst the  $k$  classes in the dataset, yielding a model  $f_{\theta} = g \circ \Phi$ . Train the backbone jointly with the feature space loss from eq. (4), and with linear classification losses based on the output of  $g$ .
(1) Generate pseudo-labels for an image,  $\pmb{x}_i$ , as  $\pmb{p}_T(\pmb{x}_i) \in [0,1]^k$ , in order to train the classifier,  $f_{\theta}$ . Infer pseudo-labels on all images in a batch,  $\mathcal{B}$ , and compute an additional supervised cross-entropy loss on the labelled subset,  $\mathcal{B}_{\mathcal{L}}$ .
- Pass two views of an image to the same model. Each view generates a soft pseudo-label for the other, for instance as:
$$
\boldsymbol {p} _ {T} \left(\boldsymbol {x} _ {i}\right) = \operatorname {s g} \left[ \operatorname {s o f t m a x} \left(f _ {\theta} \left(\boldsymbol {x} _ {i} ^ {\prime}\right); \tau_ {T}\right) \right] \tag {5}
$$
Here  $\mathrm{sg}$  is the stop-grad operator and  $\tau_T$  is the pseudo-label temperature.
- Compute model predictions as  $\pmb{p}_S(\pmb{x}) = \mathrm{softmax}(f_\theta(\pmb{x}); \tau_S)$  and a standard pseudolabelling loss [14, 18, 26] (i.e. soft cross-entropy loss) as:
$$
\mathcal {L} _ {c l s} ^ {u} (\theta ; \mathcal {B}) = - \frac {1}{| \mathcal {B} |} \sum_ {\boldsymbol {x} _ {i} \in \mathcal {B}} \left\langle \boldsymbol {p} _ {T} \left(\boldsymbol {x} _ {i}\right), \log \left(\boldsymbol {p} _ {S} \left(\boldsymbol {x} _ {i}\right)\right) \right\rangle + \left\langle \boldsymbol {p} _ {T} \left(\boldsymbol {x} _ {i} ^ {\prime}\right), \log \left(\boldsymbol {p} _ {S} \left(\boldsymbol {x} _ {i} ^ {\prime}\right)\right) \right\rangle \tag {6}
$$
Temperatures are chosen such that  $\tau_T < \tau_S$  to encourage confident pseudo-labels [14].
- Optimize the model,  $f_{\theta}$ , jointly with: the pseudo-label loss (eq. (6)) and  $\mathcal{L}_{feat}$  (see eq. (4)). The model is further trained with: the standard supervised cross-entropy loss on the labelled subset of the batch,  $\mathcal{L}_{cls}^{s}(\theta; \mathcal{B}_{\mathcal{L}})$ ; and an entropy regularization term,  $\mathcal{L}_{cls}^{r}(\theta)$ :
$$
\mathcal {L} _ {c l s} ^ {s} (\theta ; \mathcal {B} _ {\mathcal {L}}) = - \frac {1}{| \mathcal {B} _ {\mathcal {L}} |} \sum_ {i \in \mathcal {B} _ {\mathcal {L}}} \langle \boldsymbol {y} (\boldsymbol {x}), \log (\boldsymbol {p} _ {S} (\boldsymbol {x})) \rangle , \quad \mathcal {L} _ {c l s} ^ {r} (\theta) = - \langle \bar {\boldsymbol {p}} _ {S}, \log (\bar {\boldsymbol {p}} _ {S}) \rangle , \bar {\boldsymbol {p}} _ {S} = \frac {1}{| \mathcal {B} |} \sum_ {\boldsymbol {x} \in \mathcal {B}} \boldsymbol {p} _ {S} (\boldsymbol {x})
$$
Here,  $\pmb{y}(\pmb{x})$  is a ground-truth label and, given hyper-parameters  $\lambda_{1}$  and  $\lambda_{2}$ , the total loss is defined as:  $\mathcal{L}(\theta; \mathcal{B}) = (1 - \lambda_{1})(\mathcal{L}_{cls}^{u}(\theta; \mathcal{B}) + (\mathcal{L}_{feat}^{u}(\theta; \mathcal{B})) + \lambda_{1}(\mathcal{L}_{cls}^{s}(\theta; \mathcal{B}_{\mathcal{L}}) + \mathcal{L}_{feat}^{s}(\theta; \mathcal{B}_{\mathcal{L}})) + \lambda_{2}\mathcal{L}_{cls}^{r}(\theta)$ .
$\mu$ GCD (Ours). Train a backbone representation,  $\Phi$ , and a linear head,  $g$ , to classify images amongst the  $k$  classes in the dataset, yielding a model  $f_{\theta_T} = g \circ \Phi$ . Train the backbone first with the feature space loss from eq. (4), and then with linear classification losses based on the output of  $g$ .
(1) Train a backbone  $\Phi$  using Step (1) from the GCD baseline algorithm.
(2) Append a classifier,  $g$ , to the backbone and duplicate it to yield two models. One model (a teacher network,  $f_{\theta_T}$ ) is used to generate pseudo-labels for a student network,  $f_{\theta_S}$ , as  $\pmb{p}_T(\pmb{x}_i) \in [0,1]^k$ . Infer pseudo-labels on all images in a batch,  $\mathcal{B}$ , and compute an additional supervised cross-entropy loss on the labelled subset,  $\mathcal{B}_{\mathcal{L}}$ . The student and teacher networks are trained as follows:
- Generate a strong augmentation of an image,  $x_{i}$ , and a weak augmentation,  $x_{i}'$  [49]. Pass the weak augmentation to the teacher to generate a pseudo-label and construct a loss:
$$
\boldsymbol {p} _ {T} \left(\boldsymbol {x} _ {i}\right) = \operatorname {s g} \left[ \operatorname {s o f t m a x} \left(f _ {\theta_ {T}} \left(\boldsymbol {x} _ {i} ^ {\prime}\right); \tau_ {T}\right) \right] \quad \mathcal {L} _ {c l s} ^ {u} \left(\theta_ {S}; \mathcal {B}\right) = - \frac {1}{| \mathcal {B} |} \sum_ {\boldsymbol {x} _ {i} \in \mathcal {B}} \left\langle \boldsymbol {p} _ {T} \left(\boldsymbol {x} _ {i}\right), \log \left(\boldsymbol {p} _ {S} \left(\boldsymbol {x} _ {i}\right)\right) \right\rangle \tag {7}
$$
- Optimize the student's parameters,  $\theta_S$ , with respect to: the pseudo-label loss from eq. (7); the supervised loss,  $\mathcal{L}_{cls}^s$ ; and the entropy regularization loss,  $\mathcal{L}_{cls}^r$ . Formally, the 'student',  $f_{\theta_S}$ , is optimized for:  $\mathcal{L}(\theta_S; \mathcal{B}) = (1 - \lambda_1) \mathcal{L}_{cls}^u(\theta_S; \mathcal{B}) + \lambda_1 \mathcal{L}_{cls}^s(\theta_S; \mathcal{B}_{\mathcal{L}}) + \lambda_2 \mathcal{L}_{cls}^r(\theta_S)$ .
- Update the teacher network's parameters with the Exponential Moving Average (EMA) of the student network [17]. Specifically, update the 'teacher' parameters,  $\theta_T$ , as:
$$
\theta_ {T} = \omega (t) \theta_ {T} + (1 - \omega (t)) \theta_ {S}
$$
where  $t$  is the current epoch and  $\omega (t)$  is a time-varying decay schedule.
At the end of training, the 'teacher',  $f_{\theta_T}$ , is used for evaluation.