The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image
Authors: Yuki M Asano
Aaqib Saeed
Venue: ICLR 2023 poster (2023)
Keywords: Augmentations
Single Image Learning
Distillation

Abstract: What can neural networks learn about the visual world when provided with only a single image as input? While any image obviously cannot contain the multitudes of all existing objects, scenes and lighting conditions -- within the space of all  $256^{3\cdot224\cdot224}$ possible $224$-sized square images, it might still provide a strong prior for natural images. To analyze this ``augmented image prior''  hypothesis, we develop a simple framework for training neural networks from scratch using a single image and augmentations using knowledge distillation from a supervised pretrained teacher. With this, we find the answer to the above question to be: `surprisingly, a lot'. In quantitative terms, we find accuracies of $94\%$/$74\%$ on CIFAR-10/100, $69$\% on ImageNet, and by extending this method to video and audio, $51\%$ on Kinetics-400 and $84$\% on SpeechCommands. In extensive analyses spanning 13 datasets, we disentangle the effect of augmentations, choice of data and network architectures and also provide qualitative evaluations that include lucid ``panda neurons'' in networks that have never even seen one. 

Published as a conference paper at ICLR 2023
THE AUGMENTED IMAGE PRIOR: DISTILLING 1000 CLASSES BY EXTRAPOLATING FROM A SINGLE IMAGE
Yuki M. Asano
Aaqib Saeed"
Video & Image Sense Lab University of Amsterdam
Department of Industrial Design Eindhoven University of Technology
0. Source datum
Strong augmentations
Knowledgee distillation
UL
Iageheet cheset:
1. Tench,
Pretrsined Teacher
2: Goleftah.
1000 to paper
Studeent
Oram scratch))
KT
Figure I: Extrapolating from one image. Strongly augmented patches from a single image are used to train student (S) to distinguish semantic classes, such as those in ImageNet. The student neural network is initialized randomly and learns from a pretrained teacher (T) via KL-divergence. Although almost none of target categories are present in the image, we find student performances of > 69% for classifying ImageNet's 1000 classes. In this paper, we develop this single datum learning framework and investigate it across datasets and domains.
ABSTRACT
What can neural networks learn about the visual world when provided with only a single image as input? While any image obviously cannot contain the multi- tudes of all existing objects, scenes and lighting conditions within the space of all 256 224-224 possible 224-sized square images, it might still provide a strong prior for natural images. To analyze this "augmented image prior" hypothesis. we develop simple framework for training neural networks from scratch using a single image and augmentations using knowledge distillation from supervised pretrained teacher. With this, we find the answer to the above question to be: "surprisingl, a lot', In quantitative terms, we find accuracies of 94%/74% on CIFAR-10/100, 69% on ImageNet, and by extending this method to video and audio, 51% on Kinetics-400 and 84% on SpeechCommands. In extensive anal- yses spanning 13 datasets. we disentangle the effect of augmentations, choice of data and network architectures and also provide qualitative evaluations that in- clude lucid "panda neurons" in networks that have never even seen one. Code: https:// //si:///inggeeee/ -image-aa--------- Ristill github. /i/
INTRODUCTION
Deep learning has both relied and improved significantly with the increase in dataset sizes. In turn, there are many works that show the benefits of dataset scale terms of data points and modalities used. Within computer vision, these models trained on larger datasets, such as Instagram- 1B (Mahajan et al., 2018) or JFT-3B (Dosovitskiy et al., 2021), have been shown to successfully distinguish between semantic categories at high accuracies. In stark contrast to this, there is little research on understanding neural networks trained on very small datasets. Why would this be of any interest? While smaller dataset sizes allow for better understanding and control of what the model is being trained with, we are most interested in its ability to provide insights into fundamental aspects of learning: For example, is an open question as to what exactly required for arriving at semantic
"Equal contribution.

---

Published as a conference paper at ICLR 2023
visual representations from random weights, and also of how well neural networks can extrapolate from their training distribution.
While for visual models it has been established that few no real images are required for arriving at basic features, like edges and color-contrasts (Asano et al., 2020; Kataoka et al., 2020; Bruna & Mallat, 2013; Olshausen & Field, 1996), we go far beyond these and instead ask what the minimal data requirements are for neural networks to learn semantic categories, such as those of ImageNet. This approach is also motivated by studies that investigate the early visual development in infants, which have shown how little visual diversity babies are exposed to in the first few months whilst de- veloping generalizeable visual systems (Orhan al., 2020; Bambach et al., 2018). this paper, we study this question in its purest form, by analyzing whether neural networks can learn to extrapolate from a single datum.
However, addressing this question naively runs into the difficulties of i) current deep learning meth- ods, such as SGD or BatchNorm being tailored to large datasets and not working with single datum and ii) extrapolating semantic categories requiring information about the space of natural images beyond the single datum. this paper, we address these issues by developing a simple framework that recombines augmentations and knowledge distillation.
First, augmentations can be used to generate a large number of variations from a single image. This can effectively address issue i) and allow for evaluating the research question standard architec- tures and datasets. This use of data augmentations to generate variety is drastically different their usual use-case in which transformations generated to implicitly encode desirable invariances during training.
Second, to tackle the difficulty of providing information about semantic categories in the single datum setting, we opt to use the outputs of a supervisedly trained model in a knowledge distillation (KD) fashion. While KD (Hinton et al., 2015) is originally proposed for improving small models' performance by leveraging what larger models have learned, we re-purpose this as simple way to provide a supervisory signal about semantic classes into the training process.
We combine the above two ideas and provide both student and teacher only with augmented versions of single datum, and train the student to match the teacher's imagined class pressspredistions of classes almost all of which are not contained in the single datum, see Fig. 1. While practical applications do result from our method for example we provide results on single image dataset based model compression in the Appendix- our goal in this paper is analyzing the fundamental question of how well neural networks trained from a single datum can extrapolate to semantic classes, like those of CIFAR, SpecchCommands ImageNet or even Kinetics.
What we find is that despite the fact that the resulting model has only seen a single datum plus augmentations, surprisingly high quantitative performances are achieved: e.g. 74% on CIFAR-100. 84% on SpeechCommands and 69% top-1, single-crop accuracy on ImageNet-12 12. We further make the novel observations that method benefits from high-capacity student and low-capacity teacher models, and that the source datum's 's characteristics matter - random noise or less dense images yield much lower performances than dense pictures like the one shown in Figure 1.
In summary, in this paper we make these four main contributions:
1. A minimal framework for training neural networks with single datum using distillation.
2. Extensive ablations of the proposed method, such as the dependency on the source image, augmentations and network architectures.
3. Large scale empirical evidence of neural networks' ability to extrapolate > 12 vision and audio datasets.
4. Qualitative insights on what and how neural networks trained with a single image learn.
2
RELATED WORK
The work presented builds top of insights from the topics of knowledge distillation and single- and no-image training of visual representations and yields insights into neural networks' ability to extrapolate.
2

---

Published as a conference paper at ICLR 2023
Distillation. In the original formulation of knowledge distillation (KD), the goal is to train typ- ically lower capacity ("student") model from a pretrained ("teacher") model in order to surpass the performance of solely training with label-superviscd objective. KD has also been explored exten- sively to train more performant and, or compressed student models from the soft-target predictions of teacher models (Ba & Caruana, 2014; Hinton et al., 2015; Gou et al., 2021; Furlanello et al., 2018; Czarnecki et al., 2017). Similarly, other approaches have also been developed to improve transfer from teacher student, including sharing intermediate layers' features (Romero et al., 2014), spatial attention transfer (Zagoruyko & Komodakis, 2016b), similarity preservation between activations of the networks (Tung & Mori, 2019). contrastive distillation (Tian et al., 2020a) or few-shot distilla- tion (Li et al., 2020). KD has also been shown to be an effective approach for learning from noisy labels (Li et al., 2017). More recently, Beyer et al. (2022) conducted a comprehensive empirical investigation to identify important design choices for successful distillation from large-scale vision models. In particular, they show that long training schedules, paired with consistent augmentations (including MixUp) for both student and teacher, result in better performances.
Without the training data, Distilling knowledge without access to the original training dataset was originally proposed in 2017 (Lopes et al., 2017) and first experiments of leaving out single MNIST classes from the training data were shown in (Hinton et al., 2015). Yet, this paradigm is gaining importance many recent advancements have been made possible due to extremely large proprictary datasets that are kept private. This has lead to either the sole release of trained models (Radford et al., 2021; Ghadiyaram et al., 2019; Mahajan et al., 2018) even more restricted access to only the model outputs via APIs (Brown et al., 2020). While the original work (Loopes et al., 2017), still required activation statistics from the training dataset of the network, more recent works do not require this "meta-data". These approaches are typically generation based (Chen et al., 2019; Ye et al., 2020; Micaelli & Storkey, 2019; Yin al., 2020) and e.g. yield datasets of synthetic images that maximally activate neurons in the final layer of teacher. Related to this, there are works which conduct "dataset" distillation, where the objective is to distill large-scale datasets into much smaller ones, such that models trained on it reach similar levels of performance as on the original data. These methods generate synthetic images (Wang et al., 2018; Radosavovic et al., 2018; Liu et al., 2019; Zhao & Bilen, 2021: Cai et al., 2020), labels (Bohdal et al., 2020), or both (Nguyen ct al., 2021), but due to their meta-learning nature have only been successfully applied to small- scale datasets. In contrast to GAN- and iveesion-bbsed methods, as well as to dataset distillation, our approach does not require the knowledge of the weights and architecture of the teacher model, and instead works with black-box "APT"-style -tyyle teacher models and much smaller "datasets" of just a single datum plus augmentations. This paper follows a radically different goal: We are interested in analysing the potential of extrapolating from single image the manifold of natural images, for which we choose KD as a well-suited and simple tool.
Prior knowledge in deep learning. Finally, this work is related several works which have analysed the infusion of prior knowledge to the training process. For example the prior knowledge contained in neural network architectures (Jarrett et al., 2009; Zhou et al., 2019; Sreenivasan al., 2021; Kim et al., 2021; Back et al., 2021; Ulyanov et al., 2018) or image augmentations (Xiao et al., 2020). In this work, we instead view single image as "prior" for all other natural images, which to the best of our knowledge has not been explored.
3 METHOD
We believe that simplicity the key to demonstrating and analysing the question of how far a single image can take us. We are inspired by recent work of Asano et al. (2020), which "patchifies" single-image using augmentations, and the knowledge distillation method presented by Beyer et al. (2022) and our technical contribution lies in unifying them to single- -inage distillation framework.
i. Dataset generation. In (Asano et al., 2020), a single "source" image augmented many times to generate a static dataset of fixed size. This is done by applying the following augmentations in sequence: cropping, rotation and shearing, and color jittering. For this, we follow their official implementation Asano et al. (2020), and do not change any hyperparameters. We do however ana- lyze the choice of source images more thoroughly by additionally including a random noise, and a Hubble-telescope image to our analysis. In addition, also conduct experiments on audio classifi- cation. For generating a dataset of augmented audio-clips, we apply the set of audio-augmentation from (Bitton& & Papakipos, 2021), which consist of operations, such as random volume increasing.
3

---

Published as a conference paper at ICLR 2023
Table 1: Distilling dataset. 'mmaee augmentations
Table datasets. Teacher model is WideR40- and student WideR40-1 as (Mi-
Comparison other
almost 50K in-domain CIFAR-10/100 images. Here, both teacher and student are WideR40-4 networks.
caelli & Storkey, 2019).
Distillation dataset
Accuracy
Data
C10
Name
# Images
Pixels
Size (MB)
C10
C100
CIFAR-10
92,61
CIFAR-10
50K
51M
145
95.26
76.29
Fractals
33.26
CIFAR-100
50K
51M
145
94.51
78.06
StyleGAN
83,42
CIFAR-10
10K
10M
29
94.58
72.95
ZeroSKD
86.60
Ours
1
2.8M
0.27
94.14
73.80
Ours
89.27
background noise addition and pitch shifting to yield log-Mel spectrograms from raw waveforms (see Appendix for complete details).
ii. Knowledge distillation. The knowledge distillation objective is proposed in (Hinton et al., 2015) to transfer the knowledge of pretrained teacher to a lower capacity student model. In this case, the optimization objective for the student network is weighted combination of dual losses: standard supervised cross-entropy loss and a "distribution-man maatiiinnmmmmmnn" objective that aims to mimic the teacher's s output. However, in our case there are class-labels for the patches generated from a single image, so we solely use the second objective formulated as a Kullback-Leible (KL) diver- gence between the student output p" and the teacher's output p":
CKL
-p'logg + +p' log CEC
where are the teachers' classes and the outputs of both student and teacher are temperature flattened probabilities, p so soflt)) that are generated from logits
For training, we follow (Beyer et al., 2022) in employing function matching strategy, where the teacher and student models are fed consistently augmented instances, that include heavy augmen- tations, such MixUp (Zhang et al., 2018) or CuiMix (Yun et al., 2019). However, in contrast to (Beyer et al., 2022), we neither have access to TPUs nor can train 10K epochs on ImageNet-sized datasets. While both of these would likely improve the quantitative results,
believe that this handicap is actually blessing in disguise: This means that the results we show in this paper are not specific to heavy-compute, extremely large batch-sizes, but instead are fundamental.
4
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Source data. For the source data, we utilize the single images of (Asano et al., 2020), except for one image, which replace by similar one of sizes up to 2,560x1,929 (the "City" image of Fig. 1). For the audio experiments, we use two short audio-clips from Youtube, 5mins BBC newselip, well languages. Images and audio visualizations, sources and licences are provided in Appendix A.
could not retrieve its licence. These images
5mins clip showing 11 Germanic
Tasks. For simplicity, we focus on classification tasks here and provide results on sample appli- cation of single-image based model compression in Appendix 10 For our ablations and small- scale experiments we focus on CIFAR-10/100 (Krizhevsky ments using 224 x 224 sized images, we evaluate our method on datasets with varying number of classes (see Table 7) and additionally conduct experiments in the audio domain (see Table 5) and video (see Table 6). Further implementation details are provided in Appendix C.
al., 2009). For larger-scale experi-
4.2 ABLATIONS
1-image vs full training set. We first examine the capability to extrapolate from one image to small-scale datasets, such as CIFAR-10 10 and CIFAR-100. In Table 1, we compare various datasets for distilling a teacher model into a student. We find that while distillation using the source dataset always works best (95.26% and 78.06%) CIFAR-10/100, using single image can yield models
4

---

Published as a conference paper at ICLR 2023
Table 3: Ablations of single image distillation. We analyze key components in our experimental setup. We report top-1 accuracies for CIFAR-10/100. For the teacher we use WideR40-4 and for the student WideR16-4 and train for 1K epochs.
Distillation dataset
Accuracy C10 C100
Accuracy C10
Accuracy Hilip. RCrop. MixUp CutMix C10 C100
Augmentations
Image # Pixels
Signal
C100
"Noise"
4.9M 69.30 19.50
Full
93.32
68.69
89.34 55.05
"Universe"
4.8M 88,18 39.68
Top-5 92.98
64.72
91.03 58.24
"Bridge"
1.1M 92.24 57.87
Argmax
91.89
60.75
92.86 64.26
"City"
4,9M 93.13 64.85
(b) Teacher signal. Even with only top-5, hard distillation, performance only slightly degrades.
92.41 63.50
"Animals"
2.8M 93.28 66.12
93.32 68.69
(a) Distilling image. Content of source image matters.
(c) Varying Augmentation.
More helps,
CutMix is important.
R-squared
OH4I 116
0.00
0.05
0110
0.15
0.00
Size (MB)
Figure 2: Varying source images for CIFAR-100 distillation. Setting as in 3a.
which almost reach this upper bar (94.1% and 73.8%). Moreover, we find that one image distillation even outperforms using 10K images of CIFAR-10 when teaching CIFAR-100 classes even though these two datasets are remarkably similar. To better understand why using a single image works, we next disentangle the various components used in the training procedure: (a) the source image, (b) the generated image dataset size and (c) the augmentations used during distillation, corresponding Tables 3a and 3c.
Choice of single image. From Table 3a, we find that the choice of source image content is cru- cial: Random noise or sparser images perform significantly worse compared to the denser "City" and "Animals" pictures. We compare 23 further images in Fig. 2 and find that distillation quality roughly correlates with the density in images and JPEG sizes. This is in contrast to self-supervised pretraining of (Asano et al., 2020), suggesting that the underlying mechanisms are different.
Varying loss functions. In Table 3b, we show that the student can learn even with much degraded leamning signals. For example even
receives only the top-5 predictions or solely the argmax prediction (i.e. hard label) of the teacher without any confidence value, the student is still able extrapolate at a significant level (> 91% 01%/60% /60% for CIFAR- 10/100). Even ImageNet scale, the performance remains at high 43.8% top-1 accuracy (see Table row (m)). This also suggests that copying models solely from API outputs is possible, similar to (Orekondy al., 2019). We also evaluated LI and L2 distillation functions (see Table 9 for details) and found that these perform slightly worse. This echoes the finding of (Tian et al., 2020a) that standard KD loss of Eq. (1) is actually strong baseline, hence
this in the rest of our experimental analysis.
Augmentations. In Table 3c, we ablate the augmentations we use during knowledge distillation. Besides observing the general trend of "more augmentations are better". we find that CuMMix per- forms better than MixUp on our single-image distillation task. This might be because in our case the model tasked with learning how to extrapolate towards real datapoints, while MixUp is derived and therefore might be more useful for interpolating between (real) data samples (Zhang et al., 2018; Beyer et al., 2022).
Comparison to synthetic datasets. In Table 2, we find that there is something unique about using a single image, as our method outperforms several synthetic datasets, such as FractalDB Katuoka et al. (2020), randomly initialized StyleGAN Baradad et al. (2021), well the GAN-based ap-
5

---

Published as a conference paper at ICLR 2023
Table 4: Distilling various architectures on CIFAR-10/100. We compare student accuracy when distilling with full training set vs our 1-image dataset
Distillation Ours
Teacher
Acc.
Student
Acc-
Full
A 5%
CIFARIOU CIFARIO
VGG-19
93.28
VGG-16
92.42
92.84
92.14
ResNet-56
93.77
ResNet-20
92.32
92.29
90.70
WideR40-4
95:42
W6dcR16-4
95.20
95.00
93.32
WideR40-4
95.42
WideR40-4
95.42
94.36
94.14
WideR 16-4
95.20
WideR40-4
65.42
94.30
94.02
VGG-19
70.79
VGG-16
73.26
71.19
58.66
X
ResNet-56
70.99
ResNet-20
65.74
67.04
52.43
X
WideR40-4
78.14
WideR16-4
75.56
76.26
68.69
WideR40-4 78.14
WideR40-4
78.14
75.54
73.80
WideR16-4 78.14
WideR40-4 75.56
76.29
74.08
Table 5: Distilling audio representations. audio clip + augmentations provide stronger supervi- sory signal for the student to distill teacher's knowledge.
Distillation
Dataset
Categories (#Classes)
Teacher
Full
Ours
Ours
MUSAN (Snyder et al., 2015)
Generic sounds (3)
98.76
96.78
89.85
96.28
Voxforge (MacLean, 2018)
Languages (6)
91.13
89.04
72.85
78.47
Speech Commands (Warden, 2018)
Keywords (12)
95.15
94.86
82.90
84.19
LibriSpeech (Panayotov et al., 2015)
Speakers (100)
99.89
99.54
78.67
84.12
proach of (Micaelli & Storkey, 2019). This is despite the fact that these synthetic datasets contain 50K images. We provide insights on the characteristics of this 1-image dataset in Section 4.4.
Next, using the insights gained this section, we scale the experiments towards other network architectures, dataset domains and dataset sizes.
4.3 GENERALISATION
Architectures In Table 4, we experiment with different common architectures on CIFAR-10 and CIFAR-100. On CIFAR-10, for the case of ResNet-56 to ResNet-20 which could be attributed to the lower capacity of the stu- dent model. We find that the distillation performance of WideR40-4 WideR40-4 on CIFAR-10 is very close to distilling from original source data; achieving accuracy of 94.14% and only around one-percentage point lower than supervised training. This analysis shows that our method gener- alizes to other architectures hints that larger capacity student models might be important for high performances.
find that almost all architectures perform similarly well, except
Extension audio. So far, our proposed approach has shown success in distilling useful knowl- edge for images. To further test the generalizability of our approach on other modalities, we conduct experiments on several audio recognition tasks of varying difficulty. We perform distillation via 50K randomly generated short audio clips from two (i.e. Ours and Oursæ), 5mins YouTube videos (see Appendix for more details). In Table 5, we compare the results against the performance of the teacher model and distilling directly using source dataset. We find that even for audio, distill- ing with merely a single audio clip's data provides enough supervisory signal to train the student model to reach above 80% accuracy in the majority of the cases. In particular, we significant improvement in distillation performance when a single audio has a wide variety of sounds to boost accuracy challenging Voxforge dataset from 72.8% 78.4%. Our results on audio recognition tasks demonstrate the modality agnostic nature of our approach and further highlight the capability to perform knowledge distillation in the absence of large amount of data.
Extension to video. We conduct further experiments video action recognition tasks. As dis- tillation data, we generate simple pseudo-videos of 12 frames that show a linear interpolation of two crop locations taken at the beginning and end for the City image and X3D video archi-

---

Published as a conference paper at ICLR 2023
Table 6: Scaling to video. Student models trained from seratch. Top-1 accuracy is shown with 10- temporal center-cropped clips per video. Details in Appendix B.1
Model Type
Trainnn
2000
Studem: (CFAB)
6000
Evaluation
Tescher (Patches)
4000
Teacher (CFAR)
2000
Softmax scores
Dataset
Teacher
Distill
Figure 3: Output confidence scores. Temperature- scaled softmax scores of the predicted classes for 5K patches and the 5K CIFAR-10 validation set images.
UCF-101
90.4
75.2
K400
67.8
51.8
Table 7: Larger scale datasets. Teacher is ResNet-15 18 and student is varied between ResNet-50 and ResNet-50x2. Students are trained from scratch. Gap to teacher in brackets for ResNet-50x2.
1-Image- Minage-Distill +R50x2
Dataset
Categories (#Classes)
Teacher (R18)
-R18
STL-10 (Coates al., 1011)
Generic objects (10)
96.3
93.9
95.0 (-1.3)
ImageNet- 100 (Tian al., 2020b)
Objects (100)
89.6
84.4
88.5 (-1.1)
Flowers (Nilsback & Zisserman, 2006)
Flower types (102)
87.9
81.5
83.8 (-4.1)
Places (Zhou et al., 2017)
Scenes (365)
54.0
50,3
53.1 (-0.9)
ImageNet (Deng et al., 2009)
Objects (1000)
69.5
66.2
69.0 (-0.5)
tectures (Feichtenhofer, 2020) as teachers and students, see Appendix B.1 for further details. In Table 6, we find that our approach indeed generalizes well to distilling video models with perfor- mances of 75.2% and 51.8% on UCF-101 (Soomro et al., 2012) and K400 (Carreira & Zisserman, 2017)- the fact that the source data not even real video.
4.4 SCALABILITY TO LARGE-SCALE DATASETS AND MODELS
From this section on, we scale our experiments to larger models utilizing x24x224-sized images, and evaluate these various vision datasets.
Varying datasets. In Table 7 we find that, overall, a single image not enough to fully recover the performance
difficult datasets. This might be because of the fine-grained nature of these datasets, e.g. ImageNet includes more than 120 kinds dogs and Flowers contains 102 types of flowers. Nevertheless, we find a surprisingly high accuracy of 69% on ImageNet's validation set, even though this dataset comprises 1000 classes, and the student only having seen heavily augmented crops of single image. Table 8, we conduct further analyses on distilling ImageNet models.
Patchification more data. At the top part of the Table 8, analyse the effect of using original vs patchified (p) random images from the training set of ImageNet. We find that while for 10 and 100 source images, patchification improves performance, this is not the case when larger number of images is used. This indicates that increased diversity is especially a crucial component for the small data regime, while capturing larger parts of objects becomes important only beyond this stage. We also make the surprising observation that patches from a single high-quality image ("City") obtain better performances than patches generated from 1000 ImageNet training images. To understand why this might be the case, we note that the high coherency of training patches from single image strikingly resembles what babies see during their early visual development, e,g., looking at only few toys and people but from many angles (Bambach et al., 2018; Orhan et al., 2020). It is hypothesized that this unique combination of coherence and variability is ideal for learning to recognize objects (Orhan et al., 2020).
Teacher/student architectures. In Fig. 4, we analyze the performances varying ResNet models by depth width. We find that overall varying the width is more parameter-efficiet way to obtain higher performances, almost reaching the teacher's 69.5% performance with a ResNet-50x2 69.0%.

---

Published as a conference paper at ICLR 2023
Table 8: IN-1K distillations. We vary distillation dataset and teacher/student configurations. '-' refers to experiments continued due to compute cost outweighing insights. (p)
70
means images are patchified. <t' refers to models trained with half the batch size due to memory constraints. 'AM'
refers to only using the argmax as teaching signal.
60
IN-1K val Accuracy
Setting
Epochs
Images
Teacher
Student
10
20
30
50
200
(a) 10
R50
R50
7.6 13.3
(b) 10 (p)
R50
- R50
17.2 27.3
(c) 100
R50
R50
14.7 25.1 32.3
(d) 100 (p)
R50
R50
23.6 36.1 42.8
1000
R50
R50
38.0 52.4 57.4 62.5
30
D450020.25,030000 0.1.5.0.
1000 (p)
R50
R50
27.1 39.4 45.2 50.9
Teecher (RN-18)
10
Noise (p)
R18
R50
0.1 0.1 0.1
Number of parameters (M)
Bridge (p)
R18
R50
21.2 34.8 40.0
City (p)
R18
+ R50
34.5
47.0
52.2 56.8 66.2
Figure 4: Effect of student archi- tecture. We vary the architecture of the student by depth and width. Teacher is ResNet-18, settings as Table with half batch size.
City (p)
R101
+ R101
22.4 31.4
(k) City (p)
R50"
R50
4.6 6.6
City (p)
R50
R50
18.0 34.0 39.8 45.6 55.5
(m) City (p)
R18 (AM)
R50x2
12.5 20.7 24.4
30.2 43.8
(n) City (p)
R18
R50x2
46.1 55.6
59.7
63.1 69.0
In addition, in the bottom half of Table 8 (rows g-n), we find surprising results: First, in contrast to normal KD, we find that the teacher's performance is not directly related to the final student performance, e.g. ResNet-50 is less well-suited for distilling than a ResNet-18 18 (rows i vs /).
Second, we find that the choice of source image is even more important for ImageNet classification (see rows g-i), switching from the "City" even to the "Bridge" one decreases performance siderably and the Noise image does not train at all. Third, we find that the best settings are those in which the student's capacity is higher than that of the teacher (rows i,n and Fig. 4). In fact this R18->R002 setup (row n) obtains a performance on ImageNet-12 of 69.0% which matches the teacher's performance within 0.5%. In Appendix B.3, we further compare distillations to ViT archi- tectures (Dosovitskiy et al., 2021) 1), but generally find lower performances (up to 64%), showing that the inductive biases of CNNs might positively aid in learning.
4.5 ANALYSIS
To better understand how our method achieving these performances, we analyse the models.
Output confidence scores. In Fig. 3, visualize the distribution of the confidence scores of the predicted classes for student and teacher models for the training and evaluation data. We observe that the student indeed learns mimic the teacher's predictions well for the patch training data. However, during evaluation on CIFAR images, the student model exhibits much broader distribu- tion of values, some with extremely high confidence scores. This might indicate that the student has Reeatur analasis. Ne nad whe.aeeeeaaaaiiiaaaaaaaa
whriigrrrtiiiiiigaaaaalccoo (ghier donfidenice viluuetton, 2008) using the features of 5K training patches and 5K test set images in Fig. 5. We find that, as expected, the training patches mapped to individual CIFAR classes do resemble real counter parts. Moreover, we observe that all the training patches are embedded close to each other in the center, while CIFAR images are clustered towards the outside. This shows that the network is indeed learning features that are well suited for effective extrapolation.
laee
stimpggeded whllsdstheewweeeddaaaae
Neuron visualizations. We next analyze a single-image distilled model trained to predict IN-1k classes. In Fig. 6, visualize four final-layer neurons using the Lucid (Olah et al., 2017) library. When we compare neurons of the standard ImageNet supervised and our distilled model, we find that the neurons activate for very similar looking inputs. The clear neuron visualizations for "panda"
8

---

Published as a conference paper at ICLR 2023
Figure 5: Feature space visualization. t-SNE of jointly embedding random patches (-) and CIFAR-10 test set (x) instances with our model distilled to clas- sify semantic classes. We also show example im- ages from patches and actual validation set images (enclosed in box). Note how most training images are contained within a small region while the network needs extrapolate for the real images which occupy the outer regions in this plot. Best viewed zoomed-in
Airplane
Deer
Horse
Automobile
Dog
Ship
Bird
Truck
Cat
Harp
Monaacc butterfly
Panda
Lifeboat
Figure 6: Visualizing neurons via activation maximisation final layer neurons. We compare an ImageNet su- pervisedly trained model against our model which has been trained with sin- gle image distillation. Best viewed zoomed-in. We provide further exam- ples in Appendix B.6.
ImageNet
supervised
1-image distilled
or "lifeboat" (and more are provided in Appendix B.6) are especially surprising since this network was trained using only patches from the City Image, and has never seen any of these objects during its learning phase.
5 DISCUSSION
The augmented image prior for extrapolation. The results in this paper suggest summary formulation as follows: Within the space of all possible images I, single real image x E I and its augmentations A(x) can provide sufficient diversity for extrapolating to semantic categories in real images.
Limitations. In this paper, we were mainly concerned with showing that extrapolating from single image works, empirically. Due to the limited nature of our compute resources, we have not exhaustively analyzed the choice of initial image or patchification augmentations, and longer training (as shown in (Beyer et al., 2022)) would further improve performances.
Potential negative societal impact. While research question is of fundamental nature, one possible negative impact could be that the method used to steal models and thus intellectual property from API providers although in practice the performance varies especially for large-scale datasets (see Table 7).
Conclusion and outlook. In this work, we have analyzed whether it is possible to train neural net- works to extrapolate to unseen semantic classes with the help of supervisory signal provided by pretrained teacher. Our quantitative and qualitative results demonstrate that our novel single-image knowledge distillation framework can indeed enable training networks from scratch to achieve high accuracies on several architectures, datasets and domains. This demonstrates that knowledge dis- tillation can be done with just single image plus augmentations, and also raises several further research questions, such as the dependency of the source image and the target semantic classes; how networks combine features for extrapolation; and the role and informational content of augmenta- tions; all of which we hope inspires further research

---

Published as a conference paper at ICLR 2023
REFERENCES
Yuki Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, what we can learn from a single image. In ICLR, 2020. 2, 3,4, 4, 5, 1, 15
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NeurlPS, volume 27, 2014. 3
Seungdae Baek, Min Song, Jaeson Jang. Gwangsu Kim, and Se-Bum Paik. Face detection in untrained deep neural networks. Nature communications, 12(1):1-15, 2021.
Sven Bambach, David Crandall, Linda Smith, and Chen Yu. Toddler-inspired visual object leamnin. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garett (eds.), NeurIPS, volume 31, 2018. 2, 7, 10, 12, 17
Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning see by looking at noise. NewrIPS, 2021. 5,
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good teacher is patient and consistent. CVPR, 2022. 3, 5, 9, 16, 17
Joamna Bitton and Zne Papakipos. Augly: A data augmentations library for audio, image, text, and video.
https: /gi thub. com/ facebook wetoookkkeeeeee resear rch Auglly, 2021. 3, 16
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006. 08572, 2020. 3
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Berbert-Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few- shot learners. NeurlPS, volume 33, pp. 1877-1901, 2020.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. PAMI, 35(8):1872 1886, 2013.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In CVPR, pp. 13169-13178, 2020. 3
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image
features on non-curated data. In ICCV, 2019. 13
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR. pp. 6299-6308, 2017. 7,
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang. Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Dafl: Data-free learning of student networks. /CCV, 2019. 3
Adam Coates, Andrew Ng. and Honglak Lee. analysis single-layer networks in unsupervised feature learning. In AISTATS, pp. 215-223. PMLR, 2011.
Wojeiech Czamecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. In NeurIPS, 2017. 3
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lueas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 1, 8,
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. CVPR, pp. 203-213, 2020. 7,6 6
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In JCML, pp. 1607-1616. PMLR, 2018. 3, 14
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In CVPR, pp. 12046-12055, 2019.
10

---

Published as a conference paper at ICLR 2023
Jiamping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey, IJCV, 129(6):189-1181)) 2021. 3
Kaiming He, Xiangyu Zhang, Shanqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast unsupervised visual representation learning. In CVPR, pp.
2020. 13, 14
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and
perturbations. ICLR, 2019. 14
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2019. 6
Geoffrey Hinton, Oriol Vinyals. and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint
vrXiv:1503.0233// 2015. 2, 4, 14
Kevin Jarrett, Koray Kavukcuoglu, Marc' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference on computer vision, pp.
2146 2153. IEEE, 2009. 3
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. ACCV, 2020. 2,5
Gwangsu Kim, Jaeson Jang, Seungdae Back, Min Song. and Se- Bum Paik. Visual number untrained deep neural networks. Science advances. 7(1)eabd6127, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 17
Simon Korbblihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, pp. 3519-3529, PMLR, 2019. 10,
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Junnan Li. Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. ICLR, 2021.
Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few sample knowledge distillation for efficient network compression. CVPR, pp. 14639-14647, 2020.
Yuncheng Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Learning from noisy labels with distillation. In ICCV, pp. 1910-1918, 2017. 3
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie. Feature pyramid networks for object detection. In CVPR, pp. 2117-2125, 2017.
Pengpeng Liu, Irwin King, Michael Lyu, and Xu. Ddflow: Learning optical flow with unlabeled data distillation. In AAAI, pp. 8770-8777, 2019.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint 1rXiv:1710.07535, 2017. 3
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 16, 17
Ken MacLean. Voxforge. Ken MacLean.| Available: http://www.vofffore.orgghoooo 2018.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming Hc, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining, In ECCV, pp. 181-196, 2018. 3,3
Paul Micaelli and Amos J Storkey. Zero-shot knowledge transfer via adversarial belief matching. In NewrIPS, pp. 9551-9561, 2019. 3,
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In ICLR, 2020. 11
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. arXiv preprint 101::2107.13034 2021.

---

Towards Published as a conference paper at ICLR 2023
Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In CVPR, vol- ume pp. 1447-1454, 2006.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11): 2017. 8
Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. ICCV, 42(3):145 12555155-1 2001. 10, 17
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, (6583):677666 1996. 2
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-bor box models. In CVPR, 2019. 5
Emin Orhan, Vaibhav Gupta, and Brenden M Lake. Self-supervised learning through the eyes of a child.
NeurlPS, 33, 2020. 7
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In ICASSP, pp. 5206-5210. IEFE, 2015. 6
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 200:::0000000000 2021.
Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: uuni-supervised learning. In CVPR, pp. 4119-4128, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep
/4 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for scale image recognition. arXiv preprint 3xXtv:1409..55 1556, 2014. 17
David Snyder, Guoguo Chen, and Daniel Povey, Musan: A music, speech, and noise corpus. 110.0445, 2015. 6
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXi::/2/2.0402,, 2012. 7,5
Kartik Sreenivasan, Shashank Rajput, yo-y Sohn, and Dimitris Papailiopoulos. Finding everything within random binary networks. 1rXiv:2110.08996 2021.
Marco Tagliasacchi, Beat Gfeller, Pélix de Chaumont Quitry, and Dominik Roblek. Self-supervised andio representation learning for mobile devices. arXiv: 1796, 2019. 16
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. ICLR, 2020a. 3, 5,5,
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 776-794. Springer. 2020b. 7
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pp. 32-42, October 2021. 6
Frederick Tung and Greg Mori.
knowledge distillation. In ICCV, pp. 1365-1374, 2019.
3
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In CVPR, pp. 9446-9454, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. JMLR, 911). 2008.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexci A Efros. Datasct distillation. 1rX1v:1811.10099, 2018. 3
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. orXi::880..03200. 2018. 6
Yuxin Wu and Kaiming He. Group normalization. In ECCV, pp. 3-19, 2018.
Yuxin Wu, Alexander Kinillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: 2019. 13
//githtu Ligtthu... ccm/ httpc///ccccoooeeerrreeeeeeeeeee
12

---

Published as a conference paper at ICLR 2023
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In ICLR, 2020. 3
Jingwen Ye, Yixin Ji. Xinchao Wang, Xin Gao, and Mingli Song. Data-free knowledge amalgamation via group-stack dual-gan. In CVPR. pp. 15516-12525 2020.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data knowledge transfer deepinversion. CVPR, pp. 8715 8724, 2020. 3
Sangdoo Yun. Dongyoon Han, Seong Joon Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6023- 6032, 2019.
4, 6, 17
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016a. 17
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance convolutional neural networks via attention transfer. arXiv:1612.03928, 2016b.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. 5,
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. /CML, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. PAMI, 2017. 7
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and supermask, NeurlPS, 32, 2019. 3
13

---

Published as a conference paper at ICLR 2023
The augmented image prior: Distilling 1000 classes by extrapolating from a single image
Appendix
Table of Contents
Additional Visualizations
Input images
A.2 Training patches and predictions
B Additional Experiments
5
B.I Video action recognition
5
B.2 Varying Distillation Loss Functions
B.3 ImageNet Vision Transformer Distillations
9
B.4 Training with various random noises
6
B.5 Standard deviations in performance
7
B.6 Neuron activation maximizations
7
B.7 t-SNE of feature space
10
B.8 Centered Kernel Alignment comparison
10
B.9 Single-image training data vs toddler data
10
B.10 Data-free Pruning and Quantization of Pre-trained Models
11
B.11 Transfer learning experiments
13
B.12 Robustness experiments
14
B.13 Analysis of per-class accuracies
14
C Implementation Details
15
C.I Initial patch generation
15
C.2 Computing log
log-Mel Mel spectrograms
16
C,3 Audio neural network architecture
16
Training
16
GIST features comparison
A ADDITIONAL VISUALIZATIONS
A. INPUT IMAGES
Image sources and licences. From top to bottom, the images in Fig. 7 have the following licences and sources: (a): CC-BY (we created it), (b)" public domain (NASA) (c)* CC BY-SA 3.0 (by David Ball), (d)² pixabay licence (personal and commercial use allowed), (c)$ personal use licence.
Images (b), (d) and (e) are identical to the ones in the repo of (Asano et al., 2020), while (a) we have recreated on our own and (c) is a replacement for a similar one whose licence we could not retrieve.
reg.i___hhhh_eee_eii...g Protps:/ h/pp::///p/ xabayy wwwpppotooo/////oooooooooooooooo japan-pemat japanese- stt.eetatgg--111000
"httpe: / /tt.///////n.c///////////// wwk.e..wiimmmmm.... kineiime wereeddee..gggg.. ozg/ wiki Makk//Pile www.bileeeetueeeeeeeeeea
Instpe: //wc///////n.////////////// wikimedia, htg//wkk////t/ Paikk/Pileeeee supeint-brrige 2.jpg
4yttps: //www/ www..ttahhbb.. teahub. wwwp/ http:/ooooooooooeoe/ g-antmal-ou [paper-wal Ipapersafarl -ungg jangle-animal

---

Published as a conference paper at ICLR 2023
(a) The "Noise" Image. From uniform noise [0,255]. Size: 2,560x1,922 PNG: 16.3MB
(b) The "Universe" Image. Size: 2,300x2,100 JPEG: 7.2MB.
(c) The "Bridge" Image. Size: 1,280x853, JPEG: 288KB.
(d) The "City" Image. Size: 2,560x1,920, JPEG: 1.9MB.
(c) The "Animals" Image. Size: 1,300x600, JPEG: 267KB.
Figure 7: Single Images analysed. Here we show the images analysed in Table 1.
2

---

Published as a conference paper at ICLR 2023
Audio source. For the experiment audio representation distillation, 5.5mins English news-clip taken from BBC about how can Europe tackle climate changel and a 5mins clip about 11 Genmanic languages'.
A.2
TRAINING PATCHES AND PREDICTIONS
aL1E
0.2803%:
packet (692)
0.1563%: cleaver (499)
0.1994%:
rubber eraser (767)
0.1549%: can opener (473)
0.1947%: envelope (549)
0.1521%: whistle (902)
0.1910%:
Band Aid (419)
0.1511%:
screw (783)
0.1865%:
lighter (626)
0.1507%:
spatula (813)
0.2653%: comic book (917)
0.3454%:
grocery store (582)
0.2453%:
sarong (775)
0.3175%:
(612)
0.2436%: sombrero (808)
0.2830%:
restaurant (762)
0.2344%: shopping basket (790)
0.2367%:
toyshop (865)
0.2336%:
toyshop (865)
0.2322%:
barbershop (424)
0.2736%: nail (677)
0.2557%: slide rule (798)
0.2657%:
screw (783)
0.2362%:
rule (769)
0.2121%: hook (600)
0.2135%: letter opener (623)
0.2035%:
knot (616)
0.2116%: Windsor tie (906)
0.1901%:
letter opener (623)
0.2061%:
matchstick (644)
0.3694%:
bubble (971)
0.2275%: plane (726)
0.2253%:
jellyfish (107)
0.2041%: mousetrap (674)
0.2226%:
water bottle (898)
0.1982%: joystick (613)
0.2177%:
abaya (399)
0.1939%: screw (783)
0.2062%:
hourgiass (604)
0.1932%:
rubber eraser (767)
0.1931%:
African chameleon (47)
0.1931%:
African chameleon (47)
0.1793%:
vine snake (59)
0.1793%:
vine snake (59)
0.1744%:
toucan (96)
0.1744%:
toucan (96)
0.1726%:
green mamba (64)
0.1726%:
green mamba (64)
0.1725%:
jacamar (95)
0.1725%:
jacamar (95)
0.2433%: theater curtain (854)
0.5568%:
carpenters kit (477)
0.2241%:
red wine (966)
0.4062%:
slot (800)
0.2030%: perfume (711)
0.3014%:
screwdriver (784)
0.2027%: spotlight (818)
0.2651%:
hammer (587)
0.2006%:
velvet (885)
0.2597%:
accordion (401)
Figure 8: ImageNet training data. Here we visualize the augmented training data used for training along with the temperature-scaled Top-5 predictions of the ImageNet pretrained ResNet-50 teacher.
In Fig. 8, we show several training patches as they are being used during training along-side the teacher-network's temperature-adjuster predictions. Only very few training samples can be inter- preted (e.g. 3rd row: "nail" or "slide rule"), while for most other patches, the teacher is being very "creative".
Prctpa://www youttwee oom/ wst Pwwn Pappo Irctpa:// ///ee/////ww//// http://wwwwwwwtuuooe youtube, com/ wat hwwptt/wwggppp_pp_
3

---

Published as a conference paper at ICLR 2023
Airplane Autamabile
Bird
Dog
Horse
Ship
Truck
Frog
Cat
Dog
Airplane Automobile
Deer
Harse
Ship
Frog
Airplane Automobile
Bird
Cat
Dog
Ship
Truck
Airplane
Bird
Automabile
Cat
Dog
Frog
Horse
Ship
Deer
Airplano Automobile
Cat
Dog
Frog
Horse
Ship
Truck
Cat
Airplane Autamabile
Deer
Dog
Frog
Harse
Truck
Cat
Bird
Dog
Airplane Automobile
Horse
Truck
Airplane Automobile
Deer
Dog
Frog
Horse
Ship
Figure 9: CIFAR-10 training data. We visualize temperature scaled predictions of NeteResNet- 40-4 teacher random patches generated from the Animal image (see Figure 7).

---

Published as a conference paper at ICLR 2023
In Fig. 9, we show a similar plot for the training patches used in CIFAR-10 training. Here we show the whole teaching signal, and highlight the top-5 predictions of the teacher for visualization.
Figure 10: Audio spectrograms. We visualize spectrograms generated from second segment of the augmented audio. We generate 50K audio clips from a 5 minutes audio recording by randomly taking a cropped audio segment of 2 seconds and augmenting via one of the transformation functions mentioned in Section C.I. During model training, we use second segment and apply MixUp augmentation that varies across batch.
Finally, in Fig. 10, show some augmented training samples for the audio classification experiment. This figure shows how various spectrograms can be generated from single clip by utilizing many augmentations.
(a) Source image.
(b) Patches.
Figure 11: Satellite image patches. We visualize random patches of size 32 32 generated from single satellite image of Yogyakarta Airport, Indonesia. The image is of size 3846 x 4087 and 17MB large as a JPEG.
B ADDITIONAL EXPERIMENTS
B. VIDEO ACTION RECOGNITION
We conduct further experiments on video action recognition tasks. For this we use the common UCF-101 (Soomro al,, 2012) dataset, more specifically the first split. As distillation data,
5

---

Published as a conference paper at ICLR 2023
generate simple source videos of 12 frames that show linear interpolation between two crops of the City image. We generate 200K these videos and apply AugMix (Hendrycks et al., 2019) and CutMix (Yun et al., 2019) during the distillation for additional augmentations. For the architectures we use the recent, state-of-the-art X3I) architectures (Feichtenhofer, 2020) as they obtain strong performances and allow for flexible scaling of network architectures. We use X3D-XS model as the teacher, which is trained with imputs of 160 x 160 and 4 frames and temporal subsampling factor of 12. For the student model scale this X3D-XS model in terms of width and depth by increasing the depth factor and width factor from 2.2 to 3 and 4 respectively, yielding a network with 18. 1M parameters compared to 3. 3. For UCF-101 we finetune the teacher model starting from supervised Kinetics-400 (Carreira & Zisserman, 2017) pretraining, which achieves a 90.4% performance. We use batch size of 128 per GPU on two GPUs, learning rate of 10-3, a temperature of 5, cosine learning rate schedule with warmup of 5 epochs, with no weight decay and no dropout, As this experiment takes even longer than the image ones, we have not conducted systematic ablations on these parameters. Instead, mercly picked ones which looked promising after a few epochs but believe that even this small experimental evidence is enough to investigate to what extent these 1-image "fake-videos" can be used for learning extrapolate.
The results are given in Table 6. We observe a performance of over 72% on UCF-101 using just single fake-video the training data along with the pretrained teacher. While this number is far below the state of the art or the teacher's performance, for example outperforms CLIP's 69.8% zero-shot performance with ViT-B16 model and shows that the student is able to extrapolate the action classes of UCF from just one datum.
B.2 VARYING DISTILLATION Loss FUNCTIONS
In Table 9 we compare the standard KD loss against LI and L2 losses (using scaled logits as in the KD loss) for training the student. We find that while for CIFAR-10 the choice of the distillation method does not impact the performance,
the more difficult ImageNet dataset, there is small difference of around -0.4% and -0.5% for L1 and L2 losses as compared to the KD loss at epoch 30. This shows firstly that our method is not dependend on a specific loss for distilling knowledge from a teacher to a student. Secondly, it shows that the standard KD loss is actually a strong baseline, as was also reported in (Tian al., 2020a).
Table 9: Distillation Loss Function Comparison. We compare the distillation performance with different loss functions along with teacher and student supervised training. Results are shown for CIFAR-10 10 with a ----eNeero---iddeRRReeee
setting and for ImageNet with R18-R50. For ImageNet we show the performance progression after 10/20/30 epochs
Data
Teacher
Student
L1
L2
KD
C10
95.4
95.2
93.3
93.4
93.4
IN-IK
69.8
76.1
36.2/45.1/51.8
25.1/45.
34.5/47.0/52.2
B.3 IMAGENET VISION TRANSFORMER DISTILLATIONS
In Table 10 we compare the best results from the main paper of distilling to ResNets to distilling to the ViT (Dosovitskiy et al., 2021) and CaiT (Touvron et al., 2021) architectures. We observe that while convolutional networks learn significantly faster, with e.g. differences of more than 30% at the tenth epoch. While the ViT architecture almost catches up at epoch 200 with final performance of 63.9%, the CaiT model remains fairly low at 58.5%. Nevertheless, this experiments shows that distilling across architecture types works and that even less constrained Vision Transformers can be trained with our method simply.
B.4 TRAINING WITH VARIOUS RANDOM NOISES
In Table 11, experiment with adding various types of noise structures as inputs to the teacher and student instead of augmented patches from the images. Notably, we either input random uniform noise between [0,1] (row (w)), random normal noise with mean of 0 and standard deviation of

---

Published as a conference paper at ICLR 2023
Table 10: Training Vision Transformer student models. Setting as in Table (from which first row is copied for comparison).
Setting Teacher
Epochs
Image
Student
10
20
30
50
200
Ix City
R18
R50x2
46.1
55.6
59.7
63.1
69.0
City
R18
CaïT-S24
9.2
25.9
34.0
42.5
58.5
Ix City
R18
+, VIT-B
15.7
31.9
39.3
47.2
63.9
(row (x)) and convex combinations of the augmented input patches with the normal noise. We find that just like the augmented patches from a random noise image (row (g)), random noise as input also does work for distilling semantic classes. While generating new random noise at every iteration does create more variability, in practice, a random noise image with augmentations likely already achieve high amount of variance, showing that this is not the crucial component for learning, but instead having structures from a real image. This is further confirmed by the steady increases in performance when moving from row (x) (z).
Table 1I: Training with various random noise. Setting as in Table (from which first 3 rows are copied for comparison).
Setting
Epochs
Input
Teacher
Student
10
20
30
50
200
(g) Noise Image (p)
R18
R50
0.1
0.1
0.1
(h)
Bridge Image (p)
R18
+ R50
21.2
34.8
40.0
(i)
City Image (p)
R18
+ R50
34.5
47.0
52.2
56.8
66.2
(v) StyleGAN Baradad et al. (2021)
R18
R50
15.4
30.1
37.8
44.4
60.4
(W) Random Uniform [0,1] Noise
R18
+ R50
0.1
0.1
0.1
0.1
Random Normal (0,1) Noise
R18
R50
0.1
0.1
0.2
0.2
0.8x Random Normal (0,1) 0.2x City (p)
R18
+ R50
0.2
0.3
0.4
0.7
0.5x Random Normal (0,1) 0.5x City (p)
R18
R50
7.6
13.1
17.1
22.3
B.5 STANDARD DEVIATIONS IN PERFORMANCE
In Table 12, report the performance of five independent runs CIFAR-10 dataset to further highlight the robustness of our experimental results. We notice consistent accuracy scores across different training runs with minor changes of & +0.2.
Table 12: Model training with different random seeds. We compare the performance of five inde- pendent runs CIFAR-10 dataset. We use WRN40-4 as teacher and WRN16-4 as student. "Full" shows the accuracy distilling using CIFAR-10 data, and "Ours" represents distillation using random patches generated from "Animals" image.
Teacher
Student
Full 93.26 1 0.17
Ours 93.38 0.07
95.38 1 0.09
94.76 + 0.22
B.6 NEURON ACTIVATION MAXIMIZATIONS
In Fig. 12, we show further class-neuron activation maximization visualizations for the R50-R550 distillation experiment and compare them to the corresponding ones of the supervised teacher model. Similar to the examples in the main paper, we find that the neurons visualizations are remarkably similar.
In Fig. 13, we show further visualizations of the trained student model from the R50-R51 distilla- tion experiment, We show five randomly chosen neurons for every layer in the ResNet.

---

Published as a conference paper at ICLR 2023
Balloon
Pretzel
Altar
IN-1k supervised
1-image distilled
Fountain
Hourglass
Zebra
IN-1k
supervised
1-image
distilled
Figure 12: Visualizing neurons via activation maximisation of final layer neurons. We compare ImageNet supervisedly trained model against our model which has been trained with single image distillation.
8

---

Published as a conference paper at ICLR 2023
ResNet layer
Five random neurons in each layer
1
2
3
4
Figure 13: Visualizing intermediate neurons via activation maximisation. We visualize five ran- domly picked neurons for each layer and show their maximally responding input images.

---

Published as a conference paper at ICLR 2023
B.7 T-SNE OF FEATURE SPACE
Airplane
Deer
Horse
Automobile
Dog
Ship
Bird
Frog
Truck
Cat
Figure 14: t-SNE embeddings of CIFAR-10. We visualize t-SNE embeddings of WideR16-4 model distilled from WideR40-4 with random patches generated from an 'Animal' image.
We visualize representations of distilled student model on CIFAR- 10 validation set in order to high- light semantic relevance of the features in Fig. 14. We extract the feature from penultimate layer of WideR 16-4 model for computing 2-D t-SNE embeddings. The visualization clearly highlights that distilled features contain substantial amount of semantic information to correctly differentiate between different object categories.
B.8 CENTERED KERNEL ALIGNMENT COMPARISON
In Fig. 15, we visualize similarity between different layers of the models trained in general super- vised manner and with distillation using patches 1-image for CIFAR-10 and ImageNet datasets. We use centered kernel alignment (Kombbith et al., 2019) method to identify the resemblance of features learned with these different training approaches. We notice high similarity in representa- tions, which reinforce empirical evaluation that the student models indeed learn useful features required for differentiating between semantic categories.
B.9 SINGLE-IMAGE TRAINING DATA VS TODDLER DATA
In Fig. 16a, we compute the distances of GIST features (Oliva & Torralba, 2001) of 10K training images of the "City" image at resolution 256 256 Fig. 16a. Following (Bambach 2018), we L2 normalize these GIST features before computing waie-wise distances and plotting the histogram of the values. To make the comparison our training data with the visual inputs of toddlers more concrete, we compare low-level GIST (Oliva & Torralba, 2001) features of our 1-image dataset against those computed from visual inputs toddler in (Bambach et al., 2018). As we do not have access to the dataset of (Bambach et al., 2018), 8), we copy their Figure 3b for reference in Fig, 16b. While the GIST distances for ImageNet are very different with a mean around 0.75 (Bambach al., 2018), find that single image dataset's distribution in Fig. 16a closely resembles that of the visual inputs toddler. While this one possible explanation for why this type of data might work well for developing visual representations, further research still required.
10

---

Published as a conference paper at ICLR 2023
51
47-
45
CIFR 10 3
41
0.8
15
29
0.6
25
21
19
13
0.2
13.5 211253772222222325579442222
Layer (1-image distiled)
a WideR16 via CIFAR-10 supervised training l-image distilled.
Layers (ImageNes supervised)
20
20
0.4
62
R50 distilledd)
(b) R50 via ImageNet supervised training vs I 1-image distilled.
Figure 15: Representation similarity with CKA Kornblith et al. (2019). We visualize the simi- larity between supervised representations and those distilled using random patches of 1 -image. As Nguyen et al. (2020), we compute CKA for all the layers model using CIFAR-IO testset. The layers in the same block group shows high similarity though both models are trained in different ways, indicating that distilled features are highly similar to those learned in a supervised way.
B.10 DATA-FREE PRUNING AND QUANTIZATION OF PRE-TRAINED MODELS
Neural network compression has been studied extensively in the literature to produce light-weigh models with the objective of improving computational efficiency. In addition knowledge distilla- tion, network pruning and quantization are other well-known approaches. In the former, the goal is remove sparsify (zeroing-out) the network's weights based specific criteria, such subset of weights with the lowest magnitude. The later is concerned with reducing precision from 32-bits floating numbers 8-bits and even single bit as binary neural networks. In general, from practical standpoint post-training compression is largely employed in conjunction with fine-tuning to avoid accuracy drop.
Here, we study utilize our proposed single-image distillation framework for post-training model compression without using any real data. Specifically, we ask the question, whether tra-trdined model can be compressed when no in-domain data is available? To this end, we utilize Tensorflow Model Optimization *oolkit" to prune and quantize various pre-trained model on CIFAR- and
"https::/ http:/twwttennee. www.wwo.woooooeeoooooeoooooooooe _opt optiim mazioon

---

Published as a conference paper at ICLR 2023
Toddler
0.2
0.20
Prop. of Data
0.15
Prop. Data
0.15
0.1
0.10
0.05
0.05
0.25
0.5 0.75 1
0.00
0.00
0.25
0.50
0.75
1.00
GIST Distance
GIST Distance
(b) Toddler data. Figure from (Bambach et al.,
(a) Our 1-image dataset.
2018).
Figure 16: Pair-wise distances 1.2-normalized GIST features for 10K images from our 1-image training dataset used for our ImageNet experiments
Quantization (source)
Ours Quantization (single-image)
Pruning (source)
Ours Pruning (single-image)
Dataset
Model
Standard
ResNet56
93.77
93.64
93.4S
93.38
93.18
VGG11
91.57
91.22
90.97
91.14
90.86
VGG19
93.28
92.94
93,00
92,84
92,96
CIFAR-10
WideRNet16-4
94.81
94.40
94.59
94.76
94.43
WideRNet40-4
95.42
95.09
94.90
95.29
94.82
ResNet56
70.99
70.89
69.85
70.74
70.42
VGG11
69.65
69.92
68.86
69.77
69.13
VGG19
70.79
70.60
70.21
70.89
70.56
CIFAR-100
61deRNet16-4
75.81
75.45
74.71
75.68
75.51
WideRNet40-4
78.14
78.27
77.60
77.94
77.78
Table 13: Self-distillation with single-image for efficient deep models. We perform model com-
pression via self-distillation using source data and 50k random patches generated from 'Animals' image for 50% sparsity (in case of pruning) and 8-bits quantization without any noticeable loss in performance.
12

---

Published as a conference paper at ICLR 2023
CIFAR-100 100 datasets. We self-distillation, where a pre-trained model acts teacher and student clone of it is compressed during fine-tuning phase. Table 13, we report accuracy scores for the pruned and quantized models in comparison with the standard models and those compressed using real in-domain data. In all the cases, we can notice that there is negligible loss in accuracy, i.e., <= 1 %, while using random patches.
Sparsity
NesNe
VGG11
WideRNet16-4
C10
C100
C10
C100
C10
C100
0%
93.77
70.99
91.57
69.65
94.81
75.81
25%
93.55
70.85
91.03
69.37
94.55
75.40
50%
93.18
70.74
90.86
69.13
94.43
75.51
75%
92.68
67.87
90.54
67.81
94.10
74.05
85%
91.50
61.22
89.12
65.23
93.05
71.46
Table 14: Varying Sparsity. Pruning results achieved CIFAR-10/100 for different sparsity rates using self-distillation with random patches generated from the 'Animal' image.
51
47
41
0.8
37
35
31
0.6
21
0.4
15
0.2
13.5 91113151111 wtw.
Layer (Ours Pruned]
Figure 17: Representation similarity with CKA of a pruned and standard WideR16-4 models trained on CIFAR-100.
Further, in Table 14, vary sparsity ratio and can observe that model can be pruned up 75% sparsity with three-points drop in accuracy but with 25% pruning ratio the loss is merely notice- able. In Figure 17, analyze representational similarity of pruned WedeRNet16-4 model with the original trained on CIFAR-100, the diagonal entries highlight that even after 50% sparsity the rep- resentations are very similar. To summarize, with model compression use case, we barely scratched the surface of what is possible with our single-image distillation framework and we hope it will inspire further studies and applications other domains.
B.11 TRANSFER LEARNING EXPERIMENTS
In this section, compare the performances downstream tasks of teacher network (ResNet- 18), our imgle- distilled network (ResNet-50) and another ResNet-50, purely trained a self- supervised manner on our augmented single-image dataset. For this, we pretrain using the official code of MoCo-v2 (He et al., 2020) in the standard 200 epoch setting, while applying their default "v2" augmentations during training.
Evaluation. We follow standard linear evaluation procedure from the MoCo-v2 repo. which uses a batch size of 256, learning rate of 30 which is multiplied by 0.1 at epochs 60 and S0 for total of 90 epochs. For the data-eflicient full-fine tuning, we utilize the implementation from SWAV (Caron et al., 2019), which trains for 20 epochs using cosine decay learning rate schedule and a batch size of 256. For the data-efficient SVM classification experiments, we follow the implementation of (Li et al., 2021) and report average results from trials and keep the SVM's cost parameter fixed at a value of 0.5. For the COCO object detection experiments
the detectron2 repo (Wu et al.,
13

---

Published as a conference paper at ICLR 2023
Representation learning performance. We compare our distillation approach (R18-R50) against pretraining via MoCo-v2 on the same data. We report: linear eval. accu- racy on IN-1k and Places; accuracy when finetuning and using 1% of ImageNet with labels; SVM classification mAP on PVOC07 and accuracy on Places with varying numbers of images per class; mAP for COCO-detection with FPN.
Table 15:
IN-Ik
Places ~0K
IN-IK (1%)
PVOC
Places
COCO
imgs / class
~13
16
16
NA
IN-IK teacher
[69.8]
44.1
[69.8]
65.7
77.1
21.4
30.4
IN-Ik R50
[76.2]
51.5
[76.2]
73.8
82.3
27.0
35.4
38.9
1-image MoCo-v2
28.5
28.8
14.4
17.2
26.6
4.5
9.7
35.6
1-image Distill
68.8
47.2
64.1
58.9
73.5
21.0
31.3
35.4
Noise
Blur
Weather
Digital
Network
Error
mCE
Gauss, Shot Impulse Defocus Glass Motion Zoom
Snow
Frost
Bright
Contrast Elastic Pixel JPbG
IN-IK RI8
30.2
84.7
87
91
87
89
86
69
90
80
students
1-image R50x2
31.0
85.9
88
89
91
85
92
88
89
88
86
82
71
80
91
82
87
1-image R50
33.8
89.8
93
93
96
87
90
92
91
90
84
77
83
96
87
93
Table 16: IMAGENET-C evaluations. Clean Error, mCE, and Corruption Error values of different corruptions and architectures. The mCE value the mean Corruption Error of the corruptions in Noise, Blur, Weather, and Digital columns.
2019) and the Ix schedule using FPN (Lin et al., 2017) as (He et al., 2020). We additionally vary the learning rates from Ix and 4x as the models are trained with very different losses compared the supervised variant for which the learning schedule made. The results are shown in the last row of Table 15 and show that our models benefit from a higher initial learning rate, which also does have any negative effect on training longer e.g. with the 2x schedule, while the MoCo pretrained model works best with the Ix learning rate.
We find that freezing the backbone and only training a linear layer on top of it, can achieve per- formances of 68.7% and 47.2% for ImageNet and Places, respectively, vastly outperforming the self-supervised variant. similar trend is observed for data-efficient classification reaching close but lower performances to the teacher model.
B.12 ROBUSTNESS EXPERIMENTS
Table 16, we report results on ImageNet-C (Hendrycks Dietterich, 2019).
B.13 ANALYSIS OF PER-CLASS ACCURACIES
Worst underperforming 10 classes. The 10 classes that have the highest top-1 accuracy difference compared to the teacher model (for the R18- setting) are: 'Tibetan terrier, Lakeland terrier, golden retriever, dhole, Border terrier, cocker spaniel, Welsh springer spaniel, Brabancon griffon, collie, mountain bike', with accuracies differences ranging from -34% to -20%. From the un- derperforming classes, we find that 9/10 of these are dog breeds. This illustrates how out single-image trained model lacks behind in very fine-grained classification compared to the ImageNet trained model. This is likely because the single-image lacks the necessary structures and patterns for disambiguating these classes despite the help of augmentations.
In Fig. 18, we plot the validation performance against the frequency of how often the class appears as t prediction of the teacher during epoch of training and find that it is unrelated, showing the student is profiting heavily from the knowledge contained in the pret-predictions, echoing findings from (Hinton al., 2015; Furlanello et al., 2018). We further compare these per-class performances
14

---

Published as a conference paper at ICLR 2023
A/Sludent-1 Teacher) IN-1K accuracy (%)
100
20
R2-0.013
Studont IN- 1k val accuracy (%)
10
60
-10
-30
R2=0.078
0
10'
10²
10
10"
10"
10"
10"
10'
Teacher training argmax frequency
Teacher training argmax frequency
Figure 18: Comparison of per-class aceu- racy. Per-class student performance vs fre- quency of the class being the top-I prediction of the teacher.
Figure Comparison of per-class rela- tive accuracy. Per-class performance rela- tive to the teacher is plotted vs frequency of the class being the top-1 prediction of the teacher.
against the teacher's performance in Fig. 19 and find that there is relationship between when the student under or overperforms the teacher vs how often a particular likeness of a class appears.
C IMPLEMENTATION DETAILS
All code will be released open-source and attached as supplementary information to this submis- sion.
C.I INITIAL PATCH GENERATION
We do not tune the individual patch generation and instead adapt the procedure directly from (Asano et al., 2020y*t. The augmentations applied to the input image of size H x W to arrive at individual patches of size x P is (in order) as follows:
I) RC (size=0. (H, )
scale= (2e-3, 1)) 2) RRC (size=
3) (enddmmeffinneeeeeereessss shear=30)
4) RandomVF14p(0=00.55
5) Random
6) CenterCrop (P,P))
7)
(0. (0.4, 0. p-0.5)
All transformations are standard operations in PyTorch: RC stands for RandomCrop, i.e. taking random crop in the image with a specific size; RRC for RandomResizedCrop i.e. taking a random sized crop withing the size specified in the scale tuple (relative to the input); RandomAffim for random affine transformations (rotation and shear); Random VFlip and RandomHFlip for random flipping operations in the vertical and horizontal direction, and CenterCrop crops the image in the center to square image of size P x P. Finally, ColorJitter computes photometric jittering, where the parameters for the strengths are given in the order of brightness, contrast, saturation and hue and applied with a certain probability.
tthtess: /gi thub. wwwb.com/yukiiii wtt..////wooaaoo//////oo//o ng1 wwliingpretrrinnnn
15

---

Published as a conference paper at ICLR 2023
Similarly, for audio clips generation given single audio, augmentation operations from (Bit- ton & Papakipos, 2021) with default settings. Specifically, to create a single example we apply the following procedure: randomly crop segment of 2-seconds, and randomly sample an aug- mentation function to create transformed instances and save them in mono format. In our work, we these augmentations, all with their default settings:
1) add-bbackkround-noo.e.
2) change-volume
3) clicks
4) clip
5) harmonic
6) high-pass-filter
7) Low-pass-filter
8) normalize
9) equliizee
10) percussive
11) pitch-shift
12) reverb
13) speed
14) time-stretch
C.2 COMPUTING LOG-MEL SPECTROGRAMS
The log-Mel spectrograms are generated on-the-fly during training from randomly selected 1- second crop an audio waveform as the model's input. We compute it by applying short-time Fourier transform with window size of 25ms and hop size equal to 10ms extract 64 Mel- spaced frequency bins for each window. During evaluation, we average over the predictions of non-overlapping segments of entire audio clip.
C.3 AUDIO NEURAL NETWORK ARCHITECTURE
Our audio convolutional neural network is inspired by (Tagliasacchi et al., 2019) and it consists of four blocks. We perform separate convolutions in each block with kernel size of 4. One on the temporal and another on the frequency dimension, we concatenate their outputs afterward to perform joint 1 x convolution. It alllow model to capture fine-grained features from each dimension and discover high-level features from shared output. We apply L2 regularization with a rate of 0.0001 in each convolution layer and also group normalization (Wu & He, 2018). Between the blocks, we utilize max-p max-pooling to reduce the time-freque free-frey dimensions by factor of 2 and use a spatial non-linear activation
dropout rate with rate of 0.2 to avoid fitting. We apply ReLU function and feature maps in the convolutions blocks are 24, 32, 64, and 128. Finally, we aggregate the feature with global pooling layer which are fed into fully-connected layer with number of units equivalent to the number of classes.
C.4 TRAINING
C.4.1 OPTIMIZATION.
For each of these datasets, we first train a teacher network in a usual supervised manner, that is then used for the distillation. For the distillation to the student model, use temperature motivated by findings in (Beyer et al., 2022). We keep this temperature fixed throughout the whole of the paper due to limited compute. For optimization,
Adam (Loshchilov & Hutter, 2018).
16

---

Published as a conference paper at ICLR 2023
C.4.2 SMALL-SCALE EXPERIMENTS
For experiments on CIFAR-10, CIFAR-100, and other smaller datasets, use Tensorflow for run- ning experiments on a single T4 GPU with a batch size of 512 using an Adam (Kingma & Ba, 2015) optimizer with fixed learning rate of 0.001. We use standard augmentations including, random left right flip, and random crop. The supervised models also uses cutout augmentations with a cutout size of 16 x 16. For Mix-up, we sample a uniformly at random between zero and one. In Cut-Mix, we use fixed value of 0.25 for and 8. With the following setup, each of the single image distillation experiment of 1K epochs took around 2 3 days. The supervised and standard (using source data) distillation models are trained for 100 epochs (per epoch 2000 steps) with a batch size of 128 using SGD for optimization. For VGG (Simonyan & Zisserman, 2014) and ResNet (He et al., 2016) models, at following steps 400, 32000, 48000, 64000 with momentum of 0.9. For WideResNet (Zagoruyko & Komodakis, 201 6a), we use a learning rate schedule of 0.1, 0.02, 0.004, 0.0008 that is decayed following steps 24000, 48000, 64000, 80000 with Nesterov enabled.
a learning rate schedule of 0.01. 0.1, 0.01, 0.001 decayed
In audio experiments, we an Adam (Kingma & Ba, 2015) optimizer with fixed learning rate of 0.001 and use batch size of 128 and 512 for standard models and single-clip distillation, respectively. Furthermore, we also utilize Mix-Up augmentation during our knowledge distillation experiments.
C.4.3 LARGE-SCALE EXPERIMENTS
We use PyTorch's 's DistributedDat arallel engine for running experiments 2 A6000 GPUs in parallel with batch-sizes of 512 each. For optimization we use AdamW (Loshchilov & Hutter, 2018) with learning rate of 0.01 and a weight-decay of 10-1, These values were determined by eyeballing the results from (Beyer et al., 2022) to find setting that might generalize across datasets, as we do not have enough compute to run Hyper-paramete sweeps. With this setup, each 200 epoch experiment took around 5 days. We have also found that halving the batch-size performs equally well, and is more amenable lower-memory GPUs. For the distillation experiments, we use Cut-Mix (Yun et al., 2019) with its default parameters of a = 3 = 1.0.
C.5 GIST FEATURES COMPARISON
In Fig. 16a, we compute the distances of GIST features (Oliva & Torralba, 2001) of 10K training images of the "City" image at resolution 256 256 in Fig. 16a. Following (Bambach et 2018), L2 normalize these GIST features before computing pair-wise distances and plotting the histogram of the values.
17


Reviews:
[{"review_id": "zazAHSXcuXo", "reviewer": "Reviewer_rSwz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates knowledge distillation using a data set consisting of a single or a few images which are heavily augmented. The paper finds that this type of knowledge distillation is as effective as using the training set, at least for small scale data sets such as CIFAR-10/100. Different design choices such as student and teacher architecture, the effect of the teacher training set as well as the image(s) chosen for distillation and the augmentations used are explored in-depth. Furthermore, the method is also shown to be effective for other modalities such as audio and videos, albeit these are less explored than images.", "summary_of_the_review": "The paper provides an interesting and comprehensive investigation of knowledge distillation based on a single or a few images. I think the paper should be published, but I also believe the points I raised above regarding calibration/robustness should be answered before publication, which is why I chose my rating defensively.", "strength_and_weaknesses": "Overall I feel that the paper provides a comprehensive investigation of single/few image based distillation. Nevertheless, I have a few suggestions on how to improve the paper.\n\n*Strengths:*\n- The observation that a single or a few images are enough for distillation is surprising and interesting (at least to me).\n- The paper covers a lot of datasets, architectures, and modalities, and provides many insightful ablations, and as such answers many important questions on the phenomenon.\n\n*Weaknesses/suggestions for improvement:*\n- One important open question: What happens to transfer and robustness properties of the distilled models? The output confidence scores seem to change significantly on the testing set (Figure 3), and the paper also delivers a potential hint why this could be happening: The output probabilities on the samples used for training are very flat for the teacher (Figure 8). Given this observation, the models might be mis-calibrated or have reduced OOD robustness, which would merit more investigation.\n- A related analysis would be to look at examples for which are misclassified by the student but correctly classified by the teacher. Are there some patterns.\n- It would be interesting to see if more sophisticated synthetic data than uniform noise can lead to better transfer, such as the synthetic data sets considered in Baradad et al. 2021.\n- The observation that the method works less well for transformer models could be discussed in the main paper. Given that transformers are harder to train on small data sets than ConvNets this is not very surprising.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MN4K9RjMkHZ", "reviewer": "Reviewer_AYLh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a distillation framework that makes a network learn with crops and augmentations from a single image. The learning target comes from a pre-trained teacher's outputs. The results obtained is of decent level (69% ImageNet top1). Various design options and internal properties of the resulting network are analyzed.", "summary_of_the_review": "The paper presents some interesting findings but I think there are exaggerations in claims as discussed in weaknesses, so both considered my recommendation is \"marginally above\".", "strength_and_weaknesses": "Strengths:\n1 ) It is interesting to know with a teacher and a single large image, what level of accuracy a student can achieve on certain images.\n\n2 ) It demonstrates (vanilla) knowledge distillation's immense power. It means KD can be done with extremely small amount of data. The paper may consider citing related few-shot KD methods such as [1].\n\n3 ) The paper's ablation study is comprehensive and reveals many interesting points. \n\t\n\nWeaknesses\n\n1 ) The teacher is supervised pre-trained. It contains information from a full dataset. Thus it may not be fair to say this study helps to understand the mechanism of NNs when provided with the minimum data. Moreover, the semantic categories are explicitly  modeled with teacher's final classifier, which is clearly learned from large data instead of the single image. \n\t\nTherefore, I don't think the question in \"we go far beyond these and instead ask what the minimal data requirements are for neural networks to learn semantic categories\" can be answered with \"only one image and augmentations\" without mentioning the teacher.\n\nThe conclusion sentence in the discussion section may add \"together with a supervisory signal provided by a teacher pretrained on large data with many semantic categories, provide sufficient …\"\n\t\nThere is another way to frame the conclusion and finding: KD can be done with one image and augmentations to reach decent accuracy. I think this one represents the finding more accurately and concisely.\n\t\n2 ) As the paper suggested in the intro, the use case of the method is not obvious.\n\t\n3 ) It may be an exaggeration to say that this is learning from a single datum also because the image used is very high resolution. Small crops of CIFAR-size (32x32) with data augmentations easily surpass thousands. The \"image space\" doesn't specify image size so in extreme I could tile all ImageNet images into one and still call it one \"image\". \n\n[1] few sample knowledge distillation for efficient network compression\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LMjaUgatArH", "reviewer": "Reviewer_rxVL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses a single image along with its augmentation and a pretrained classification model (teacher) together to train a student classifier. Through experiments on different datasets including images, audios, and videos, the authors find that the student classifier can achieve reasonably good classification performance.", "summary_of_the_review": "Overall, this paper is easy to follow and understand. It also has a lot of experiments. The big issue is it is not clear what benefits this method can bring to the community; also “random neural network with frozen weights” can also achieve impressive classification accuracy, then the finding in this paper does not sound as surprising as it is. \n\n", "strength_and_weaknesses": "Strength:\nIn order to verify its effectiveness, the authors have conducted experiments on different datasets and also conducted ablation studies on different components of this proposed method. \n\n\nWeakness:\n\n1) The motivation is not clear. Since this proposed method only uses a single image and its augmentations as its training data (it also uses a “teacher” model which should also be considered as an input information source), thus it might be acceptable that it will sacrifice some performance/accuracy. But the question is do we really want to do that? I do not see any benefits from doing this. If we think there are some domains where obtaining training data is expensive, then we say this model may be helpful. However, the thing is we still need a “good” teacher which should be trained on a large amount of dataset in this domain.\n\n2) Using a single image to train a classifier and gain reasonable accuracy sounds surprising; however, there are several recent papers finding that even a “randomly initialized neural network with frozen network” can also yield good classification performance (I list several references below). Given this background, the finding may not be as impressive as it sounds since in any way it has updated its network weights and take the advantage of the knowledge from the “good teacher”.\n\nRef1: K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, “What is the best multi-stage architecture for object recognition?” 2009 IEEE 12th International Conference on Computer Vision, Sep. 2009, pp. 2146–2153, ISSN: 2380-7504.\n\nRef2: H. Zhou, J. Lan, R. Liu, and J. Yosinski, “Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc.,\n2019.\n\nRef3: K. Sreenivasan, S. Rajput, J.-y. Sohn, and D. Papailiopoulos, “Finding Everything within Random Binary Networks,” 2021.\n\nRef4: S. Baek, M. Song, J. Jang, G. Kim, and S.-B. Paik, “Face detection in untrained deep neural networks,” Nature Communications, vol. 12, no. 1, p. 7328, Dec. 2021.\n\nRef5: G. Kim, J. Jang, S. Baek, M. Song, and S.-B. Paik, “Visual number sense in untrained deep neural networks,” Science Advances, vol. 7, no. 1, p. eabd6127, Jan. 2021.\n\n\n\n\n\n\n/********************************** After Rebuttal******************************/\n\nThanks for your response. After reviewing the responses and the discussions, most of my major concerns have been resolved. I would slightly raise my rating. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster