Amortised Invariance Learning for Contrastive Self-Supervision
Authors: Ruchika Chavhan
Jan Stuehmer
Calum Heggan
Mehrdad Yaghoobi
Timothy Hospedales
Venue: ICLR 2023 poster (2023)

Abstract: Contrastive self-supervised learning methods famously produce high quality transferable representations by learning invariances to different data augmentations. Invariances established during pre-training can be interpreted as strong inductive biases. However these may or may not be helpful, depending on if they match the invariance requirements of  downstream tasks or not. This has led to several attempts to learn task-specific invariances during pre-training, however, these methods are highly compute intensive and tedious to train. We introduce the notion of amortized invariance learning for contrastive self supervision. In the pre-training stage, we parameterize the feature extractor by differentiable invariance hyper-parameters that control the invariances encoded by the representation. Then, for any downstream task, both linear readout and task-specific invariance requirements can be efficiently and effectively learned by gradient-descent. We evaluate the notion of amortized invariances for contrastive learning over two different modalities: vision and audio, on two widely-used contrastive learning methods in vision: SimCLR and MoCo-v2 with popular architectures like ResNets and Vision Transformers, and SimCLR with ResNet-18 for audio. We show that our amortized features provide a reliable way to learn diverse downstream tasks with different invariance requirements, while using a single feature and avoiding task-specific pre-training. This provides an exciting perspective that opens up new horizons in the field of general purpose representation learning.

Published as a conference paper at ICLR 2023
AMORTISED INVARIANCE LEARNING FOR CONTRASTIVE SELF-SUPERVISION
Ruchika Chavhan's Jan Stuehmer Calum Heggan', Mehrdad Yaghoobi', Timothy Hospedales' 'University of Edinburgh. "Karlsruhe Institute of Technology, 'Heidelberg Institute for Thcoretical Studies *Samsung AI Research Centre, Cambridge, "Work done while at Samsung Al. Correspondence: R. @ ed. ac..ac.uo t.hospedalesfed..cc ac.uk
ABSTRACT
Contrastive self-supervised learning methods famously produce high quality transferable representations by learning invariances to different data augmenta- tions. Invariances established during pre-training can be interpreted as strong in- ductive biases. However these may or may not be helpful, depending on if they match the invariance requirements of downstream tasks not. This has led several attempts to learn task-speciff invariances during pre-training, however, these methods are highly compute intensive and tedious train. We introduce the notion of amortised invariance learning for contrastive self supervision. In the pre-training stage, we parameterize the feature extractor by differentiable invari- ance hyper-parametee that control the invariances encoded by the representation. Then, for any downstream task, both linear readout and task-specific invariance requirements be efficiently and effectively learned by gradient-descent We evaluate the notion of amortised invariances for contrastive learning over two dif- ferent modalities: vision and audio, two widely-use used contrastive learning meth- ods in vision: SimCLR and MoCo-v2 with popular architectures like ResNets and Vision Transformers, and SimCLR with ResNet-18 for audio. We show that our amortised features provide a reliable way to learn diverse downstream tasks with different invariance requirements, while using a single feature and avoiding task- specific pre-training. This provides an exciting perspective that opens up new horizons in the field of general purpose representation learning.
INTRODUCTION
Self-supervised learning has emerged driving force in representation learning, as it eliminates the dependency on data annotation and enables scaling up to larger datasets that tend to produce better representations (Ericsson al. 2022b). Among the flavours of supervuuervi contrastive learning has been particularly successful in important application disciplines such as computer vi- sion (Chen et al. 2020c Caron et al., 2020 Zbontar al. 2021), medical AI (Azizi et al. 2021 Krishnan et al., 2022), and audio processing (Al-Tahan & Mohsenzadeh) 2021). The key common element of various contrastive learning methods is training representations that are invariant to par- ticular semantics-presert preservinn input transformations (e.g., image blur, audio frequency masking) that are applied synthetically during training. Such invariances provide strong inductive bias that can improve downstream learning speed, generalisation, and robustness (Geirhos et al.) 2020).
major vision motivating self sulervupervi research has been producing a general purpose represen- tation can be learned once, albeit at substantial cost, and then cost-effectively re-used for different tasks of interest. Rapidly advancing research (Chen et al.) 2020c Caron et al. 22200) /hoonar al.) 2021), as summarized by various evaluation studies (Azizi et al. 2021; Ericsson et al. [202] 1), shows progress towards this goal. If successful this could displace the "end eend-to-end supervised learning for each task' principle that has dominated deep learning and alleviate its data annotation cost.
However, this vision is not straightforward to achieve. In reality, different tasks often require mutu- ally incompatible invariances (inductive biases). For example, object recognition may benefit from rotation and blur invariance; but cose-esttmation and blur-estimation tasks obviously prefer rotation and blur equivariance respectively. Training feature extractor with any given invariance will likely

---

Published as a conference paper at ICLR 2023
harm some task of interest, quantified recently Ericsson et al. (2022a). This has led to work on learning task-specific invariances/auggmenatiion for self-supervisiin with meta- veta-gradient (Raghu ct al. 2021) or BayesOpt (Wagner et al. 2022). which is extremely expensive and cumbersome; and training feature ensembles using multiple backbones with different invariances (Xiao et al. 2021] Ericsson al. 2022a), which is also expensive and not scalable. In this paper we therefore raise the question: How can we learn a single general-purpose representation that efficiently supports a set of downstream tasks with conflicting, and a-priori unknown invariance requirements?
To address these issues we explore the notion of amortized invariance learning in contrastive self- supervision. We parameterise the contrastive learner's 's neural architecture by set of differentiable invariance hyper- parameters, such that the feature extraction process is conditioned on particular set of invariance requirements. During contrastive pre-training, sampled augmentations correspond to observed invariance hyper-parameters. By learning this architecture on a range of augmentations, we essentially learn a low-dimensional manifold of feature extractors that is paramaterised by de- sired invariances. During downstream task learning, we freeze the feature extractor and learn new readout head as well as the unknown invariance hyperparameters. Thus the invariance requirements of each downstream task are automatically detected in way that is efficient and parameter light.
Our framework provides an interesting new approach to general purpose representation learning by supporting a range of invariances within single feature extractor. We demonstrate this concept empirically for two different modalities of vision and audio, using SimCLR Chen et al. (2020a) and MoCo Chen et al. (2020c) representative contrastive learners; and provide two instantiations of the amortized learning framework: a hypernetwork-
Ha et al. (2017) approach for ResNet CNNs, and a prompt learning approach for ViTs Dosovitskiy et al. (2021). We evaluate both classification and regression tasks in both many-shot and few-shot regime. Finally, we provide theoretical insights about why our amortised learning framework provides strong generalisation performance.
2 RELATED WORK
Invariance Learning Invariances have been learned by MAP (Benton et al. 2020), marginal likeli- hood (Immer et al. 2022), BayesOpt (Wagner et al. (022). and meta learning (Raghu et al. 2021]- -where gradients from the validation are backpropagated update the invariances or augmenta- tion choice. All these approaches are highly data and compute intensive due to the substantial effort required to train an invariance at each iteration of invariance learning. Our framework amortises the cost of invariance learning so that it is quick and easy to learn task-specific invariances downstream.
Invariances in Self-Supervision Self-supervised methods (Ericsson et al. 2022b) often rely on contrastive augmentations (Chen et al.] 2020c). Their success has been attributed to engendering in- variances (Ericsson al., [021] Wang & Isola, 2020] Purushwalkam & Gupta 2020) through these augmentations, which in turn provide good inductive bias for downstream tasks. Self-supervision sometimes aspires to providing a single general purpose feature suited for all tasks in the guise of foundation models (Bommasani al. 202I). However, studies have shown that different augmenta- tions (invariances) are suited for different downstream tasks, with no single feature being optimal for all tasks (Ericsson al. 2022a and performance suffering if inappropriate invariances are provided.
This leads to the tedious need to produce and combine an ensemble of features (Xiao et al.) 2021
Ericsson et al. 2022a), to disentangle invariance and transformation prediction (Lee et al 202
or to costly task-specific self- self-supervised pre-training (Raghu et al., 2021 Wagner et al. 2022 Our framework breaths new life into the notion of self-supervised learning of general purpose represen- tations by learning parametric feature extractor that spans an easily accessible range of invariances, and provides easy support for explicit task-specific invariance estimation of downstream tasks.
Self-Supervision in Audio and Beyond The design of typical augmentations in computer vision benefits from a large collective body of wisdom (Chen et al.) (2200) about suitable augmentation- s/invariances for common tasks of interest. Besides the task-dependence (e.g., recognition vs pose- estimation) of invariance already discussed, bringing self-supervision to new domains with less prior knowledge- such as audio - often requires expensive grid search to find good augmentation suite to use All Tahan & Mohsenzadeh 2021; Wagner et al. 2022). where each step consists of self- followed by downstream task evaluation. Our framework also benefits these
supervised pre-traiming situations: we can simply pre-train once with a fairly unconstrained suite of augmentations, and then quickly search for those augmentations beneficial to downstream tasks in this modality.
2

---

Published as a conference paper at ICLR 2023
3 METHODOLOGY
3.1 PRE-TRAINING
Features and Invariance Descriptors We begin by denoting a large unlabeled dataset available for pre-training by D' 1=1 where n, is the number of samples available in the raw dataset. Contrastive self-supervision typically trains a feature extractor h((z) that bakes in invariance to single pre-defined set of augmentations. We introduce the concept of invariance descriptor that denotes whether a parameterised feature extractor h(r; i) should be invariant or sensitive to K possible factors of variation. This is a vector E [0, 1] K over K possible transformations, where i and & = 0 indicate invariance and sensitivity to the kth factor respectively. We denote the set of binary invariance descriptors descriptors by I, where | =
Learning an pearriiicee-prrmaterriin feature extractor Every unique invariance descriptor i can be paired with a corresponding combination of stochastic augmentations, which are denoted by A. To learn our invariance-prramatrried encoder Iew i) we extend the standard contrastive self- supervised learning paradigm. At each iteration, contrastive learning, we sample an invariance descriptor i which can thus be considered observed. We then the corresponding augmentations to generate two views of the same example for invariance descriptor i, denoted by Te, and "x,-
Similarly, set of N negative samples denoted by Xs, (0 ) are also augmented using the same augmentations, i.e. invariance descriptor i.
Like all contrastive learning methods, a network projection head g+(+) projects representations the feature space in which contrastive loss is applied. Following the convention introduced in MoCo-v2 CCen et al. 2020c), the two views i7,, and 2%,- are considered as input for query qA, and positive key KA, representations respectively. A set of encoded samples form the keys of a dictionary denoted by KA,, = ( 1 k R2, ..... Eq. shows the forward propagation pipeline of the invariance encoder backbone to generate the query and keys of K...-
qA: = (bm((((),
t go (
R = go (N (72, i))
Both SimCLR and MoCo-v2, employ the contrastive InfoNCE loss Oord et al. (2018). In SimCLR variants, negative keys are from the same batch. while MoCo-bar based methods maintain negative keys as a queue. Finally, for particular augmentation operation A,, we formulate the InfoNCE loss as:
cxp
(qm, ktt,TT
Lcontrastive
(grriistiiiieegg (ga,s Kin.) log [KA, ==
qr. k2//TT
exp
where 7 is temperature hyper- parameter. In the pre-training stage, the invariance encoder and the projection head are trained using the contrastive loss governed by the contrastive method employed.
arg min ve
Ceontrastive (gA, KA.)
(3)
I
NET
In practice, we randomly sample an invariance descriptor from I for each batch and w, & are learned for corresponding Ccontrustive (In,; K.,.). ). The ferred to a downstream task.
encoder h (a:; i) is then trans-
3.2 DOWNSTREAM TASK LEARNING
We next consider set of downstream tasks Turget that may have different opposing and priori unknown invariance requirements. We denote the training data available for a downstream task & E Trarget Dr {0f v 1 In the downstream task training stage, we employ the learned parametric encoder h (=;-) learned from the pre-training to encode data as h hx;x;;), For each downstream task follow the linear evaluation protocol by learning prediction head &,, but extend it by also learning the corresponding task-wise invariance vector àg. Thus the predicted output given an invariance hyper-paree parameter i,
(Øt, h" (x; it)). For each task t, we find the optimal invariance
We exclude the where all hits correspond to 0, implying that augmentations are applied.
3

---

Published as a conference paper at ICLR 2023
hyper-parameters &n and prediction heads & by minimizing the task-specific loss of the training set, : 2i arg min Se 74g
Ty
Cllor 1 (1 (x);
Quantised Invariance Learning We remark that invariance parameters learned for downstream tasks are continuous vectors i E [0, 1]K in our model. In the previous pre-training phase, all ob- served occurrences of i are discrete E {0, 1) However, during downstream learning continuous values learned which can represent continuous degree of invariance. Nevertheless, later than there are learning theoretic benefits for modeling i as members of a discrete set. To exploit this, while retaining the ease of continuous optimisation for in downstream task learning, we can simply quantize to a desired number of bits ö* = (((";b) where Q(=; b) is the quantization operator that quantises each element of into b-bit representation.
will show
3.3 ARCHITECTURES
We next describe architectures i() capable of supporting wwwaarinnc-paraaaattroo feature encod- ing for ResNet CNNs and VIT transformers.
Hyper Netee: To incorporate differentiable invariances, parameterise the ResNet50 backbone in the form hypernetwork Ha et al. (2017). conditioned on an invariance descriptor. Previous work generating ResNet-50 parameters using hypernetworks (2018) is tailored for super- vised learning on small-scale dataset like CIFARÍÓ. This architecture relies on multiple forward passes through the hyperettork architecture to generate single convolutional kernel, which leads to prohibitively slow pre-training with constrastive learning on large datasets like ImageNet. Thus, we develop different hypernetwork architecture that can generate weights of full ResNet50 with single forward pass of the hypernetwork. This is easier to optimise and faster to train for contrastive learning. Details about the architectures are provided in the supplementary material A.1.I I.
Crompt-ViT's It is well known that ViTs are difficult to train, and extremely hyperparameter sen- sitive, especially for contrastive learning as discussed in Chen al. (202T) While we were able to successfully learn invariance paramaterised ViTs with hypernetworks analogous to those described for ResNet above, these were even harder to train. We therefore developed an alternative approach based on prompt learning that was easier to train. Specifically,
invariance vectors embedded by two-layer MLP network denoted by (prompe()) (-) and then appended after ViT input tokens from the corresponding task. Therefore, features from image h(:;) i) = ViT([CLS, E(x), (proomp( where (zx) denotes the image tokens with added position embedding. Thus invariance preferences are treated the same as image and class tokens. The invari- ance prompt guides the feature encoding of VIT as it is passed through all the attention and MLP layers together with the image tokens. Further details are given in the supplementary material A.1.2
extracted with desired invariance as
EXPERIMENTS: COMPUTER VISION TASKS
We evaluate our proposed framework on two widely-used contrastive learning methods: SimCLR and MoCo-v2 and with ResNet and VIT architectures.
4.1 AUGMENTATION GROUPS
Most contrastive learning studies use suite of augmentations consisting of standard data aug- mentation strategies like random resized cropping, horizontal flipping, color jitter, gaussian blurring, etc. We consider two cases of treating these as K independent invariances, and grouping several augmentations into single invariance.
Grouping Augmentations For simple analysis and ease of comparison to prior work, we conduct experiments by grouping the augmentations into K = 2 groups suggested by Ericsson et al. (202a)). The default set of augmentations have been divided into two groups called Appearance and Spatial augmentations. Spatial augmentations (crop, flip, scale, shear, rotate, transform) those that mainly transform image spatially while Appearance based augmentations (greyscale, bright- ness, contrast, saturation, hue, blur, sharpness) those that mainly act the pixels of the image,
4

---

Published as a conference paper at ICLR 2023
augmenting its appearance. Thus we amortise learning Appearance, Spatial, and default (combina- tion of Sppearance+Spatiaa) augmentations in a single feature extractor. During training, we assign invariance hyperparameters 2-way binary vectors i = [1, 11, i [1, 0] and [0, 1] for default, Spatial and Appearance based augmentations respectively.
perform amortised learning among the five default augmentations to learn invariance to all possible combinations of these augmentations. Every com- bination of augmentation is specified by a K 5-way binary vector, indicating which augmentation is switched on. Since SimCLR Chen et al. (2020a) draws two augmentations out of the entire set, we exclude the invariance descriptors that indicate that less than two augmentations have been applied. Thus, 26 unique invariances are encoded into a single backbone.
Individual Augmentations In this condition,
4.2 IMPLEMENTATION DETAILS
Pre-training Datasets: We perform self-supervised pre-training for VIT-B and ResNet50 on the 1.28M ImageNet training set Deng et al. (2009) and ImageNet-100 (a 100-category subset of Im- ageNet) following Chen et al. (2021) and Xiao et al. (2021) respectively. Both the models
pre-trained for 300 epochs with batch size of 1024.
Learning rates and Optimisers: We find that the optimal learning rate and weight decay obtained by Chen et al. (2020c) work well for Hyper-ResNets and Prompt-ViT's, in both SimCLR RR and MoCO- v2/v3 experiments. We follow the optimization protocol in Chen al. (202T) and use the AdamW optimiser along with learning rate warm-up for 40 epochs, followed by cosine decay schedule.
MLP heads: Following Chen al. (2020b) 2021), we use 3-layer projection head and 2-layer prediction head for both ViT-B and ResNet50. The hidden and output layers of the all projection and prediction MLPs of both architectures is 4096-d and 256-d respectively.
Loss: Following the protocol in Chen et al. (2221) for training ViT-B models under the MoCO-v3 framework, we abandon the memory queue and optimize Prompt-ViT models on the symmetrised contrastive loss (Caron et al.] 2020 Grill et al. 2020). However, we observe that the symmetrised contrastive loss and discarding queues is not effective for training Hlper-ResNetth models. There- fore, for ResNet models we stick to the MoCO-v2 framework, where memory queue is used. This leads to fair comparison between different baselines for both the architectures. Additionally, we maintain separate queue for each type of invariance encoded in the Elyper-ResNet50 model so that augmented keys corresponding to the same invariance are used for contrastive loss.
Downstream Evaluation: In our framework, evaluation downstream tasks consists of supervised learning of the task-specific invariance hyperparameters and a linear classifier using backpropaga- tion. We use the Adam optimiser, with batch size of 256, and sweep learning rate and weight decay parameters for each downstream dataset based on its validation set. We apply weight decay only on the parameters of the linear classifier.
Downstream tasks: Our suite of downstream tasks consists of object recognition on standard bench- marks CIFAR (Krizhevsky et al. 2009), Caltech101 (Fei-Fei et al.) 2004), Flowers (Nilsback)
& Zisserman 2008). Pets (Parkhi et al. 2012). DTD (Cimpoi et al. 20014), CUB200 (Wah et al. 2011). as well as set of spatially sensitive tasks including facial landmark detection on 300W
SAG (2016), and CelebA Liu et al. (013), and pose estimation on Leeds Sports Pose Johnson &
Everingham 2010). More details can be found in A.2
Few shot downstream evaluation We also evaluate the pre-trained networks on various few-shot learning benchmarks: FC100 (Oreshkin et al.] 2018). Caltech-UCSD Birds (CUB200), and Plant Disease Mohanty et al. (2016). We also show results for few-shot regression problems on 300w, Leeds Sports Pose and CelebA datasets, where we repeatedly sampled small subset of training data to create a low-shot training set. More details can be found in A.2
Competitors We compare our Amortised representation learning framework based on Hyper- ResNet and Prompt-VIT as backbones (denoted AI-SimCLR, etc) with default SimCLR and MoCo alternatives. We also compare with SimCLR and MoCo variants that we re-trained to specialise in Appearance and Spatial group augmentations, (denoted A- and S-), and two state of the art ensemble- based alternatives LooC (Xiao et al. 202T) and AugSelf (Lee et al. 202T), with the same pretraining setting us.
5

---

Published as a conference paper at ICLR 2023
Paut
Purpler
apperrance spotial
SON
Figure I: Left: Radar plots comparing strengths of 5 Appearance (left) and 5 Spatial invariances (right) for default ResNet50-MoCov2 and ViT-MoCov3 (green dots) vs our corresponding amor- tised models. By varying runtime invariance parameter, a single feature extractor can provide one or other group of invariances on demand. Right: While training was performed discrete invari- ance vectors (cornrrs), we can interpolate smoothly between different invariances (vertical axis) by continuously varying the invariance parameter (horizontal axes).
4.2. RESULTS
Can we successfully amortise invariance learning? We investigate this question for Appear- ance and Spatial invariance groups explained in Section 4.1 Specifically, we follow Ericsson et al. (202aa) in measuring the invariance between original and augmented features using the cosine sim- ilrrity in a normalised feature space between input images and their augmented counterpart. A high cosine similarity indicates high invariance, and vice-versa. Using this measure, compare the invariances learned by (1) default MoCo model, and our single amortised model fed with (2) Ap- [0, 1J) hyperparameter. We also compare two MoCo models re-trained from scratch to specialize on the Appearance and Spatial invariance groups, following Eriesson al. 2022a). Full details are given Tables and 7 in the appendix, and a summary of the first three comparisons in Figure [[left) From the figure we can see that: (i) Our amortised invariance learner can access comparably strong invariances the default model where desired (convex hull of both colored regions is similar to the dashed region). More impor- tantly, (ii) while the default model is fixed and cannot change any (in)variances without retraining. amortised invariance learner can dial-down invariances ance model increases sensitivity to flipping. We will show that this is a useful capability for general purpose representation when downstream tasks contain both pose estimation and classification, that require conflicting spatial invariances.
pearance (i [1, 0) hyperparameter, and (3) Spatial
command. For example, the Appear-
Can We Continuously Scale Invariance Strength? Recall that during training, model observed three invariance hyperparameter vectors {(0, 1), [1, 0j, 11]. However, once trained we can easily interpolate along the 2d-manifold of Appearance and Spatial group invariances. Figure [ rightt illustrates this for amortised ResNet50 and grayscale invariance within the Appearance group. We can see that interpolating between Spatial [1, 0] and Appearance [0, 1] parameter vectors leads to a corresponding smooth increase in grayscale invariance. We expect that if a downstream task benefits from grayscale invariance it will essentially perform gradient ascent on this surface reach the Appearance [0, 1] corner. We show similar plots for Spatial amortised ResNet50 and amortised variants of ViT in the appendix. (Figure
Does amortised invariance learning benefit downstream tasks? We next evaluate suite of downstream learning tasks. We compare (1) default SimCLR and MoCo models, (2) variants re- trained to specialise in Appearance and Spatial invariances (Ericsson et al.) (022a) (See C), (3) For MoCo CNNs, we compare LooC (Xiao et al.) 2021) and AugSelf (Lee et al. 202 I), which are ensemble based approaches to supporting multiple invariances, (4) Fine-tuning (FT) and fine-tuning with time constrained to match our method (FT-"), and (5) Our amortised framework including linear readout and invariance learning for each downstream task (denoted AI-SiCCL//Mooo etc).
From the results in Table I we can see that (i) our Hypernet approaches for ResNet50 have com- parable better performance compared to baselines. This is especially the case for the regression

---

Published as a conference paper at ICLR 2023
Parteen wwrd dravd set
Investaree aloee coumirann
Figure 2: Learned invariance vectors for AI-SimCLR based on ResNer50 models for the suite of downstream tasks when using 2- 2-invariance (right) and 5-invariance (left) condition.
300w
CelebA
CIEARTO
CHEARTO
Flowers
101
Avg.
Kank
MoCo
85.3
58.7
61.0
$4.6
61.6
K2.4
17.3
641.5
70.1
32.2
67.8
1.3
McCo-
61.6
65.1
80.4
75.8
71.5
MaCo
S. MoMo
74.7
46.1
49.0
68.8
41.7
51.6
53.8
62.5
56.3
31.3
53.4
5.5
A-MoCo
78.5
49.2
62.0
75.0
37.4
18.8
43.8
56.3
39.6
17.6
47.8
5.5
AugSelr*
77.3
63.9
77.0
85.3
63.9
85.7
78,9
66.2
73.5
37.0
70.9
LoaCC"
39.6
Al-MoCo
90.0
65.2
81.3
74.0
41.4
72.7
SimCLR
51.3
54.5
61.0
81.8
61.4
66.6
71.9
$1.5
69.1
37.9
60.7
4.0
SimCLR
SimCLR FT
57.6
58.1
64.0
83,7
63.4
68.1
73.4
53.8
70.4
39.8
66.2
2.0
S-SiCLR
28.4
48.4
54.9
75.0
60.7
61.8
$2.8
43.8
42.9
31.3
50.0
5.3
A-SimCI R
67.6
58.2
72.5
61.5
40.0
50.0
43.7
25.0
29.8
18.8
46.7
5.2
Al (2)
87.1
65.5
68.8
37.5
67.1
A1-SimCIRR R (5)
88.0
63.0
77.2
83,9
63.1
68.3
74,2
53.7
69.5
38.6
68.1
1.6
Table 1: Downstream performance of MoCo (top) and SimCLR (below) based ResNet50 models
pre-trained on ImageNet-100. The first three results columns are regression tasks (R², %); the last six are classification (accuracy, %). numbers taken from Xiao et al. (2021). + numbers for classification taken from Lee et al. (2021), rest are our runs. The invariances learned by our amortised models AI-SimCLR(2). AI-SimCLR(5) for each downstream task indicated in Figure 2 (right and left respectively).
datasets, where the results are often dramatically better than baselines. EG: +21% for hypernet vs baseline on MoCo based CelebA, +12.7% for hypernet vs AugSelf on MoCo based 300w.
To understand how these strong results are achieved, we report the Appearance and Spatial in- variances estimated by our framework in the case of SimCLR for each downstream task in Fig- ure right). The results show that while the invariance strengths learned vary continuously across tasks, there is systematic difference: Classification tasks (right seven) prefer stronger invariances overall, and a greater Spatial- than Appearance invariance. Mcanwhile regression tasks (left three) prefer more moderate invariances with similar strength tendency towards Appearance invari- ance. Invariance parameters learned for AI-MoCo models are shown in Table 10 in the appendix.
Can scale to learning more invariance?? We next repeat the above experiments for the case of SimCLR, using five invariances (Sec 4.1) rather than the two considered before. The results in Table for AI-SimCI.R(2 AI-SimCL.R(5) show that indeed using more distinct invariances is possible, and often improves performance compared to the case of two invariances.
Does amortised invariance learning benefit few-shot learning tasks? To answer this question we focused on MoCo-v2 CNN models trained on ImageNet100, For classification tasks, we followed Lee al. (202T) Xiao et al. (202T) in sampling C-way K-shot epsiodes from the target problem and training linear readouts (and invariances for the case of our method) for each episode. For regression tasks we repeatedly sampled 5%, and 20% to generate low-shot training sets. From the results in Tables 2 we can see that our AI-MoCo framework usually performs better than all competitors, with substantial margins in several cases, for example 12% and improvement on default MoCo for 5-way/5-shot flowers classification dataset, and 10% R² improvement on AugSelf in CelebA 20% low-shot regression. Invariances hyper-parameters for few-shot classification tasks and few-shot regression tasks are shown in shown in Table 13 in the appendix. Results for few-shot learning on audio are given in the Supplementary.
Does amortised invariance learning benefit downstream tasks when using transformer archi- tectures? The experiments described so far have used ResNet50 CNNs. To investigate the impact of amortised invariance learning on transformers, apply framework ImageNet- 1k pretrained

---

Published as a conference paper at ICLR 2023
Methods
CUB
Flowers
FC 100
Plant Discase
300h
LS Pose
Calbb
Rank
(5
(5.5)
(5.
(5.5)
(5,5)
(5
(5. 5)
0.40
0.2
0.05
0.2
0.00
0.2
MoCo
56.9
50.1
51.2
60.3
3.2
52.5
66.5
73.5
40.8
38.2
45.0
49.0
56.0
37.2
48.2
4.3
S-MaCo
36.0
46.5
69.6
70.0
26.0
35.6
64.4
76.8
12.1
20.7
38.9
43.7
30.2
44.2
5.5
A-MoCo
24.1
34.7
24.4
36.4
21.0
21.2
21.6
36.6
36.5
40.8
43.2
48.2
43.8
66.3
5,3
AugSelf
574
76.0
85.6
35.0
48.8
87.8
42.0
51.8
53.8
60.1
53.2
66.3
2.1
709
Al-MCCo
45.0
58.0
76.7
85.7
37.4
48.4
72.60
89.1
49.2
57.9
333
62.0
56.
76.0
1.1
Few-shot classification and regression accuracy (%, R²) of our AI-MoCo based on ResNet50 models pretrained on ImageNet-100, Values are reported with 95% confidence inter- vals averaged over 2000 episodes on FC100, CUB200, and Plant Disease. (N, K) denotes N-way K-shot tasks. For regression tasks (300w, LS Pose, CelebA), we report downstream performance for different splits with train proportion given by More details given in Tab. I and 12
Table 2:
300w
CIFARIO
CIFARIOD
Flowers
Caltech
Avg.
MoCo-v3
81.6
59.1
18.0
94,8
63.4
87.7
83.5
61.1
79.4
26.5
71.5
2.7
MoCo-v3-
85.7
S -MoCo-3
50.9
48.1
72.0
74.3
62.1
78.9
72.1
59.3
73.0
21.4
61.2
4.6
MoCo-v3
78.9
59.2
MoCa
89.0
67.0
84.0
93.8
63.7
87.5
81.3
60.4
81.5
28.2
73.7
1.9
Table 3: Downstream performance of Al-MoCo-v3 based ViTs pretrained on ImageNet- 1k for many-shot classification (accuracy, %) and regression (It², %).
MoCo-v3 VIT-B, and repeat the many-shot and few-shot learning experiments above with this ar- chitecture. From the results in Table and few-shot results in Table we can see that the outcomes are broadly consistent with those the CNNs. Our AI-VTT performs comparably or better than conventional MoCo-v3 VITs. In many cases it performs substantially better, e.g., 18% and 19% improvement R² for 5% low-shot regression on 300w and LSP respectively. Invariance prompts learned for AI-VIT models shown in Table 10 in the appendix.
How do amortised invariances compare to Fine-Tuning? Our framework provides a new operat- ing point between linear readout and fine-tuning in terms of expressivity and efficiency for adapting to a downstream task. To explore this point, we compare our framework with fine-tuning variants that update 1, 2, 3 all 4 ResNet blocks in Figure The figure also shows the Pareto front of accuracy vs parameter/clock time cost. AI-MoCo with ResNet50 only requires updating essentially as few parameters linear readout, and much less than FT; and it provides greater accuracy for less time compared to FT. In both cases AI-MoCo dominates a section of the Pareto front.
5 THEORETICAL ANALYSIS
We finally provide some theoretical analysis to give insight into the value and empirical efficacy of our approach when applied novel downstream tasks not seen during pre-training. Specifically, for downstream tasks, our amortised invariance framework admits the generalisation bound below.
Theorem 5.1. For I I-Lipschitz loss function, L. taking values in [0. M). if for all & we have that ||ol| < B and (z)]] < X. the following holds with probability at least -
1 nh j=1
2/2cXB /o:
In((41/8)
Ex (C(0 v/)
C(05, u3)
+ 3M
2n
where I = {0, 13a is the space of possible invariance hyperpurameters and is the number of classes.
The proof is in the appendix along with corresponding existing theorems for (i) simple linear readout and (ii) fine-tuning alternatives. Comparing Theorem [5T with the alternatives shows that the over- fitting behaviour (generalisation error) of our approach when applied novel tasks scales similarly to conventional linear models (i.c., with BxB). However due to the parameterised feature extractor have potential to obtain a much better fit the training data (reduced first term RHS) a comparatively limited cost to complexity (third term on RHS). In contrast, improving train data fit by end-to-end fine-tuning of deep network to adapt task-specific invariance requirements
8

---

Published as a conference paper at ICLR 2023
Methods
CUB
Flwwers
FCT000
Plant Discase
300w
LS Pose
CalbA
Rank
(5.
(5.5)
(5.
(5.5)
(5.1)
(5.5)
(5.1)
(5,5)
0.05
= 0.2
0,05
0.2
0.06
0.2
MoCo-va
688
818
913
79.2
1500
2.5
MaCovv FT
88.8
72.4
72.2
Al
65.6 772
84.2
92.2
67.8
79.8
76.8
84.6
33.1
74.6
48.0
56.0
35.4
72.3
1.3
Table 5: Few-shot classification and regression accuracy (%, R²) of the AI-MoCo-v3 for ViTs pretrained on ImageNet- 1k. Values are reported with 95% confidence intervals averaged over 2000 episodes on FC100, CUB200, and Plant Disease. (N, K) denotes N-way K-shot tasks. For regression tasks (300w, LS Pose, CelebA), we report downstream performance for different splits with train proportion given by Full results for regression and classification are in Tat 13 and Tab 14
500W
Out Flat
Peturmnc
Vafomuue
FT:4
(1=-7)
Ing Umm))
(dockttmal
Figure 3: Comparison of linear readout (LR), AI-MoCo (Ours), and fine-tuned MoCo (FT) in terms of parameter update cost (left) and clock time cost (mid, right) vs performance. Here, we present the pareto front for two datasets: 300W (regression) and CUB 200 (classification). (x) denotes corresponding FT baseline run with time constraint while (+) denotes FT baseline for intermediate iterations. We present pareto fronts for more datasets in the appendix.
induces third complexity term that depends exponentially on the depth of the network due to the product of norms of the weights in each layer (Bartlett et al. 2017 Golowich et al.) [0188 Long & Sedghi 2020). This analysis requires working with a discrete space of invariances, while most
of our experiments have worked with continuous invariance learning for simplicity. As mentioned in Section 3.2, we can trivially discretize our estimated invariances. and we show that invariance discretization does not substantially affect our results in Table
Bringing all this together, we illustrate the significance by instantiating all the terms in Theorem 3.Jeemm55ppp. to compute the guaranteed worst case gen- cralisation error for our model and the alternatives in Table 4 The FT baseline has vacuous bound with that only guarantees an error rate >> 1. The linear model can provide an error rate guarantee, while our AI-MoCo provides stronger guaranteed error rate thanks to the trade- off discussed above. Altogether this analysis shows that our framework provides an exciting new operating point in the bias- vlas-variance trade-off compared to the established paradigms of linear classifiers and end-to- end deep learning. See also See for further details and discussion.
Method
Guaranteed
Error (L)
LR-MoCo
0.78
AI-MoCo
0.67
FT-MoCo
>>
Table 4: generalistaion error for CIFAR-10
Guaranteed
6 CONCLUSION
We have introduced the concept of amortised invariance learning, and shown that a manifold spanning multiple invariances (up to K dimensional our experiments) can be pre-learned by single feature extractor of either CNN or VIT types. Our amortised extractor provides an effective general purpose representation that can be transferred to support diverse down- stream tasks that cannot be supported by single conventional contrastively trained representation. With our parametric representation, each downstream task can rapidly select an appropriate invari- ance in efficient and parameter- classification and regression tasks in the fnw-and many-shot regimes. Amortised invariances provide an exciting new direction of study for general purpose features suited for diverse downstream tasks.
way. This leads to strong improvements across range of

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
Ruchika Chavhan was supported by Samsung Al Research, Cambridge.
REFERENCES
300 faces in-the-wild challenge: database and results. Image and Vision Computing, 2016. 300-W, the First Automatic Facial Landmark Detection in-the- Wild Challenge.
Haider Al-Tahan and Yalda Mohsenzadeh. Clar: contrastive learning of auditory representations. In International Conference on Artificial Intelligence and Statistics, 2021.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesslingam, Simon Kornblith, Ting Chen, Vivek Natarajan, and Moham- mad Norouzi. Big self sulf-supervised models advance medical image classification. In International Conference on Computer Vision, 2021.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research.
Peter L Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalizee margin bounds for neural networks. Advances in neural information processing systems, 2017.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. In Advances in Neural Information Processing Systems, 2020.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. ArXív, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong supervsse learners. Advances in neural information pro- cessing systems, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. Arxiv, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. International Conference on Computer Vision, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and paltern recognition, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural audio synthesis musical notes with wavenet autoencoders. arXiv, 2017.
10

---

Published as a conference paper at ICLR 2023
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self- supervvee models transfer? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2021.
Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks. In British Machine Vision Confer- ence, 2022a.
Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M. Hospedales. Self-supervised rep- resentation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, may 2022b.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training exam- ples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, 2004.
Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Favory, Jordi Pons, and Xavier Serra. General-purpose tagging of freesound audio with audioset labels: Task description, dataset, and baseline. Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop, 2018.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2021.
Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo. In Proceedings of the 21 ACM International Conference on Multimedia. Association for Computing Machinery, 2013.
Robert Geirhos, Jörn-Hearik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory. PMLR, 2018
Henry Gouk, Timothy Hospedales, et al. Distance-based regularisation of deep networks for fine- tuning. In International Conference on Learning Representations, 2021.
Jean-Bastin Bastinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaroo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 2020.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. In International Conference Learning Representations, 2017.
Calum Heggan, Sam Budgett, Timothy Hospedales, and Mehrdad Yaghoobi. Metaaudio: few-shot audio classification benchmark. ICANN, 2022.
Alexander Immer, Tycho FA van der Ouderaa, Vincent Fortuin, Gunnar Rätsch, and Mark van der Wilk. Invariance learning deep neural networks with differentiable laplace approximations. Advances in Neural Information Processing Systems, 2022.
Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC. Aberystwyth, UK, 2010.
Rayan Krishnan, Pranav Rajpurkar, and Eric Topol. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-avw aware self-supervision. Advances in Neural Information Pro- cessing Systems, 2021.
11

---

Published as a conference paper at ICLR 2023
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, 2015.
Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations, 2020.
Andreas Maurer. A vector-contraction inequality for rademacher complexities. In International Conference on Algorithmic Learning Theory, pp. 3-17. Springer, 2016.
G. Mittal. Hypemeeworkk(github)) 2018.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image- based plant disease detection. Frontiers in plant science, 2016.
A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: large-scale speaker identification dataset. In INTERSPEECH, 2017.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing. IEEE, 2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. Arxiv, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 2018.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012.
Karol Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd
ACM international conference on Multimedia, 2015.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: In- variances, augmentations and dataset biases. Advances in Neural Information Processing Systems, 2020.
Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, and David K Duve- naud. Advances in Neural Information Processing Systems, 2021.
Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirmeister, Samuel Müller, and Frank Hutter. On the importance of hyperparameters and data augmentation for self-supervised learn- ing. First Workshop of Pre-training: Perspectives, Pirfalls, and Paths Forward at International Conference on Machine Learning, 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-22 dataset. Technical report, 2011.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere supplementary material. International Conference Machine Learning. 2020.
Ross Wightman. Pytorch image models.
http:::/ggtthbb..omm/v.gggttgt
-ytorch-image-monn
2019.
Tete Xiao, Xiaolong Wang, Alexei Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL IMPLEMENTATION DETAILS
A.1.1 HYPER-RESNETSO
Hypernetworks Ha et al. (2017) are small neural networks that are optimised generate parameters for another network (also called as the 'main' network'). Typically, they are designed as 2-layer MI.ss conditioned on an embedding vector that describes the entire weights of given layer. There- fore, hypernetworks are excellent choices for conditioning neural network weights and inculcate desired properties in features. In this paper, we use a hypernetwork conditioned on invariance de- scriptors to generate ResNet50 parameters that exhibit desired invariances in features.
In Ha et al. (017)) the network architecture is assumed to consist of kernels with the same dimensions. Taking this into consideration, this work also in- troduces& a formulation to generate weights of deep convolutional network architecture that consists of kernels of varying dimensions. Most ResNet variants consist of kernel dimensions typically multiples of a standard size. For example, in ResNet50, each kernel dimension is a multiple of 64. To modify hypernetworks for this architecture, Mittal (0118); [Ha et al. (2017) propose generate kernels of a basic size of 64 x 64 x 3 x 3 and concatenate multiple basic kernels together to form larger kernel. Each basic kernel is generated by a unique z embedding, and generate larger kernels, more such embeddings are passed to the hypemnetork. For example, if we need to generate a kernel of dimension 64m x 64n x x 3, m X n learnable embeddings are passed to the hypemet- work and then all these kernels are concatenated to form a kernel of the desired size. Therefore, this architecture relies on multiple forward passes through the hypernetwork to generate single kernel, which slows down pre-training with contrastive learning less memory but is slower to train due to the recursive manner of weight generation. To tackle this, we adopt block-specific hypernetwork design that can generate weights for kernel in a single forward pass.
Previous Implementation for ResNet50:
large datasets. This method requires
Single forward-pass Hypernetwork: Unlike Ha al. (2017). we propose share embedding vector generated from invariance descriptors for each residual block in the ResNet architecture. More specifically, we learn block-specific embeddings as a function I - Rd d is the dimension of the embedding vector and I denotes the set of invariances descriptors. We change previous imple- mentation by abondoning a single hypernetwork to generate all kernels of the architecture. Rather, we propose to learn one hypernetwork for each kernel in a residual block given the embedding for that particular block. Thus, this architecture requires only one forward pass through particular hy- pernetwork. Pytorch code has been shown in [ Weights for each kernel are generated in fan-out fashion, thus trading off between more memory and speed. However, we observe that this model is faster and easier to train.
A.1.2 PROMPTVIT
is well known that ViTs are difficult to train, especially for contrastive learning as demonstrated by Chen et al. (202T). Downstream performance and training stability with ViT's for contrastive learn- ing is highly sensitive to the choice of hyper-parameters. In our initial experiments, we empirically realise that a ViT-hypernetwork architecture similar to the one designed for ResNet 50 is extremely difficult to train. Therefore, we adopt a prompt learning based approach to amortize invariance learn- ing in ViTs, such that a range of invariances can be embedded into generated features. Specifically, binary invariance vectors are projected onto an embedding space by two-layer invariance projector network denoted by (prompt(() and then appended after ViT input tokens from the corresponding task. In practice, we observe that averaging over multiple invariance projectors followed by concatenation better for optimization and leads to a more encoding of a range of invariances.
Therefore, features with desired invariance are extracted as VIT([CLS, E(TA.)). Apromp (i))), where A, is the set of augmentations corresponding to invariance descriptor and E(IA,)) denotes the im- age tokens with added position embedding. Code snippet in 2 In the vision transformer, invariance prompts are treated the same as image tokens and the class token as they pass all attention layers and MLP layers. The knowledge required to learn desired invariances is further utilized through the attention mechanism, where image and class tokens can extract suitable information from the invariance tokens.
13

---

Published as a conference paper at ICLR 2023
Model
Resized crop
H Hiippp
V Vilip
Scale
Rotation
Translation
Gmysccle
Brightness
Contratt
Saturation
Hue
A-Baseline
0.03
0.65
0.26
0.09
0.71
1.00
0.88
0.86
0.98
0.99
A-AI
0.04
0.48
0.55
0.22
0.14
0.74
0.82
0.72
0.82
0.81
0.74
S-baseline
0.25
0.96
0.26
0.70
0.95
0.65
0.43
0.35
0.60
0.42
S-AT
0.26
0.75
0.73
0.72
0.76
0.80
0.35
0.34
0.36
0.41
0.28
De-Baseline
0.19
0.95
0.75
0.63
0.11
0.92
0.96
0.77
0.79
0.94
0.93
De-AI
0.28
0.81
0.81
0.62
0.15
0.76
0.81
0.73
0.78
0.75
0.79
Table 6: ImageNet- 1000 pre-trained AI-MoCO based on Hyper-ResNer50 (300 epochs) evaluated on invariances to transforms on ImageNet- 100 validation images
100
Model
Resized crop
H
Vhlip
Scale
Rotation
Translation
Gmyscale
Brightness
Contrast
Saturation
Hue
A-Baseline
0.17
0.75
0.58
0.45
0.32
0.89
0.95
0.81
0.83
0.91
0.90
A-AI
0.14
0.78
0.52
0.43
0.34
0.94
0.99
0.85
0.87
0.90
0.94
S-baseline
0.26
0.38
0.65
0.99
0.79
0.53
0.96
0.96
0.89
S-AI
0.21
0.94
0.84
0.55
0.66
0,96
0.72
0.41
0.49
0.68
0.61
Baseline
0.45
0.97
0.96
De-AI
0.21
0.97
0.79
0.55
0.36
0.97
0.94
0.63
0.74
0.91
0.84
Table 7: ImageNet-pre-traieee AI-MoCo-v3 with Prompt-ViT (300 epochs) evaluated on invariances to transforms on 1000 ImageNet validation images
Initialization trick: We experimentally observe that the Prompt- Vit model, when initialized with the conventional Kaiming initialization seemingly unstable in the first few epochs of pre-training. Since exploring principled techniques for weight initialization for the Prompt-ViT model is out of the scope of this study, we adopt a simple yet effective trick for faster and stable training. We ini- tialize the Prompt- ViT model with an ImageNet k-MoCO pretrained model obtained from Chen et al. (202T), which is trained to be invariant to the default set of augmentations. We then fine- tune the entire model along with /promp (-) and learnable class tokens encode range of multiple invariances.
A.2 DATASETS AND EVALUATION METRICS: COMPUTER VISION TASKS
We evaluate our framework on seven object recognition and three spatially-sensitive sensitive regression tasks. For all datasets, train-test split protocol is adopted from Lee al. (2021); Ericsson et al. 2022a).
For linear evaluation benchmarks, randomly choose validation samples in the training split for each dataset when the validation split is not officially provided. For CIFAR 10, Flowers, CelebA, DTD, Pets, CUB200, sets for cross-validation and training the linear classifier and invariance hyper-parameters. For Cal- tech101, we randomly select 30 images per class form the training set and we test on the rest of the dataset. For CIFARIO, DTD, CIFAR100, and CUB200 we report top-1 accuracy and for Caltech101, Pets Flowers, mean per-class accuracy.
Classification tasks:
use the existing test sets and their train (or train+val)
Regression tasks: On 300W, we use both indoor and outdoor sets and join 40% of the images in each set to form a test set. Similarly, for Leeds Sports Pose we use 40% of the images for testing. For both these datasets, the rest of the images are used for cross-validation and fitting the final model and invariance pyper-parameters. On 300W and CelebA we perform facial landmark regression and report the R2 regression metric and for Leeds Sports Pose we perform pose estimation and report R2.
Few-shot benchmarks: We use the meta-test split for FC100, and entire datasets for CUB200 and Plant Disease. For evaluating representations in few-shot benchmarks, we simply perform ridge re- gression using the representations generated using the paariince-hyperprramee samples without finetuning and data augmentation in a N-way K-shot episode.
and NK support
B AMORTIZED INVARIANCES FOR AUDIO
Setup To complement the computer vision experiments, we also demonstrate our framework's ap- plication to invariance learning for audio representations. To evaluate this we build upon the recently released Meta Audio Heggan et al. (2022) benchmark, which provides few-shot recognition tasks across range of audio datasets. We follow the data encoding and neural architecture of Meta Audio,
14

---

Published as a conference paper at ICLR 2023
Dataset
AI-SimCL.R
SimCLR
Learned Invariance
Accuracy
Accuracy
FDSKaggle18
[74,
41, 41, 41, 40, 36]
40.99 = 0.21
39,98 = 0.20
VoxCeleb1
[57. 43, 44, 49, 46, 49]
43,01 = 0.20
36,91 = 0.19
ESC-50
[62, 49, 42, 59, 39, 39, 44]
62.14 = 0.20
60.85 = 0.20
Nsynth
176, 24, 39, 48, 40, 45, 24]
89.49 = 0.14
88.06 0.15
Table 8: Quantitative results for downstream few-shot audio recognition tasks from Meta Audio
Heggan et al. 2022) benchmark suite, after training Al-SimCLR for ResNet18 on FSDSOK.
using spectogram encodings of the audio clips and ResNet18 feature extraction on the spectograms. In this case we conduct SimCLR-ban R-baeed pre-training on FSD50k Fonseca et al. (202T), and then deploy the learned representation for few-shot recognition on four of the target datasets provided by Meta Audio Heggan et al. (2022), covering diverse audio types including environmental noises (ESC-50) Piczak Piczak (215)) instruments (NSynth) Engel et al. (017), human voices (VoxCelebl) Na- grani et al. (2017), and eclectic mixture (FSDKaggle11) Fonseca et al. (2018). We use the K audio augmentation types suggested by CLAR Al-Tahan & Mohsenzadeh (202I), meaning that our feature extractor spans 120 unique invariance combinations. More details on datasets and augmen-
tations in A.2 and C.I
Results The results in Table compare the downstream few-shot recognition performance of our AI- Al-SimCLR with vanilla SimCLR They show that we outperform conventional SimCLR, with sub- stantial margins in some places (E.g., for VoxCeleb). We also report the learned invariance param- eters (in %) for each target dataset. We can see that (i) they deviate substantially from full vector of that corresponds to SimCLR, (ii) they deviate substantially from each other, demonstrating the value of our ability to efficiently tune the representation to the invariance requirements of any given task.
B.1 DATASETS AND EVALUATION METRICS: AUDIO
We show the performance of our framework on four target datasets covered by Meta Audio namely:
ESC-50 Piczak (2013) This dataset consists of 2000 labeled samples of various environ- mental sounds categorized into 50 classes. Classes have been grouped animal sounds, natural soundscapes and water sounds, human (non-speech) sounds, interior/domestic sounds, exterior/urban noises with 10 classes per group. Audio clips provided in short 5-second format.
NSynth Engel et al. (2017) NSynth is a large scale dataset that consists of 306043 musical notes, each with a unique pitch, timbre, and envelope. These samples have been divided into 1006 classes formed on the basis of the source of sample.
|oxCeleb1 Nagrani et al. 2017): This datasets contains over 100,000 utterances for 1 1,251 celebrities, extracted from videos uploaded to YouTube.
FSDKaggle Fonseca et al. (2018): FSDKaggle2018 is composed of audio content col- lected from Freesound Font et al. (2013). This dataset consists of 11,073 samples divided into 41 classes from sounds originating from a wide range of real-world environments like musical instruments and human/animal sounds.
For all datasets, report 5-way 1-shot classification results in Table g
C AUGMENTATION POLICIES: COMPUTER VISION TASKS
Here we provide details about the augmentation policies used to train amortized versions of SimCLR and MoCo. We use the default suite of augmentation Ericsson et al. (202aa) that previous works have also employed. Spatial augmentation a set of spatial transformations including random resized crop and random horizonal flip. Appearance based augmentation is a set of appearance changing augmentations consisting random grayscaling, random color jitter and gaussian blurring. Finally, the default augmentations is combination of both Spatial and Appearance augmentations. In addition
15

---

Published as a conference paper at ICLR 2023
Methods
500w
1.5 Pose
CelebA
CIFARTO
CIFARTOD
Flowers
Callech 101
15115
Pets
CUB
Rank
MoCo-v3
81.6
59.1
78.0
94.8
614
87.7
83.5
61.1
79.4
26.5
54.1
2.1
A1-MoCo-v3
89.0
67.0
84,0
93.8
63.7
87.5
81.3
60.4
81.5
28.2
58.6
1.4
Al-MoCo-v3 (dise.)
85.7
64.0
81.6
91.5
62.9
8688
80.9
59.3
79.6
27.1
56.4
2.5
Methods
800w
CalchA
CIFARTO
CIARR
Caltec
CO1
Avg.
MoCo
85.5
58.7
61.0
84.6
61.6
82.4
77.3
64.5
70.1
32.2
58.9
2,5
90.0
65.2
AI-MoCo (disc.)
88.7
64.1
78.3
80.4
61.9
80.9
77.1
66.2
72.1
38.2
63.5
2.3
Table 9: Downstream performance of amortized models (Prompt- ViT and Hyper-ResNet after dis- cretization of learned invariances. Classification tasks report accuracy (%) and regression tasks It
(%)
Methods
300w
LS Pose
CelebA
CIFARIO
CIFAR100
Flowers
Caltech 101
DTD
Pets
CUB
ResNet50
[49,
[53, 54]
[43. 62]
195, 59]
[37, 89]
[100, 96]
185, 88]
[73. 681
[69,55]
[79. 77]
Prompt VIT
[67.66
[0, 54]
[35, 61]
[100, 100]
[55, 66]
[100, 100]
[100,
[83.0]
[100.0]
[0. 66]
Table 10: Invariance
learned by amortized MoCo models on a suite of downstream
tasks. Accuracies stated in the main paper in Table T
the default set, both MoCo-v2 Chen et al. and MoCo-v3 Chen al. (202T) also use the solarization to augment pre-training images. Thus, for fair comparison, we also choose to apply solarization as an augmentation for our MoCo experiments. It is categorized Appearance based augmentation.
Appearance (A-) and Spatial (S-) baselines are models pre-trained with the corresponding augmenta- tions only. Hence, they are capable of exhibiting only one type of invariance, while default baseline model is moderately invariant to both groups of augmentations.
C.1 AUGMENTATION POLICIES: AUDIO
We employ the augmentation strategies suggested in Al-Tahan & Mohsenzadeh (2021) for con- trastive learning with audio. Augmentations introduced this work are categorised as either fre- quency temporal transformations. The frequency transforming augmentations are Pitch Shift, Fade in, Fade out, and Noise Injection. Time masking, Time shift and Time stretching are the tem- poral augmentations used. We construct the 7-way invariance vector in same order.
D ADDITIONAL RESULTS
Discreted Invariances: In Ericsson et al. (2022a), invariance between original and augmented fea- tures was quantified using the Mahalanobis distance or cosine similarity in a normalised feature space. Low value of Mahalanobis distance and high value of cosine similarity indicates high invari- ance. Similarly, we investigate the invariances learned by amortized frameworks using the measure- ment protocol introduced in Eriesson et al. (0022a). In Table 6 and 7 show that the proposed amortized invariance learning frameworks indeed learn invariance to augmentations by measuring the invariances of feature vectors under input transformation.
To measure invariance to a particular set of augmentations, we pass the corresponding invariance descriptor the model and measure invariance between resulting features. For the sake of sim- plicity, we only report measured cosine similarity for Appearance (A), Spatial (S) and Default (De) models. We confirm that there is broad agreement between invariances learned by amortized and baseline models, hence confirming the validity of the concept of amortising invariances. A visual comparison of invariances measured for default MoCo and amortized model for Appearance and Spatial invariances is shown in Figure]leftt..
Continuous scaling of Invariance Strength: Similar to Section 4.2.1 we present qualitative visu- alization to demonstrate smooth interpolation between polar opposite invariances. We visualize this ResNet 50, Appearance Prompt- ViT and Spatial Prompt-Vis ViT in Fig- ure 4 To evaluate this curve for Spatial- model, the Spatial
interpolation for Spatial
[1,0] is passed through
the model and invariance is measured between original image and rotated image. Thus, we see that maximum invariance achieved at the respective corners for each model. The general trend is that
16

---

Published as a conference paper at ICLR 2023
Methods
300w
LS Pose
CalebA
Rank
= 0.06
6 0.1
= 0.2
= 0.05
8 0.1
8 0.2
8 0.05
8 0.1
= 0.2
MoCo
0.2
0.
50.1
0.12
0.3
60.3
0.7
03
52.
MoCo- FT
38.2
0.5
4..
0.3
51.0
56.0
07.
42.1
48.
4.1
S-MoCo
12.1
14.0+02
20.7
0.7
43.7
4.3
30,2
36,
44.20011
A-MoCo
36.5 0.3
07
40.
432
0.2
4.2
4.1
43.8 +
01.
6.3
4.3
AugSelf
42.0
45.7
0.5
518
53.8
0.3
0.3
60.1
0.8
53.2 0.1
0..
6.2
2.3
AI-MoCo
49.2 * 0.1
53.1 0.2
57.9 0.1
55.3
2 0.1
59.0 0.6
62.0 0.3
56.2 + 0.1
62.3s 0.3
76.0 0.5
1.0
Table 11: Few-shot regression accuracy (%) of the ResNet50 MoCo-v2 models pretrained on
ImageNet- 100. We test the models over 100 train-test splits given by and report average R score with 95% confidence intervals.
Piant Disease
Average
(5. 1)
(5.5)
(5, I)
15. 5)
(5, 1)
(5.5)
(5.
(5.5)
(5. I)
(5,5)
63.7
MoCo-Er FT
37.8
52.
66.5 0.3
0.1
0.2
40.8 0.2
6..0
8.3
48.7
61.7
4.5
S-MaCo
0.3
0.4
69.6 0.6
0.
0.2
35.6
64.4 0.8
76.8 0.8
49.0
57.2
4,9
A-MoCo
24.1
0.3
2.4
3..4
0.1
0.2
21.6
36.6
22.7
32.2
6,3
AugSelf
0.5
0.5
76.0 0.4
8..6
0.
48.8 0.4
71.8 0.5
8..
56.8
69.9
1.9
70.9
AL-MaCo
0..
0.2
76.7 0.2
88.7
0.2
0.
48.4
72.6
0.20 89.1
57.9
71.1
1.1
Table 12: Few-shot classification accuracy (%) of the AI-MoCo (Hyper-ResNed) models pretrained on ImageNet- 100. Values are reported with 95% confidence intervals averaged over 2000 episodes on FC100, CUB200, and Plant Disease. (N, K) denotes N-way K-shot tasks.
Methods
CUB
Flowers
FC 100
Plant Discuse
(5. I)
(5.5)
1)
(5.5)
(5. I)
(5,5)
(5,
(5.5)
MoCo-v3
65.8
7..0
83,
9..6
0.5
0,5
91.4 0,5
MoCo-v3 -F
6.1
66.7 0.3
812
0.2
6.2
0.14
87. 0.3
AI-MoCo-v3
6.76
0.4
0.2
9.2
0.2
7.5
76.8
84,6
Table 13: Few-shot classification accuracy (%) the AI-MoCo-v3 models pretrained on ImageNet- 1k. Values reported with 95% confidence intervals averaged over 2000 episodes on FC100, CUB200, and Plant Disease. (N, K) denotes N-way K-shot tasks.
Melhods
300w ==
LS Pose
CelobA
Rank
== 0.05
==02
== 0.05
==
==
== 0.05
=
McCo-v3
1
33.1
1.4
6.1
29.1
39.
5..3
13.
6.52
68.7
1.5
3.0
MoCa-v3
0.4
A1-MoCow33
0.5
45.2 0.1
7.
48.0 0.2
52.4
56.0
0.14
6..3
72.3 1.3300 0.5
1.0
Table 14: Few-shot regression accuracy (%) of the AI-MoCo-v3 models pretrained on ImageNet- 1k. We test the models over 100 train-test splits given by s and report average R² score with 95% confidence intervals.
Spatial- Hyper-ResNett0
Appearance- Prompt-VIT
Spatial Prompt-ViT
D. sopearance
D0 appeerance
appeerance spotial defaul
O spelial
0j spolial
11 dafiult
11 defaul
02
34
De
Figure 4: Qualitative visualisation of the smooth innterpolation between different invariances (ver- tical axis) by continuously varying the invariance parameter (horizontal axes) for Spatial Hyper- ResNet50, Appearance Prompt- VIT and Spatial Prompt-ViT.
17

---

Published as a conference paper at ICLR 2023
Methods
CUB
Flowers
FC 100
Plant Disease
(5. 1)
(5, 5)
(5, I)
(5, 5)
(5. 1)
(5. 5)
(5, 1)
(5. 5)
ResNet50
[47, 51]
[35. 51]
177, 75]
[95. 1]
[50, 49]
[50, 49]
[50, 49]
[50, 49]
Prompt-ViT
[50,
[50, 49]
[50, 48]
[50, 47]
[50, 49]
[50, 49]
[51, 51]
[51, 51]
Methods
300w
LS Pose
CelebA
0.05
=
= 0.2
= 0.05
8 = 0.1
8 0.2
= 0.05
8 = 0.1
8 == 0.2
ResNer50
[1]
[43, 52]
[43, 56]
[S1. 53]
[50, 56]
[49, 60
[48, 54]
[46, 56
[42, 60]
Prompt-ViT
51]
[48, 51]
[58, 52]
[45, 52]
[40, 51]
[50, 53]
[52, S7]
[54, 53]
[46, 57]
Table 15: Invariance hyper-parametent learned by amortized MoCo models on few shot classification and regression tasks.
A- or S- models display high invariance for corresponding input augmentation while default models shows a moderate invariance.
During downstream evaluation, we initialise the invari- with a vector of 0.5*. This implies that we provide moderate invariance to each type of invariance and let the model automatically descend to the invariances that are favored by the task. Invariance presented in Table [0 As expected, regression based tasks tend to slightly prefer Appearance aug- mentations while object recognition tasks prefer the set of default or Spatial augmentations. We notice that fine-grained classifcation tasks like CUB200 prefer Appearance augmentations.
Learned Invariance Hyper-paaametten ance
papper-prraeters learned for MoCo models for all downstream tasks are
Next, we present invariance hyper-parameters learned for few-shot regression and classification tasks in Table [5] Invariances are learned episode-wise and we present the invariance parameters averaged over all episodes. For few shot experiments it can be observed that classification tasks prefer slight Spatial or default invariance. For regression tasks, Appearance invariance prefered in most cases.
E COST ANALYSIS
Pareto Front Analysis: We previously argued that our proposed framework is essentially new operating point between linear readout and fine-tuning. In this section, we present more Pareto fronts in Figure 5 and 6 for parameter cost and clock time cost respectively. The time cost is the most visible cost the downstream user of a feature, and the parameter cost is important because of its connection to overfitting
will see in the Section F
For this analysis, we consider 2 regression datasets (300W, Leeds Sports Pose) and 3 classification datasets (DTD, CIFARIO, 10, and CUB200), In both Figure TigureBlande and the bottom right figure corresponds to the average over all the datasets. In Figure it is evident that AI-MoCo dominates the section of the Pareto front for all datasets. With respect clock time cost in Figure AI-MoCo dominates for both regression datasets and CUB 200. Even though our model does not fall on the Pareto front for CIFARIO 10 and DTTD, AI-MoCo dominates section of the Pareto front on average.
Downstream Memory requirements: We provide a comparison of memory requirement for down- stream training in Table [7 for both ResNet50 and ViTs. Our memory requirement higher than baselines for ResNet50 and significantly lower for Prompt-ViTs. This is due to the fact that Hyper- ResNe150 consists of more parameters than baseline ResNet50 whereas the Prompt-ViT is compa- rable to baseline ViT' in terms of both architecture and number of parameters.
Pre-training a one-off cost that is often expensive self-supervision, but is expected to be amortised across multiple downstream problems. Our method goes further by pre-training a model that represents i multiple invariances, indeed multiple combinations of invariances, to make them available for downstream models. We discuss the computational efficiency of pretraining our method and compare it with training one baseline model, and 2 baseline models Table 16 In this case, provide a comparison with A 5. Table [6] shows the time taken in GPU days Tesla V100-SXM2-3226
Pretraining:
for 300 pre-training epochs for ResNet50 and ViTs with ImageNet-100 and ImageNet- 1k respectively. The results show that pre-training model takes more time and memory than a standard MoCo model, but less than the baseline of pre-training 2 unique MoCos for each combination of invariances.
18

---

Published as a conference paper at ICLR 2023
Method
Ours
Baseline
RN50
VIT
RN50
2 RN50s
1 ViT
2 ViTs
Time (GPU days)
5.07
29.3
2.9
92.8
27.3
874.7
Memory (Gb)
446.4
481.6
134.4
134.4
476.8
476.8
Table 16: Comparison of pretraining cost for amortized MoCo models and baselines. We assume that the models are run serially where one baseline model will require the same memory as 2 base- line model. If we consider parallel training, then memory requirements for 2 baselines increases drastically while clock time remains the same.
Memory Ours
Baseline LR
FT all
FT -5%
-T-50%
FT -5%%
ResNet50
13690
1452
10930
7250
5238
4420
ViT
14156
1452
18916
17608
15976
14468
Table 17: Memory requirements (in Mb) for downstream training. FT - denotes the variants of finetuning where 5% of the number of layers/blocks is finetuned. In the case of ResNet50, FT 50% indicates that the out of 4 layers, the final two layers (3 and 4) are finetuned. For ViTs, FT 50% is the baseline where block 6 to block 12 of the architecture is finetuned. FT all denotes the finetuning baseline where the entire model is finetuned.
F THEORETICAL ANALYSIS
Generalisation Bound for Amortised Invariance Framework We first present the proof for The- orem 5.1 which gave the generalisation bound of our amortised invariance framework.
Proof. There one-to-one mapping between elements of the finite set of invariance hyperparam- eters and potential feature extractors, implying that there is also a finite number, [4] of potential fea- ture extractors for novel task. For a linear model coupled with pre-selected feature extractor, one can apply the bound on empirical Rademacher complexity for multi-class linear models developed by MMurerr (2016). Combining this with the standard Rademacher complexity generalisation bound (Bartlett & Mendelson) tell us that with probability at least 1 1-8 8
The
C(j w/) + 2V2cXB 3M In In(2/8")
Ex ( ((C("''''))
Σ
TI.
VTt
2n,
If one were evaluate the above bound using every possible choice of feature extractor in I, then the bound can only be guaranteed to hold with probably at least 1 - 7788,, due to the union bound. To compensate, on can instead instantiate the bound with 8' 8/| Substituting this definition for 8' yields the result.
Generalisation Bound for Standard Linear Models In comparison to our amortised invariance framework, the generalisation bound for standard linear readout is given by Eq. Note the differ- ence to Theorem 5.1 is the reduced third complexity term with no dependence on I, as well as the fact that the two models will have different empirical risk (first loss term), which will be less in the case of our model.
Generalisation Bound for Fine-Tuning It is also interesting to consider of the state of the art bounds for fine-tuning. In practice, there are no bounds for deep networks that take into account both fine-tuning and residual connections. We therefore concern ourselves with the main result of Gouk et al. (202T), given in the theorem below, which provides a result for a class of deep networks without residual connections.
F= {(. v) + f(F)) [1 S Bj, [W S Bj, IW, - Will S D₃}. Theorem F.1. For all 5 E (0, 1), the expected risk of all models in F is, with probability 1 5, bounded by
E [C(( < Σ c(f(E.).
4/1o(22)ccX
25=1 B,
D3 I1
2B,
+3M
1og(2/6) 2m
m
19

---

Published as a conference paper at ICLR 2023
DTD
69
68-
Our
FT-3,4
2T-2.3,4
Our GA
Performance
04-
FT
62
FT-2.33
66-
Baceline- LB
ITAA
Baseline -R
Beseino LR
6.
0.5
1.0
1.5
2.0
1.0
1.5
Number parameters updated (1e+7)
CUB 200
CIFAR10
Average
72
Our
FT-a
Baseline
FT-
1T-2.
FT-2,4
FT-2
Performance
IT-14
Bosebee LR
3.0
2.0
2.0
Number of parameters updated (1e+7)
Figure 5: Comparison of linear readout (LR), AI-MoCo (Ours), and fine-tuned MoCo (FT) in terms of parameter update cost vs performance. Here, we present the pareto front for five datasets.
Posa
DTD
300W
Oue
Oun
Pofommnce
7T-34
IT-2
35
log (clonc time]
CUR 200
CFAR1D
Average
075
PT.24 IT-23
775
Dosl UR
2.5
15
log tcock timee)
Figure 6: Comparison of linear readout (LR), AI-MoCo (Ours), and fine-tuned MoCo (FT) in terms of clock time cost vs performance. Here, we present the pareto front for five datasets. (x) denotes corresponding FT baseline run with time constraint while (+) denotes FT baseline for intermediate iterations.
20

---

Published as a conference paper at ICLR 2023
D6and (003)
(tuns)
0.25
0.50
0.25
0.23
4.00
2.00
(weigh decol!
Figure 7: Generalization bound and empirical test enror for range regularisation strengths. Here, the X-axis corresponds to the log of the weight decay parameter.
where mn is the number of training examples, e is the number of classes, T, Rd, || S X, and it is assumed the loss function is 1-Lipschitz and takes values in [0, M).
This bound will obviously be vacuous ( 1) for any reasonable neural network, due to the ex- ponential dependence on the depth of the network. In contrast, we demonstrate that our bound is non-vacuous in common experimental setting based on CIFAR-10.
Detailed Bound Instantiation Results To expand upon the results briefly reported in Table 4 we present Figure which plots the guaranteed test error and empirical test error for range of regu- larisers. We can see that our model always has a tighter bound and sometimes has better empirical еrror.
G CODE SNIPPETS
Code for generating parameters of BottleNeck block of the ResNet50 architecture with our modified design
Modi fied downsample layer
c1 class DownSampleCons (nn. Module)
def _init_ (self, stride, padding, Arss_dilation
super super()._iii___l self. stride
seld.padding padding
self. self.first_dhlation = first_dilation
def (srwa, x, w_down):
11
return (x, w_down, stride self.stride,
12
padding self. dilation self. first dilattdddlaaio
13
14
Modifies Bettleneck layer for Reanet50
class Bottleneck (nn Moniullee))
16
expansicn
17
def init
18
self, inplanes, planes, stride=1, downsample-None,
cardinality=l, 644 ffirstll dilation=1,
20
first_dilatioonNonn,, act_layer=m.RRilu norm_layer-on Batchhorm2d, actn_layer-None, aa_layer=None, brop_block- drop srop_path-None)
21
super (Bottllenckkkkk, self) init
24
25
width = int (math. floor (planes (base_width / 64)) + cardinality) first_planes = width // reduce_first
26
outplanes planes sel felf.expaa self.expansion
first_dilation first_dile dilttddiltton dilation
21

---

Published as a conference paper at ICLR 2023
layer is not Nene and (stride == first dilation
30
!- dilation)
31
self use_aa
32
self.stride = stride
snF.fiirttddiattion = first_dilation
self.cardinality cardinality
35
36
self.emb = nn Lnnein (2, 16, bias-False)
37
Hypernet for convl sel self.hypernetl HyperNetwerk (kernel 1,
40
in_size inplanes, out_ first_planes
41
42
Hypernet for conv2 self self. hal.hhyperet22 (pperNetwozk(keemee) size 3,
44
in_ first_planes, out_ size width)
45
46
Hypernet for conv3
self hypernet3 in_size width, out_size outplanes)
(keereetttockkkeeele__ee 1,
49
50
Hypernet for downsampling it downsample is not Nene:
(kernel self hele.hypernet_ce down HyperNetwork (kernel size 1, in size inplanes, out size outplanes)
53
54
55
self.bnl - norm_layer (first_planes)
self act_layer (inplace-True)
(width) self.bn2 norm norm_layer (withh)
58
self.drop_
drop_block() if else nn. In.ddeiiity()
not None
59
60
sclf.act2 (t__aayyr
self.aa create_aatca_llyer, channels=widthh,
62
stride=strids, enable=use_aa)
63
64
self flf.bb3 - norm_ (cttpaay
sel f. felf.bn_deel bbb_doon norm_layen (cutplanees)))
67
self.se create attn (attn Jaye,, outplanes)
68
self act act_ laye self dolf.downsample = downsample
71
self.stride stride
72
sclf.dilation dilation
13
path = drop_path
Forward function with imarriance-hppeergaaaeteet input def (srward(self, x, inv): shortcut =
76
77
18
B1 Block-specific embedding emb_ self.embl (inv)
81
82
# One forward pass to generate first kernel wl self hypernet1
c.conv2d wl)
X sclf.bn1 (x)
86
x sclf.act1 (x)
87
One forward pass to generate first kernel
w2 self hylFf.hyperet22eeme (emb_)
90
x conv2d w2, stride if self.use_as else dilation,,
diletiinnself..irss...dd dilation, croupssself.cardinee
22

---

Published as a conference paper at ICLR 2023
94
self. (x)
95
X sclf.drop_bloce (x)
96
X sclf,act2
self.aa (x)
99
# One forward pass to generate first kernel
100
w3 self, 3 (emb_)
X F cxn w3)
self.bn3 (x)
104
if self.se is not None:
105
x self.se
if self delf.drop path is not Nene:
108
x self delf.drop... (athoppaaahaa
109
downsample [0] is convolution and downsample [1] is norm if self. downsample is not Nene:
w_down sel hyw.rnet_ddwwn (emb_)
113
shortcut self felf.downd dol..downsmmpee/ (self. downsample(0) (shortcut,
x_down)
+= shortcut
self
117
return X
Listing I: Code for ResNet50 Bottleneck modified to incorporate hypernetworks. Basic code
block has been adapted from the Pytorch Image Models Library Wightman (2019)
Code for concatenation of invariance tokens in Prompt-ViT
Concatenation ion of class and invariance tokens with invariance projectors
embed_dim 768 For ViT-Base
self. salf.inv_at fc nn ModuleList([
nn Sem.Seunntiall (nn. Linear (2, self.embed dim), nn. nn. ReLU(), nn. embedd.
self.embed_dimg for i in range 1)6)
We only show concatenation and pesition embedding of the tiaioa_transformer class from MoCo-v3 httpe://github.ce
10
def _pos_embed (self, inv)
12
calculate average over multiple invariance tokens
13
inv tokens torch, ([self. ine (ttfiiivfclll (inv) vview 1, embed dim)
14
for i in range (0, 256) ,, dim = 1)
inv tokens hov_tokens.mmer (1) unsqueesseeeeeeeeeee (1)
inv_tokens inv_tokens.expane
shape -1, -1)
17
if selk.els_token not None:
18
x corch.eat (sclf.cls_tokoel expand (x. shaphp (0), -1, -1), x). dim=1) x_cls selffpos_eemed
terch.cat ([x, ([x, inv_tokens), dim
1)
Dropout
22
return polf ddoo (x)
Listing Code snippet for invariance token based prompting for ViTs. Basic code block has been
adapted from the Pytorch Image Models Library Wightman (2019)
23


Reviews:
[{"review_id": "xVaznyUGoO", "reviewer": "Reviewer_gbta", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to condition the final representation on an invariance descriptor. By doing so, the representation can be adaptively adjusted for target tasks that may require different kinds of invariance. ", "summary_of_the_review": "Overall a good and novel paper. Thorough analyses are conducted to support the claims made in the paper and the final results indeed outperform vanilla SSL methods on various downstream tasks that may potentially require different kinds of invariance.", "strength_and_weaknesses": "## Strength\n1. Thorough analysis to support the claims made in the paper. For example, Fig 1 shows that the final representation indeed adjust adaptively according to the invariance descriptor. Fig 2 also demonstrates that the model can identify the invariance required intrinsically for each downstream task.\n\n1. Strong transfer results to various downstream tasks. This also implicitly confirms the (somewhat intuitive) hypothesis that each downstream task requires different invariance.\n\n1. The authors demonstrate the generalizability of the proposed method on different (1) archs (ResNet and ViT), (2) SSL methods (MoCo and SimCLR), (3) downstream tasks (10 of them spanning across regression and classification tasks), and (4) modalities (vision and audio)\n\n## Weakness\n\n1. It is unclear why training the model with the vanilla contrastive loss in Eq 2 and 3 can encourage the model to produce the final representation conditioned on the invariance descriptor. Without any additional loss or special training techniques, the model can simply ignore the invariance descriptor and degenerate to vanilla SSL models like SimCLR or MoCo. The authors should highlight this part in the Method section.\n\n1. Poor figure quality. For example, they are not vectorized figures and the resolution is not good. The fonts are too small and the shape is distorted (eg. Fig 2)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qdp1_rnn8T8", "reviewer": "Reviewer_SKik", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, authors propose a method to dynamically learn/adjust what invariances a self-supervised representation should be robust to such that it can cater to various types of downstream tasks which may have conflicting invariances. Towards this end, they propose a hyper-network based learning method which can learn the proper invariances for a given target task. Experiments show that learning task-specific invariances (via various stochastic augmentation policies) works better compared to the original SSL models (SimCLR and MoCo) which uses a fixed augmentation policy for all tasks. The proposed method also outperforms existing works from the literature which tries to solve the same problem (e.g. LooC). Authors  perform experiments with two family of architectures - ResNets and ViT and also two task modalities - vision and audio and show improvements on most, if not all cases.", "summary_of_the_review": "Overall, I think the paper takes a new approach to solve a well-known and important problem for self-supervised representation learning. The proposed method intuitively makes sense at a high-level and shows promise in empirical evaluations. However, I'd like for more clarifications regarding the details of the algorithm, the computational cost as well as some explanations regarding its difference with respect to a few other works from the literature. At this point, I am leaning towards marginally below acceptance threshold but I will be willing to increase my score if authors provide satisfactory responses to my comments above. \n\nUpdates after Rebuttal\n---------------------------\nUpdated score by one point based on all the discussions between the authors and myself. ", "strength_and_weaknesses": "Strengths\n========\n\nNarrative\n-----------\nThe paper is written in a clear way - starting from the problem formulation and why it is important, followed by clearly motivating their proposed approach and positioning it against the baselines and other relevant methods from the literature and finally showing experimental results with different architectures and task modalities. \n\nPracticality\n-------------\nApart from only considering the formulation or theoretical properties, authors also paid attention to make sure their proposed algorithm can be trained without a lot of tweaking and training can scale to standard SSL benchmarks like ImageNet-1K. Towards this, authors modified the hyper-network formulation for ResNets. For ViTs, they proposed a different prompt tuning based formulation as they found hyper-networks + ViTs are harder to train. There are also additional tricks mentioned in the paper for ViTs which are useful for reproduction. I appreciate the effort to make sure the proposed algorithm can be used by practitioners.\n\nCoverage\n------------\nAuthors perform experiments using multiple classification and regression tasks for visual recognition, few-shot classification and also performed experiments in the audio domain. As mentioned before, they experimented with two different family of architectures - ViT and CNN. I believe the empirical setup is strong and helps to properly evaluate the contributions of this work. \n\nWeakness\n================\nAdditional details on test-time adaptation will be helpful\n-----------------------------------------------------------------\nMore details on the test-time adaptation procedure will be helpful as that is the core contribution of this work. Along with that, some explanation of the basic hyper-network formulation will be useful as well. I have gone through the code provided in the appendix (for BottleNeck layer of ResNet) but I am unclear as to how this is plugged into the end-to-end optimization procedure. For example, I am not sure what is the shape of the invariance hyperparameters for ResNet for every block and how are those parameters getting updated. \n\nConnecting to other types of work\n----------------------------------------- \nAlthough there has not been any work in terms of learning proper augmentations or learning a differentiable masking in the particular context of SSL, there has been prior works independently. For example, [this paper](https://arxiv.org/abs/2104.04282) learns the optimal data augmentation policy by differentiable search. [AutoAugment](https://arxiv.org/abs/1805.09501) does a similar thing using reinforcement learning. How does the proposed hyper-networks based method compare to such methods to find the right data augmentation for the given task? Can these algorithms be used to search for the augmentation strategy instead which will be in sync with the optimal invariances to learn each task?\n\nAdditional Questions\n=================\n\n* While trying to learn the right augmentation strategy for a given task, I do not see a justification why a hyper-network based strategy was used as opposed to using other ideas from the literature like straight-through estimator or using a standard meta-learning algorithm. \n* While adapting to the downstream task, in Eq. 4, what will happen if we rather minimize the loss on a validation set like meta-learning? Why did you choose to minimize it on a train set instead of validation?\n* Although there has been considerable effort made to reduce the computation time, still, adding hyper-networks require making three additional forward pass calls for each block in a residual network. How much additional time does these calls cause? Assuming they cause 3x, that is equivalent to performing downstream adaptation with three different augmentation strategies as hyperparameters. In that case, there is no practical advantage of this algorithm as it also tries to choose between three combinations (dorsal, ventral and both).\n* What is the time difference between ResNet + hyper-network and ViT with prompt tuning? Also, my understanding is that these additional calls for hyper-networks are only needed during downstream adaptation and not pre-training - is that correct? \n* When you mention ViTs are harder to train with the hyper-networks formulation, does it mean it needs more hyperparameters tuning or is the training itself unstable and leads to NaN and other problems?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ETwYzgXW8Y", "reviewer": "Reviewer_oGLZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Current SSL representations are pretrained to be invariant to a prespecified fixed set of augmentations. This paper provides a simple way of pretraining representations that can be modified to be invariant to any subset of the pretraining augmentations. They essentially learn a function(al) that maps a set of augmentations to a pretrained encoder invariant to those augmentations. For CNNs this function maps augmentations to weights of the network, while for transformers this simply corresponds to appending the index of augmentations to the input sequence. \n\nThey evaluate their representation on transfer learning benchmarks by training both the linear probe and the set of augmentations to be invariant to.  They show significant gains compared to linear probing of standard SSL and  previous baselines that aim to learn representations that are not fully invariant.", "summary_of_the_review": "The overall idea of the paper is very nice and simple. I believe that it could be useful to practitioners and the research community. My main concern is that the authors only consider a single application for their method (using the data from downstream task to update the parameters) but do not compare or discuss the obvious baseline for this setting: fine-tuning. I thus do not think that it is currently ready for publications. I am happy to improve my score if the authors address this concern, for example by doing one of the following:\n- empirically showing advantages compared to fine-tuning (better accuracy, computational efficiency ...)\n- explaining why fine-tuning is not the right baseline\n- talking and ideally showcasing about other potential applications ", "strength_and_weaknesses": "**Update**: the authors updated the manuscript to address many of my concerns. In particular, improving the baselines and discussing the computational complexity of their method. I am thus increasing my score 5->6.\n\n-------\n**Strenghts**\n- **really cool idea** the idea of learning a functional from desired invariances to representations is really nice and I think will be valuable for the community. I actually think that the authors undersell it's usefulness. Eg even when downstream practitioners do know what invariances they want in their downstream tasks, they can now get the desired representations without any retraining. The current paper puts all the emphasis on learning task-specific invariances, but this is only one specific application (and arguably not the best as I say below).\n- **simple method** the proposed method is very simple for ViTs, it could thus be easily incorporated in SSL and has the potential of being impactful assuming that the method is indeed useful (see weakness about baselines below). Note: for CNNs it's more complex which will likely hinder it's usefulness.\n\n**Weaknesses**\n- (addressed in rebuttal) **Unsuitable baselines: what are the advantages compared to fine-tuning?** my main issue with all the experiments is that I don't think that any of the selected baselines are fair comparisons because those only train the linear probe. Take for example the case of ResNet50, by tuning over the augmentation index i (which directly impacts the weights of your model) you are essentially finetuning all the parameters of your model but comparing to baselines that only train the linear probe. Looking at the Mocov3 results (table 4 vs original paper) it seems that your method performs much worst than full fine-tuning. So what are the advantages of your method compared to full fine-tuning? Is it computational or sample efficiency? If so you should compare to fine-tuning methods under the same computational or sample budget (ie only doing a few passes or doing few shot fine-tuning) [^1]. Your generalization bounds try to give a potential explanation for the gains in terms of better sample efficiency, but this has to be empirically validated given that those bounds are well known to be loose. \n- (addressed in rebuttal)  **lack of discussion on scalability** specifically :\n    - how efficient are the hypernetworks? what is the increase in the number of parameters and training time?\n    - it seems that you restrict yourselves to very few augmentations. Does your method easily scale to many more augmentations?  eg the 14 augmentations set used in autoaugment? Pretraining the model for essentially all typical image transformations would be much more useful as it would allow practitioners to have a fine-grained choice on the desired augmentations.\n\n\n[^1] Note that the advantage of linear probing vs finetuning is very large in terms of computational gains: because you can prefeaturize your dataset once before training the linear probe, making it orders of magnitude more efficient. In your case, you still have to encode every example at every step. I thus doubt that your hypernetwork is more computationally efficient that fine-tuning a RN50. Even for ViT the gains will likely be minor given that the bottleneck is usually encoding.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8UQxKREXMU", "reviewer": "Reviewer_SHkE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to learn more generalizable representations. To this end, this paper proposes amortized invariance learning, which learns augmentation-specific invariance by parameterizing the feature extractor by invariance hyperparameters. This paper shows the superiority of amortized invariance learning under various transfer learning benchmarks with both CNNs and ViTs.\n", "summary_of_the_review": "Although I feel the methodological novelty, both the empirical and theoretical results are not convincing. Hence, I vote for rejection.\n", "strength_and_weaknesses": "Strengths\n- The problem of interest, learning augmentation-relevant information, is an important problem in self-supervised learning literature.\n- The proposed idea, learning a hyper-network to learn augmentation-specific invariance, is interesting.\n- When using CNNs, the proposed method outperforms existing SSL methods that use linear evaluation for downstream tasks.\n\nWeaknesses\n- This paper should be compared with fine-tuning, not linear evaluation. This is because the full propagation for whole parameters is necessary to optimize an invariance descriptor. Therefore, its time requirement is the same as fine-tuning. I think fine-tuning can perform better than the proposed method.\n- In ViT experiments, the gain over the baseline (MoCo-v3) is marginal.\n  - Prompt-ViT outperforms MoCo-v3 on 6 of 10 downstream classification tasks and 4 of 8 few-shot tasks (10 of 18, i.e., 55% in total). This means Prompt-ViT achieves a similar performance to the baseline.\n- Theorem 1 is hard to understand and seems to be meaningless.\n  - What is $\\hat{y}$? What is the definition of the prediction $\\hat{y}$?\n  - The authors should describe details in the proof of Theorem 1. The sentence \"Our result follows from using the union bound to optimise over the choice of feature extractor.\" is not enough for formal proof.\n  - I cannot see the meaning of Theorem 1. The theorem does not tell the benefit of the proposed method. I think the theorem is just a well-known generalization bound (except the third term). The bound can be obtained in any case: fine-tuning, full-invariance learning (e.g., MoCo), etc. IMO, the authors should (theoretically) discuss an advantage over the baseline (linear evaluation using representations obtained from full-invariance learning).\n- The writing of this paper can be improved.\n  - Confused notation $i$: an invariance descriptor or an instance index (e.g,. see Eq (2)).\n  - Confused notation $\\mathcal{K}$. In the denominator of Eq (2), $k_{A_i}^i$ is not defined. I suggest using $\\sum_{k\\in\\mathcal{K}}\\exp(q_{A_i}\\cdot k/\\tau)$.\n  - Confused notation $\\mathcal{D}^t$: pretraining or downstream task dataset.\n  - When using MoCo, how to compute the key representations $\\mathcal{K}$ for an invariance descriptor $i$?\n  - I suggest mentioning that $i$ is a continuous vector when defining $i$.\n  - The authors should describe formal details of Hyper-ResNet in the main manuscript.\n  - Figure 1 is too blurred. Suggest using high-resolution images.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster