Evaluating Representations with Readout Model Switching
Authors: Yazhe Li
Jorg Bornschein
Marcus Hutter
Venue: ICLR 2023 poster (2023)
Keywords: Representation Learning
Evaluation
Expert Switching
Minumum Description Length

Abstract: Although much of the success of Deep Learning builds on learning good representations, a rigorous method to evaluate their quality is lacking. In this paper, we treat the evaluation of representations as a model selection problem and propose to use the Minimum Description Length (MDL) principle to devise an evaluation metric. Contrary to the established practice of limiting the capacity of the readout model, we design a hybrid discrete and continuous-valued model space for the readout models and employ a switching strategy to combine their predictions. The MDL score takes model complexity, as well as data efficiency into account. As a result, the most appropriate model for the specific task and representation will be chosen, making it a unified measure for comparison. The proposed metric can be efficiently computed with an online method and we present results for pre-trained vision encoders of various architectures (ResNet and ViT) and objective functions (supervised and self-supervised) on a range of downstream tasks. We compare our methods with accuracy-based approaches and show that the latter are inconsistent when multiple readout models are used. Finally, we discuss important properties revealed by our evaluations such as model scaling, preferred readout model, and data efficiency.

Published as a conference paper at ICLR 2023
EVALUATING REPRESENTATIONS WITH READOUT MODEL SWITCHING
Yazhe Li
Jorg Bornschein bornsshelnnddeemmddd.comm com
Marcus Hutter
yazhe deteeedddeeiddd.cmmm com
w cor.
ABSTRACT
Although much of the success of Deep Learning builds on learning good repre- sentations, a rigorous method to evaluate their quality lacking. In this paper, we treat the evaluation of representations as model selection problem and propose to use the Minimum Description Length (MDL. principle to devise an evaluation metric. Contrary to the established practice of limiting the capacity of the read- out model, we design a hybrid discrete and continuous-valle readout models and employ switching strategy to combine their predictions. The MDL score takes model complexity, as well as data efficiency into account. As a result, the most appropriate model for the specific task and representation will be chosen, making it unified measure for comparison. The proposed metric can be efficiently computed with online method and we present results for pre-trained vision encoders of various architectures (ResNet and ViT) and objective functions (supervised and self-supervised) on a range of downstream tasks. We compare our methods with accuracy-basee approaches and show that the latter are inconsistent when multiple readout models are used. Finally, we discuss important properties revealed by our evaluations such as model scaling, preferred readout model, and data efficiency.
model space for the
INTRODUCTION
Data representation is crucial the performance of machine learning algorithms (Bengio et al. 2013). Much of the success of Deep Neural Networks (DNN) can be attributed to their capability of gradually building up more and more abstract representations (Lee et al. 2009). In supervised learning, although the network is trained to predict a specific aspect of the input, the intermedi- ate representations are often proven to be useful for many other downstream tasks (Yosinski et al. 2014). In unsupervised and self-supervised learning, the network is trained
surrogate task, such as reconstruction (Hinton & Salakhutdinov 2006 Kingma & Welling 2013 He et al. 2021) and contrastive prediction (van den Oord et al. 2018 Chen et al.] (2020), which supposed to capture generic prior of the data. In recent years, there has been significant improvements in unsupervised representation learning with state-of-the-an models achieving performance comparable its super- vised counterpart Tommaev et al. (022).
Despite the importance of data representation, the evaluation method for representations is rarely discussed. The most prevalent practice is to train a readout model on the downstream task. The readout model often has a shallow architecture, e.g. linear layer, limit its capacity, so that the task performance reflects the representation quality. The problem with this approach is that the readout model cannot adapt the nature of the representations. Deeper models and fine-tuning alleviate this issue. However, the representations are left with multiple metrics, each using different readout mechanism, making the comparison extremely difficult (Nozawa & Sato 2022).
In this paper, we treat evaluating representations as model selection problem. We propose to use Minimum Description Length (MDL) as the main evaluation metric and use model switching to accommodate the need for multiple readout models. MDL is well-studied compression-baset approach for inductive inference that provides a generic solution to the model selection problem (Rissanen] 1984, Grunwald 2004 Wallace 2005 Solomonoff 1964 Rathmanner & Hutter 2011). MDL performs a similar role as held-out validation does for Emperical Risk Minimization (Vapnik) 1991], but has the advantage of being able deal with single sequence and non-stationary data.

---

Published as a conference paper at ICLR 2023
It is closely related to Bayesian model selection and includes a form of Occam's Razor where the metric takes into account the model complexity. The complexity term can be explicitly represented as the codelength of the model in the case of 2-part code, KL-term when using variational code, or implicitly when using prequential or Bayesian codes. By including the model complexity in the evaluation metric, we automatically resolve the need of limiting the readout model complexity and are able to compare MDL scores freely between different readout mechanisms. Intuitively, if the induced representation nonlinear and requires a higher capacity model for readout, the MDL score reflects this by having a larger complexity term, Note that this also applies in the case of fine- tuning, where the pre- praited model allowed to adapt for the downstream tasks. Model switch- ing allows multiple readout models and automatically finds the best readout model for the down- stream task at each dataset size [ Therefore, MDL. with readout model switching provides unified framework for evaluating representations regardless the evaluation protocol employed.
It conjured that useful representations make the variability in the data more predictable and al- low to efficient learning human-like data (Hénaff) et al.) 2019). The MDL evaluation metric for- malizes the data efficiency perspective espe-
Model switching
Model A
Model B
Model
Switched Model
Noxtsdep Crow-Bttroy
pMDL
cially evident in the form of prequential MDL.
Prequential MDL (Dawid & Vovk 1999 Poland
& Hutter 2005) turns computing the descrip- - log p(Dlo) into
tion length L(D)))
sequential prediction problem: pog)(
>, pogg((sl Mεt), where @ is an encoder for feature extraction, Φe == ((x,) an encoded input, and (I<e; Merr) is the data seen before timestep t. In order to achieve a short description length, it is beneficial if the representation is capable of fast learning, i.c. given a good representation, a few examples enough to achieve good perfor- mance. As a by-product of our readout model switching based on MDL, we can visualize the predictive performance along the data sequence and inspect how efficient the downstream learn- ing is.
Data
Illustration of switching between models of different complexity: Depending on the number of training examples either A, B, or C' has the best generalization performance. An optimally switched model will have the best performance at each point and thus the lowest prequential description length (= area under the curve).
Figure 1:
Our contributions are as follows:
1. We propose readout model switching for evaluating representations.
2. We prove regret bound of readout model switching and discuss assumptions made.
3. We use an online learning framework for efficient computation and evaluate the perfor- mance of several popular visual representation methods on the set of downstream tasks.
4. We investigate and inspect the preferred readout models, data efficiency and model scaling.
2 BACKGROUND
Minimum Description Length is based on the fundamental idea that learning and comprehen- sion correspond compression (Rathmanner & Hutter 2011). Given data D=(w)) y and a hypothesis space M (M1, My.... where each hypothesis M corresponds to a parametric probabilistic model p(D 9, M). MDL aims to identify the model that can compress the data D best. Considering the close relationship between lossless coding and probability distributions, this can be achieved by associating a codelength function T.(D) M)) - log ((D) M) with each hypothesis. vast body of literature shows that models with shorter description length have better chance of generalizing to future data [Wallace, 22005) Grünwald 2007a| Rathmanner & Hutter 2011).
0), where (#) is the cost of encoding the parameters and L (D]#)=- log (D 8, Mf is the cost of com- pressing the data with the parameterized model. This two-part code approach is intuitive but sub- optimal and ambiguous because it does not specify how to encode the parameters. This crude MDL approach has been refined in three distinct but closely related ways:
A crude way to obtain description lengths is to consider L(D) M) (M)
2

---

Published as a conference paper at ICLR 2023
1) The Bayesian marginal likelihood:
Laayes (DIM == log J P(D|e, M)) (d) and its variational upper bound LELBO (DIM) := (l)((())p(()) - log (D M.. Both depend on the chosen prior p(0) and require access to the posterior distribution
parameters given data; or an approximation thereof.
2) Normalized Maximum Likelihood (NML):
(D]M)
log [p( (((()) (D),M)/ J// (,p(([(xa M)dz] only relies on the maximum
likelihood estimator (-). but normalizes over all potentially observable data 2 E yN,, which is often intractable and even undefined for many model families of interest. The log oog-ummerator is also called the complexity of M and measures how well the model could all possible data. Under NML, a model M achieves a short description length only when it fits the given data well (high log p( (D)) M)) as well not fit well many different data (low denominator).
3) The prequential approach:
Σ logp(y/(((D<z)) M) decomposes the description length over data- points and relies on the choice of a suitable plug-in parameter estimator d(Dc.). It emphasizes that in order to achieve short codelength, model M must not only be a good predictor given all training data D, but already given only parts of the data Der for all t, i.e. be sample efficient predictor. Model complexity and sample efficiency can thus be considered two sides of the same coin.
Lplugin (DIM)
Approaches to computing description lengths for some data D under a model family p(D|0,
called universal codes when they result in description lengths that stay close to the (generally unachievable) maximum likelihood codelength for that family: Luniverssal ( M) S
- log p( [D (D),, (D).M) .)) + Ollog N). (DML( is by definition universal; Layess (D) (D)M) and
plugin (P) are universal for many reasonable choices of the prior p(0) and the estimator ((Dc))
respectively (Grünwald)) 2007b).
Methods for Representation Evaluation. Linear probing, using a linear layer for readout, is the most prevalent evaluation protocol for representations, the performance of which is reported and compared in almost all of the recent representation learning works (van den Oord et al. 2018 Chen et al. 2020; Grill et al., 2020 Li et al., 2021 Caron al., 202T He et al. [2021). Apart from lin- ear probing, clustering a algorithms such as Hebbian linear classifier (Bengio 2012) and K-Nearest Neighbor (Caron et al., 202I) are also popular choices. However, concerns have been raised for model comparison using these simple probes. Resnick et al.] (2019) shows that the performance of linear probing can be misleading, since models performing weakly with linear probing can perform better under alternative protocols. He et al. (2021) also argue that the use of linear probes limits the development of methods that induce non-linear representations. Hewitt Liang 2019) question the practice of using downstream accuracy as the evaluation metric and propose to design probes based on their selectivity, which puts the task accuracy context with the probe's 's capacity to memorize.
Yogatama et al. (2019) and Voita & Titov (2020) propose to MDL, either variational or prequen- Iial code, as the evaluation metric, which takes into account the complexity in the final measurement. Hénaff al. 2019) bring to light the benefit of pre-training for downstream data efficiency. Whit- ney et al. (2020) propose a modified version of MDL that introduces a baseline corresponding to solving the downstream task. While previous works proposing MDL as an evaluation metric use only a single readout protocol, our method can be applied for combining different readout protocols regardless of the backbone being frozen or fine-tuned. Our use of an online framework to compute prequential MDL scores which has not been used previously to evaluate representations leads to shorter description lengths (Bornschein al. 2022) and crucially enables the model switching be- tween readout- readout-methods methods at each timestep. As result, our framework gives insights into properties of the representations that can guide us in choosing the best way to use the representation.
EVALUATING REPRESENTATIONS WITH READOUT MODEL SWITCHING
We formally define the model selection problem for representation evaluation under the MDL principle as follows: the objective is compare the encoder o(.) that minimizes the code- length L(D]o) = - log p(Dpp)))) where D = {I., 9:) is the inputs x, and associated predic- tion targets yt- Prequential MDL decomposes the codelength into the cumulative prediction loss
3

---

Published as a conference paper at ICLR 2023
= - log p(y, Φ V<t). where Φ; =(x)) is the encoded feature of imput zt. In order to perform the prediction task from the features, previous works have to fix the probe to be linear layer
shallow MLP. In our study, instead of picking a fixed model, we design hy- brid of a continuous and discrete-valued (k: 0s) model space of the readout models, where "expert" parameters corresponding the
k [1, [1....; K] is the model class and 0g are the continuous-valuu model class k. Each readout model can be seen as an expert forecasting system that learns to predict the data independently in an online fashion. At each datapoint It, the prediction De is a combination of the experts' predictions. The final MDL score is the cumulative predictive performance of the board of K experts.
By constructing the probe using multiple readout models and switching between them, the final MDL score is marginalized over the choice of the readout architectures. Since MDL automatically takes into account the complexity of the model, i.e. any extra complexity used is going to cost in the MDL score, our framework allows the representation to select the most suitable readout mechanism at any data size while still maintaining comparable final score for the representations, In addition, after computing the score, we can inspect the posterior over the readout models at any given data size, well as data efficiency along the trajectory, providing valuable insight into the characteristics of the representation.
Readout Model Switching. Given K readout models, we define random variables & [1..... K1 Vi E [1.....3 N the readout model to use at timestep t. The Expert Sequence (ES) prior (Koolen & de Rooij 2008 uses hidden Markov model (HMM) to define the joint distribution of I Pt. (y (oci, F<ce (<c) 1 where p(E ==)=((()) is the initial distribution and P=(-) is the prediction of the kth readout model.
the data 31:N and random variables Ê1-N as p(y1.N fi:v
At each timestep t. use the previously observed data to estimate (d V<e) for the parameters of the readout models. Therefore, the kth readout model prediction p& (V (éct. F<ce) is the plugin distribution p(ve 0x(0ct Uct)). The final readout model switching codelength function has the form:
N
(Swich(W1EN = logg(unNl)))) = log Σ II plucorr de. (e:
(I)
S1:N
The readout model switching defined in Equation represents a family of codes that combines prequential and Bayesian MDL. From the perspective of the HMM, ((-() is the transition matrix and, in our context, it is equivalent to a switching strategy between the readout models. Every switching strategy corresponds to a specific code in the family. For example, if p(E ((-1) = iff & Ei-1 (zero otherwise), i.c when we believe single readout model is responsible for modeling the whole data sequence, then Equation simpliiess to a Bayesian mixture code LBM (J1.N ()) -log P(SI = k)) IT P(y. l0., (e Ver.).
For the rest of the paper, we are going to use Fixed Share (FS) (Herbster & Warmuth| 1998; 2004) as the switching strategy. The prior defined in fixed share can be interpreted as follows: at timestep t, with a probability
a,, the expert is the same as in the previous timestep; with probability ar, it switches to another model according to the probability w(k). We are going to use the uniform distribution wk) = 1/K through out the paper. The switching strategy can be simplified and expressed as
P(E)(--1) 1 1 1-
{
ao K
K-1
if - if 7
where a2 o; m
E [0.
(2)
R 0/
decreasing timestep i, and m is a hyperparameter. Note that there are other switching strate- gies that could be used, e.g. elementwise mixture, fixed share with constant switching rate, switch distribution and run-length model (Koolen & de Rooij 2008). We provide a comparison these strategies in Appendix A.6 In general, we find that Bayesian mixture and clementwise mixture perform significantly worse than fixed share and other more sophisticated switching strategies. The poor codelength of the Bayesian mixture is due to the catch-up phenomena (van Erven al., 2012): assumes that a single model is responsible for modeling the whole data sequence, while evidently the best model changes with data set size.
Regret Bound. To characterize the gap between the codelength for our readout model switching and the shortest codelength that can be achieved in hindsight, we derive regret bound in Theorem
4

---

Published as a conference paper at ICLR 2023
This bound tells us whether the readout model switching converges to the optimal solution and, if so, how fast converges.
Theorem 1. Let a sequence x = (xi)i=1 be sampled ii.d. from distribution P. Let all readout models Mi,... MR be in the exponential family. Let 0g (a²) be the regularized Maximum Likelihood (ML) estimator of Mr after the first examples x' (Iz) The expected regret of readout model switching w.r.l. the best sequence of readou models Si.N and their best fit parameters 2eg (x Nx
N
N Ma. (IelOg. (1el0c(
R(N) = Ep Σ
log Ma; (2x; (2r; (I - -log Σ p(Li,N) II
EIN
Ep [log p(Ci,N)) 2 max {tr (CovTl, [T) (T(()))) log N
S
(3)
where p = [x)],(x)), and Tx). is the sufficient statistic and py is the mean value parameter of M₁.
The proof of Theorem 1 can be found in Appendix A.I The expected regret consists of two terms:
the regret of the switching strategy and the regret from the plugin distribution. The latter bounded by log N with C' being a constant determined by the model family and the true distribution. For Fixed Share with decreasing switching rate, Ep [log p(εt.x))
]] < m log K + (m - 1) log N /mm
1), where
1 is the number of switch points (Veness al. 2012 Koolen & de Rooij 2013).
Therefore, the expected regret is of the order of log N = o(N), making the corresponding code a
universal code. Many other switching strategies have the same property, such as Bayesian mixture
and run-length code (Koolen & de Rooij. 2013).
Like many theories and algorithms applied to DNN, some of the assumptions of Theorem [ do not hold in practice: DNNs are not in the exponential family and the solutions found by Stochastic Gradient Descent (SGD) training are not ML solutions. Although the exact nature of the final codelength heavily relies on the training algorithm and much of the understanding of SGD is left for future works, as long as the estimator converges asymptotically to the optimal solution, we can use the code for model selection. In fact, there exist estimators that outperform the ML estimator such that the resulting prequential code better approximates the Normalized Maximum Likelihood (NML) code which is the optimal universal code (Grünwald & Kotlowski/ 2010).
Computing MDL with Online Learning. Algorithm I describes the online implementation for computing the description length and for performing inference over the preferred readout models (.)). For each example in the sequence we first obtain a marginal prediction by weighting the predictions from the readout models according to our current belief over the mixture. After
Algorithm MDL with Readout Model Switching
Require: data D (Ie, 3/ W : K readout models; initial model probability p(Ei) and switching strategy p(ć=/{=-)); subroutine that updates the parameters given a dataset Update Initialize: model parameters 01....,0x AKk s logr({;); an empty list Q to store posterior
P(E,(De)
2: for 1= to do
3:
Compute
log Σ g- exp (logp(EelEr--)
4:
Compute posterior p&, (t=t) and store in Q:
log p(& D2t) = log p(E,, y² 1x=-11) log Σ c. p(Ee, 100-10---100 12=11
5:
Compute next-step loss of K readout models: Ch : loggp(wvrzz) log p(ge(1)
6:
Combine K models to update log [x'):
C+: +
7:
for k = 1 to K do
8: Update paramcters: 0g + UpdateParameterste Des))
9: end for
10: end for
Compute total codelength Lawitch t- -lg Σex exp(s)
12: return Lywitch and Q
5

---

Published as a conference paper at ICLR 2023
receiving the label yt, we compute the cross-entropy loss for the marginal prediction and add it to the cumulative loss. Then, the belief over the mixture is updated and the new example is added to the training data for readout model parameter estimation. The latter is done by online SGD with replay.
For the posterior belief over the model switches p(Er)D<t) the above procedure amounts to the well known, efficient, and exact forward pass HMM inference algorithm. For the neural network parameters 0x on the other hand we use SGD as the plugin estimator for prequential MDL. Prior work has shown that SGD is an effective and reliable plug-in estimator for modern neural networks (Blier& & Ollivier 2018). Note that most prior works on computing prequential MDL scores with neural networks use a block-wise approximation; i.c. they partition the data into small number of segments and evaluate each segment independently by training on all the previous segments to convergence (Voita & Titov| 2020. Bornschein et al. 2020) Whitney et al. (020). Since we need prediction at each step to update the mixture weights, we here instead use the online mini-batch incremental training with replay described in Bornschein et al. (2022). This approach updates the parameters continually with online mini-bate mini-batch gradient descent training on the new and previously seen examples. Instead of using replay buffer to store examples in- in-memory, we use replay streams to iterate through the examples in their original order from permanent storage. By tuning the num- ber of streams and the reset probability, we can adjust the amount of replay training and change from uniform sampling to a sampling strategy that is biased towards more recent samples. How- ever, in this work. we use uniform random replay samples for all our experiments. As suggested in Bornschein et al. 2022), we use Exponential Moving Average (EMA) and label smoothing to improve performance of the online training, but not weight standardization as we don't 't want alter the architectures.
A strictly equivalent alternative to the fully online implementation is a two-stage implementation (see Appendix (..3:: First, the prequential online parameter estimation is computed for each of the readout models independently while the per-step predictions and log-losses are recorded for later analysis. In a second stage, the predictions and losses can be used to compute the mixtures, switch- ing posterior ((E) and overall description lengths. To facilitate experimenting with different switching strategies we use this 2-stage implementation for all our experiments.
4 EXPERIMENTS
First, we compare our evaluation framework with linear and a combination of MLP evaluations using vision encoders pre-trained on ImageNet (Deng et al.) 2009) with different objectives. For downstream tasks. Then, use
this purpose, we use the VTAB benchmark (Zhai et al.) (2119) ImageNet classification as downstream task to showcase the insights, such as data efficiency and preferred readout model, and for comparing in-domain and out-of-domain transfer, revealed by our evaluation method.
The VTAB benchmark consists of 19 datasets that be grouped into three categories: natural, specialized and structured. For some VTAB datasets, notably KittiDistance, SmallNORBAzimuth PatchCamelyon, we discover that there distribution shift between train and test subsets (see details in Appendix A.2). Therefore, in order to make sure that data are truly i.i.d, we reshuffle the train and test data and use the reshuffled datasets for all our experiments.
To summarize the performance over multiple datasets, we follow the procedure recommended by Demsar (2066) and report the average rank of the representations: for each dataset, we rank the representations by the evaluation metric (accuracy or MDL score); then, we compute the average rank of a representation over the 19 datasets. For any pair of representations and B, the difference of the average rank can serve as statistic for hypothesis test with HO: A performs the same as B. If the hypothesis is rejected, we conclude the representations perform significantly different. We adopt the Nemenyi test (Nemenyi 1963) which suggests that the null- null-hypothesis is rejected if the average ranks differ by at least critical value. This critical value is computed as 4, /r(r 1)/6N, where by is a constant depending on the confidence level y, r is the number of representations to compare and N is the number of datasets. In the following comparisons, also report the critical value with a confidence level of = 0.1.

---

Published as a conference paper at ICLR 2023
We use the online learning framework as described in Section and compute the Fixed Share code length with decreasing switching rate a defined in 2 The raw MDL scores per dataset are reported in Appendix A.7 For ImageNet,
readout model switching between 4 model architectures: Linear and 1-3 layer MLPs. For VTAB, we include up to 7-layer MI.Ps as readout architectures. We furthermore instantiate each of these architectures with all hyperparameter settings from the hyperparameter grid outlined in Appendix[A.5 [A.5 The model switching is therefore performed between K #architectures #hyperparameters many independent models. We report results using switching rate m = 2 for ImageNet and m = 11 for VTAB. The switching rates were chosen to optimize the overall description length.
Pre-training Objectives. We compare downstream performance of different pre-training objec- tives on the VTAB benchmark. To make the comparison, we choose two backbone architectures, ResNet-50 v1 (He al., 2015) and VIT/B16 (Dosovitskiy et al. 2020). and only change the pre- training objectives. For ResNet-50, we pre- trait on supervised SimCLR and BYOL losses; for ViT/B16, we pre-train on supervised (Touvron et al. 2020 He al.
(021) DINO and MAE losses. SimCLR (Chen et al. (220)) is a self-supervised method based on contrastive learning. BYOL (Grill al. 2020) and DINO (Caron et al., (20TT) both
teacher network. BYOL is a negative-free method for self-supervised representation, while DINO uses the ViT architecture and learns through self-distillation. MAE (He al. 202I
uses masked reconstruction tasks to learn representation.
Pre-training Objectives
(b) Model Sizes
Architecture
Objective
Linear
MLP
MDL
Architecture
Obj.
Linear
MLP
MDL
VIT/B16
DINO
2.26
1.84
1.89
ResNet-101
BYOL
2.84
3.92
3
3.05
ResNet- 152
BYOT.
3.11
3.71
2
3.26
ResNet-50
BYOL
2.63
2.53
VIT/L1
MAE
3.42
2.53
3.74
VIT/B16
MAE
3.84
3.24
3.79
ResNet-50
4.24
4.26
VII/B16
SUP
4.18
4.21
4.16
VIT/B16
SUP
7.34
6.47
6.00
ResNet-50
SimCLR
4.53
4.63
4.21
VIT/B16
MAE
6.89
5.34
6.21
ResNet-50
SUP
3.82
4.45
4.42
ResNet-101
ResNet-50
SUP
SUP
6.89
6.95
7
7.55
7.47
9
8
7.21
7.11
8
ResNet and VTT with different pre-training objective and model sizes. The threshald of the rank differenze to he significamt for hypothesis testing is 1.894 (left) and 3.65 (right).
ResNet- 152
VID/11
SUP
SUP
7.05
7.47
10
8.50
7.00
10
7
9.11
7.47
10
VIT/S16
SUP
10.26
10.61
10.37
MAE
11.53
10.63
10.42
Table 1: Comparison of model selection using linear evaluation, combination of evaluation with MLPs and MDL with readout model switching. Average rank (lower is better) for representations over 19 VTAB datasets is reported. Architecture + pre-training objectives are in order of MDL rank.
Table la reports the average rank of the pre-trate pre-training objectives comparing model selection with dif- ferent cvaluation methods. As baselines, we report linear evaluation and a combination of MLPs up to 7-layers. To combine results from MLPs, we take the best performance achieved by MLPs. Given a confidence interval = 0.1, the performance of the representations are significantly different the difference is above go.1 /r(( + 1)/6N = 1.894, where go. 3.12 is the critical value for the two-tailed Nemenyi test, and number of datasets.
is the number of representations compare and N 19 is the
Despite fixing the backbone architecture, the choice of readout model has impact on model lection under accuracy-based evaluations (first two methods in Table Ta). This is consistent with previous observations (Resnick al., 2019 He et al. corporating a multitude of readout methods into the evaluation framework. Our method solves this issue by combining probing protocols and taking into account their complexity. Furthermore, under our evaluation framework the performance in the small data regime is also important factor and representations performing worse with little data are penalized in the final rank. We observe that for ResNet50 backbone, BYOL features outperforms both SUPervised and SimCLR features, while SimCLR and SUPervised features have no significant differencee. For VIT/B backbone, DINO features outperform both supervised and MAE, while supervised and MAE have no significant dif- ference. Therefore, it appears that with the same encoder capacity, distillation- based objectives have an advantage in terms of transfer. DINO seems slightly better than B BYOL, but the rank difference is not statistically significant for a clear conclusion.
202T) and demonstrates the importance of in-
In Figure we investigate the transfer performance under distribution shift by reporting statistics under each VTAB group. We observe that self-supervised representations have a better performance
7

---

Published as a conference paper at ICLR 2023
than the supervised counterpart on structured datasets, despite the latter perform decently on natural datasets. MAE generalizes well with dataset shift, but the performance on natural datasets suffers compared to other methods.
Model Scaling. Another interesting as- pect about architecture choice is the model scaling: do larger models result in bet- ter performance? We again use ResNet and VIT backbones pre-trained on Ima- geNet and evaluate on VTAB, but with in- creasing model sizes. For ResNet, lect ResNet-50, ResNet-101 and ResNet- 152, and pre-train with either SUPervised or BYOL objective. For ViT, we use VIT/S16, VIT/B16 and ViT/L16, and pre- train with either SUPervised or MAE ob- jective. We rank the performance of the 12 representations and report the average ranks Table Tb With confidence interval of 0.1, we obtain threshold of 3.65 for the difference in average rank to make a statistical claim.
Architecture
Obj.
Natural
Special
#
Structured
VIT/B16
DINO
1.57
1,00
2.63
ResNe150
BYOL
2.13
VIT/B16
MAE
5.71
3.25
3
2.38
VIT/B16
SUP
2.86
3.25
5.75
RosNet50
SimCLR
4.29
5
5.75
6
3.38
4
ResNet50
SUP
3.86
4
4.75
5
4.75
5
Table 2: Average rank on natrual, special and struc- tured categories of VTAB datasets. The thresholds for the rank difference to be significant are 3.12, 2.92 and 4.13 respectively. Architecture + pre-training objec- tives are in order of MDL rank on all VTAB tasks.
The scaling experiment more compellingly demonstrates the difference of multiple readout models and the emphasis on performance in the small data regime with our method (Table [bb. Larger models, such as ResNet-152 and ViT/L16, which perform better with MLP head need to demonstrate their performance in the small data regime to justify the additional capacity used for the readout. We can see that scaling up the ResNet architecture, in particular under supervised pre-training, doesn't t result in any improvement in the transfer performance. There is weak positive scaling effect for BYOL when scaling to ResNet-101, 101, but it is not statistically significant. Although ViT/S has approximately the same parameter counts ResNet-50, its predictive performance is much worse, suggesting that ViT's capacity is used less efficiently than ResNet's. s.. Scaling up ViT from S16 to B16 leads to a significant performance improvement. However, further scaling up for supervised loss is not useful. Interestingly, the reconstruction-bart MAE objective scales well to larger architectures and has not saturated ViT/L16.
Readout Models. One strength of our evalu- ation framework is that can compare readout models of different capacity and dynamically choose the most suitable one based on down- stream task and training data size. We com- pute the posterior over the readout models to inspect which readout model the representation prefers. We observe that model switching is more frequent for out-of-domain datasets than in-domain datasets. In Figure 2 we report the readout model that has most often the highest probability arg max, T - The result suggests that the most important fac- tor for preferring deeper models is the dataset type - structured datasets need deeper mod- cls. The influence of the pre-training objective docsn't 't seem to be significant.
Figure 2: Readout model that most often has the highest probability arg max, + p((: = k() on 19 VTAB datasets. 0 is linear readout; to 7 are MLPs with to 7 hidden layers.
Natural
Special
Structured
SmalINORBAzimmi Azimmt
DSprrtes SppiteeOriietaliin Orientation
SmalINORBElevati
Location
/ = k(Dx.).
Architeture
Objective
Callech101 101
Clari00
DTD
Flowers
Pats
SVEN
Sun
ETroSAT
PatchCCamelee
Resisc45
Reiinopaty
Clev
CleaaDistmncs
DMLah
DSprites1
KistDDst
ResNer50
SUP
4
ResNer101 SUP
2
2
ResNet152
ResNet50 SimCLR
3
ResNet50 BYOL
5
4
3
ResNet101 BYOI.
3
6
ResNer152 BYOL
VIT/S16 SUP
D
VTT/166 SUP
0
12220
3
2
3
3
Data Efficiency. Data efficiency is an integral part of MDL. With the prequential perspective on MDL, we can visualize whether an algo- rithm is more data efficient by inspecting the
VIT/L SUP
7
VIT/166
VIT/S6
MAE
2
2
6
4
VIT/B16
MAE
0
000000
4
6
VITA116
MAE
4
predictive loss as function of the data size.
As baseline we train a ResNet-50 model from scratch within our evaluation framework, and record the cumulative next-step losses as a reference
8

---

Published as a conference paper at ICLR 2023
1e5
1e4
10
Scratch
0.0
Regret
BYOL-Linear
Regret
-0.5
BYOL-MLPs
BYOL-Finetune
From Scratch
-1.5
BYOL-Linear
BYOL-MLPs
BYOL-Finetune
10
Number of Examples
1e6
Number of Examples
1e5
(a) In-domain: ImageNet
(b) Out-of-domain: PatchCamelyon
Figure 3: Data efficiency: We plot the cumulative next-step log-loss for range of readouts with ResNet-50 backbone as a function of (downstream) sample size (in nats, lower better).
baseline (comparator). Then, we evaluate BYOL pretrained ResNet-50 representations with 3 dif- ferent readout models: 1. linear probing; 2. deep MLP readout models; 3. linear readout but with fine-tuning the whole encoder. Because MDL already takes into account the model complexity, we can directly compare their MDL scores. We plot the "regret" (in this case it is more of an advantage) of different fine-tuning strategies relative to the training-from-scratte model as the baseline. Figure illustrates the "regret" along the data size for ImageNet, in-domain dataset, and PatchCamelyon, an oft-of-domain dataset for the BYOL representations. Learning ImageNet classification with a pre-trained representation is much more efficient than training from seratch. Shallow architectures, which are based on a frozen encoder, are sufficient. Only when the dataset is large enough (800k in case of ImageNet), we can finally see the benefit from fine-tuning the encoder. The picture is dif- ferent for out-of-domain datasets. In general, pre-training still has benefits if deeper, more flexible readout models are used. The amount of capacity needed depends on the nature of the downstream dataset. In Figure 3 on PatchCamelyon, training from scratch surpasses linear probing after seeing only 150k examples. We show the regret plots for all 19 VTAB tasks in Appendix A.9
CONCLUSIONS & FUTURE WORKS
We presented readout model switching, an approach derived from the MDL principle, evaluate representations given downstream tasks. Our approach supports model switching to choose the most appropriate readout model for each representation, downstream task, and data size. Our metric takes model complexity and data efficiency into account, making it unified measure for model comparison. We evaluated vision representations and compared our approach to accuracy-based methods. Using our evaluation framework, were also able to provide insights into model scaling properties and preferred readout models, as well as compare data efficiency.
In theory, different permutations of the data sequence result in different codelengths. Although this does not appear to cause too much trouble in practice (see Appendix A.7). we reckon that this an area for future work. With the MDL. formulation the performance metric under consideration is the log-loss (cross-entropy), which directly corresponds to compression. Extensions to other losses such 0/1-loss etc. will need careful consideration. We here report single metric per downstream-tate and representation. MDL supports non-stationary sequences and future work could aim for reporting a single metric per sequence-of tasks by concatenating them. However this poses questions about the order of downstream tasks, and how to deal with multiple output spaces V, and whether forward transfer between these tasks permissible.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML workshop on wnsupervised and transfer learning, pp. 17-36. JMLR Workshop and Conference Proceedings, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE transactions on pattern analysis and machine intelligence, 1798)11798-1888, 2013.
Léonard Blier and Yann Ollivier. Do deep learning models have many parameters? infor- mation theory viewpoint. CoRR, abs/1 has/1802.0744, 2018. URL http:::p///xxxi.or////aa/
1880207044
Jörg Bornschein, Francesco Visin, and Simon Osindero. Small data, big decisions: Model selection in the small-data regime. In International Conference on Machine Learning, pp. 1035-1044. PMIR, 2020.
Jorg Bornschein, Yazhe Li, and Marcus Hutter. Sequential learning of neural networks for prequen- tial mdl, 2022. URL (tLphtts:// http:://axxiv...oo. wwg. 2210.2220..000 07931
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging properties in self-supervised vision transformers. CoRR, abs/2104. 2021. URL htttp::///aattt
org/ abs abs.///....2...
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020. https:
05709
A. Philip Dawid and Vladimir G. Vovk. Prequential probability: principles and properties. Bernoulli, 5(1):125-162, 1999.
Steven De Rooij and Peter Grünwald. Luckiness and regret in minimum description length infer- ence. In Philosophy of Statistics, pp. 865-900, Elsevier, 2011.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine learning research, 7:1-30, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at
scale. CoRR, abs/2010.1 1929, 2020. https: http://aaxivvv.o orq. www.aasslii 11929
Bann-Batienn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self supervied learning. CoRR, abs/2006.07733, 2020. URL
httts:://axxis wwtpoo/g//bb///ooeee 07733
P. D. Grünwald. The Minimum Description Length Principle. The MIT Press, Cambridge, 2007a.
Peter Grunwald. A tutorial introduction to the minimum description length principle, 2004. URL
https://arxiv: http://wwww..rr//////////t//////
Peter Grünwald and Wojciech Kotlowski. Prequential plug-in in codes that achieve reduudacy rates even if the model is wrong. CoRR, abs/1002.0757, 2010. URL http://azxiv..rrg
abs/1002. 0ttp/102.005555
Peter D Grünwald. The minimum description length principle. MIT press, 2007b.
10

---

Published as a conference paper at ICLR 2023
Mark H Hansen and Bin Yu. Model selection and the principle of minimum description length. Journal of the American Statistical Association, 96(454):746 774, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. CoRR, abs/15 112.0338,, 2015. URL
03385
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- 2021. URL /tttps://ariv..
loencoders are scalable vision learners. CoRR,
org/abs/2111 06377
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. CoRR, abs/1990.00927, 2019. htk..tttpp://xxx..orrre httpaaawwwv..oggaaaaaaaaa// 1905. 09272
Mark Herbster and Manfred K. Warmuth. Tracking the best expert, 1998.
Mark Herbster and Manfred K. Warmuth. Tracking the best expert. Machine Learning, 32:151-178, 2004.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. CoRR, abs/1909.03366 2019. URL http://arxiv. org. abs http://wwwwwrraaaa//...ooo 03368
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 51375776):50---07 2006.
Diederik Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Wouter Koolen and Steven De Rooij. arXiv:0802.2015, 2008
Combining expert advice efficiently. arXiv preprint
Wouter M. Koolen and Steven de Rooij. Combining expert advice efficiently. CoRR, abs/0802.2015, 2008.
org. 0802.2015
Wouter M. Koolen and Steven de Rooij. Universal codes from switching strategies. CoRR, abs/1311.66 2013. URL
6536
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th annual international conference on machine learning. pp. 609-616, 2009.
Yazhe Li, Roman Pogodin, Danica J Sutherland, and Arthur Gretton. Sele-super learning with kernel dependence maximization. Advances Neural Information Processing Systems, 34: 15543-15556, 2021.
Peter Bjorn Nemenyi. Distribution-free multiple comparisons. Princeton University, 1963.
Kento Nozawa and Issei Sato. Empirical evaluation and theoretical analysis for representation learn- ing: A survey. arXiv preprint S2Xiv:2220.08226, 2022.
Jan Poland and Marcus Hutter. Asymptotics of discrete MDL for online prediction. IEEE Transac- fions Information Theory, 51(11):3780-377 2005.
Samuel Rathmanner and Mareus Hutter. A philosophical treatise of universal induction. Entropy, 13(6):1076-1 1136, 2. ISSN 1099-4300. doi: 10.3390/e13061076
Cinjon Resnick, Zeping Zhan, and Joan Bruna. Probing the state of the art: A critical look at visual representation evaluation. CoRR, abs/1912.00215,, 2019. URL
abs/
1912.0021 00215
J Rissanen. Universal coding, information, prediction, and estimation. IEEE Trans. Inf. Theory, 30 (4):629-636, July 1984.
Jorma Rissanen. Stochastic complexity and modeling. The annals of statistics, pp. 1080-1100, 1986.
11

---

Published as a conference paper at ICLR 2023
Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7(1): 1-22, 1964.
Nenad Tomasev, Ioana Bica, Brian Mc MiWiliims, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprini TrXXiv2200.05111 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012. 12877, 2020.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 .03748, 2018. URL
http:///gggabbbbboosee 03748
Tim van Erven, Peter Grünwald, and Steven A. H. de Rooij. Catching up faster by switching sooner: predictive approach adaptive estimation with an application to the aic-bic dilemma. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74, 2012.
V Vapnik. Principles of risk minimization for learning theory. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS'91, pp. 831-838, San Francisco, CA, USA, December 1991. Morgan Kaufimann Publishers Inc.
Joel Veness, Kee Siong Ng, Marcus Hutter, and Michael Bowling. Context tree switching. In Proc. Data Compression Conference (DCC 12), pp- 327-336, Snowbird, Utah, USA, 2012. IEEE URL http://
Computer Society. ISBN 978-1-4673-0015-4.
doi: 10. 10.1109/DCC.2012..99.
arxiv. abs abwaaabsslll....aia 1111.3182
Elena Voita and Ivan Titov. theoretio probing with minimum description length. CoRR, abs/2003. 12298, 2020.
probing 20222.20....22282
CS Wallace. Statistical and Inductive Inference by Minimum Message Length. Springer Science & Business Media, December 2005.
William F. Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, and Kyunghyun Cho. Evaluating representations by the complexity of learning low-loss predictors. CoRR, abs/2009.07366 2020. URL /ttpps:::////xxiv....
abs 2009.00000000 07368
Dani Yogatama, Cyprien de Masson d' Autume, Jerome T. Connor, Tomás Kociský, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. Learning and evaluating general linguistic intelligence. CoRR, abs//9901.1333, 2019. URL
org/ 1901 11373
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. CoRR, abs/1910.04887, 2019. URthttp:
// warxiv.org//be....... 04867
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF OF REGRET BOUND
Assume that P and a sequence of examples w. [IN are iid samples from P. Let 6(x*) be the parameters of readout model M that best fit the sequence I i.e. 0(x*)) = infg M (x 0). Since Nº are i..d samples from P. ((x)) is the ML estimator of x We assume that all readout models are from exponential family. But do assume P M, i.c. the model can be misspecifed.
Define the expected regret as the regret w.r.t the best sequence of readout head Eisv and their best fit parameters f (xN) for N::
R(n) -Ep
log Me; (xti (xxM)) - -log Σ p(EuN) ETN
II Me. (zel (xt-1))
t=1
STP
Σ log Me; (1215 Oet (zr log ((g.(() II Met (sa10ez (2t (at-1))
=Ep
(lg Me; (Izz deg (x* log Me; (1rl0e (x(- llg p((pi))
N
= Ep [log p(Ei,))) + Ep Jog Mc; (are: pc: (xx log Mc; (2ra deg (x
Σ
(4)
Since the readout models are from exponential family, they can be parameterized instead by the mean parameters. Recall that the exponential family is the probability distribution that be written in the form of P(z; r7) = h(x) exp (n A(n) where 17 is the natural parame- ters, Txx) the sufficient statistics, A(r) the cumulant function. The mean parameters are E [x]]] and have one-to-one mapping to natural parameters 7 (we use 7(μ) to denote the mapping). Therefore, the exponential family can also be parameterized by the mean parameters p(z; = (xx) exp (T()))T(=) exponential family in the form of P(T: h(T) exp (T(m)"T A(n(4))). Because of the as- sumption that M is exponential family model, Enp [log (x;; 0)] Er-r [log (T; v4]]. Here, we use the same P as the distribution induced on T. The maximum likelihood estimator of the mean parameters ;ix=~) N Σx Twx:).
A(n(11))) We also know that the sufficient statistics is also an
Given any estimator á, we can Taylor expand the likelihood at Er[T(x)):
log M (x; (i)
log M (z; μ)) + (A u) VV log Mo, A) -(( () log M (I, μ)) + 0())
Ep [log M (z: ( log M(:
(ia 1)) TM (V log (1, 11)
(A MTTI((((A p)
(i 11) NOV n))
Where I(H) the Fisher information.
We first use Ep [V log (ax,A)
VpEppllg (xx, A]] = 0. Then we use I(n) = Cov M, [T(x)], RHS is the covariance
13

---

Published as a conference paper at ICLR 2023
matrix under the distribution Mn. For readout model M, after seeing 2' examples, we have
Eg uP Dlo (xt; 1)) - log (1);
2 (i 1J) [ (T)))(( w)]
E P
10 ] Tr (Cov Mr (T)) ((x))),
Ep
2
1 Tr [T [(0 n - m)T)
2
1 1
2t-1 Tr (Cov [[r)] (T(x=>))ov)(((
We use the fact that trace a linear operator, therefore we could exchange the order with ex- pectation. The last step comes fromf((
- T(x,) denote T; = T(x:), we have -W
Ep [Aa - 1=) -
Ep (G
= - -up
(x)] here the covariance is under the true data distribution P.
For each term inside the sum over timestep in Equation 4 we have
Ep
log Me; (ax; 0e: (x - log Me; (ara 02: (x 1
=Ep [log Mea (arei Peg (x=)) - log Me: (re leg (x" [)]
-Ep
[lo Me; (zei feg (x: log Me: (2 (4) + log Me; (æti ()) log Me; (selée: (x'
1
2 - N Tr
Cov Mare [Ter (z)) [Te; (z))
O(1)
We first change it from parameterized by natural parameters to mean parameters. Then for each term, we can compare it μ [x)]
The total sum of t = I,..., T in Equation is
Ep log Mcr (1r;; (12; (x>)) log Mer (xal0g (a'
N )rr
(ov [Te (x) [Te (c)]
N
max { [Tr (Cov Maz [Ter (r) [Te: (x)])
2 1=2
N
max [Tr (Cov
Mry [T(e)] (TA(T))]
(E
(_+-1)
N
1
2
1 $ max {Tr (Covnt [T(())]ovp [T((a))) log 2
Inequality in line 2 uses that When t = 1, we use the prior and incur a constant regret. Incquality in line 5 uses harmonic series) M log N + const. Note that if there is model misspecificatioon in any of the readout models, we have Mpg P. In this case. Cov M,g [x)] [Tx())
and (r()) = k. We find the classical result for regret of prequential code lo
Combining with Equation the expected regret is then
R(n) < Ep [log((Ei.)))
+ max {Tr (Cov M,g [Th [T) (Ta))))) log N
(5)
The expected regret for our readout model switching is composed of 2 terms: one is the regret from expert switching algorithm and the other is the regret of the readout models. The growth of second
14

---

Published as a conference paper at ICLR 2023
term is well behaved and bounded by Clog N where C is constant that relates the model family chosen. The first term is not necessarily bounded by log N. but relates to the switching strategy chosen. However, for many switching algorithms, such as Bayesian mixture, fixed share with decreasing switching rate and run-length code, - E, [log p(S ]] is in the order of log N. In particular, for fixed share with decreasing switching rate, we have the result of -Ep [log p(Si.v)) S m log K (m - 1) log - where m is the number of switch points (i.e. m is the number of blocks in which the same expert used). See Koolen & de Rooij (2013) for proof of the fixed share and more details of the regret bound of other switching strategy. A similar proof is given by Veness
al. 2012) for the particular case of 2.
A.2 DISTRIBUTION SHIFT BETWEEN TRAIN AND TEST VTAB
Figure shows the top-1 accuracy train and test subsets with the original VTAB and reshuffled VTAB datasets. For the same dataset, a difference test set performance between the original and reshuffled datasets suggests data shift between the two versions. KittiDistance have significant data shift. DMLab, PatchCamelyon and SmallNorbAzimuth also have distribution shift but to a much less degree compared KittiDistance. To make sure that the dataset is i.id, combine all its subsets, randomly shuffle then partition into the same number of train/test examples. We only take the training subset for online learning, which results in sequence lengths showed in Table B To ensure the distribution shift doesn't 't affect the conclusion, our experiments in the main paper are with the reshuffled version of the VTAB dataset. However, we provide additional results on the original VTAB datasets in Appendix A.7
0.8
Accuracy
Original-Train
0.4
Reshuffled-Trai
Original-Test
02
Reshuffled-Tern
Caltech101
Gfar100
ClevrCount
GevrDistance
DMLab
DTD
EuroSAT
Flowers102
KittDistance
PatchCamelyon
Pets
Resisc45
Retinopathy
SVHN
SmalINORBElevatin eeeetiiati
Sun397
dSpritesLocatton
dSprtesOOrentation
Dataset
Figure 4: Comparison of train and test accuracies on the original and reshuffled VTAB datasets.
A.3 2- 2-STAGE ONLINE AND OFFLINE IMPLEMENTATION
The 2-stage implementation is straight forward: first stage, we compute the cross entropy losses for all models at each timestep and store the results; second stage, we use model switching to compute the final codelength (see Algorithm
15

---

Published as a conference paper at ICLR 2023
Group
Dataset
Number of Examples
Caltech101
3060
Cifar100
50000
DTD
3760
Natural
Flowers102
2040
Pets
3680
SVHN
73257
Sun397
87003
EuroSAT
21600
PatchCamelyon
294912
Special
Resisc45
25200
Retinopathy
46032
ClevrCount
70000
ClevrDistance
70000
DMLab
88178
DSpritesLocation
663552
Structured
DSpritesOrientation
663552
KittiDistance
6770
SmallNORBAziing
36450
SmallNoRBElevation
36450
Table 3: Number of examples in the reshuffled VTAB datasets.
Algorithm MDL with Readout Model Switching (2-Stage)
Stage 1:
Require: data D (Fe, 1 K readout models; subroutine that updates the parameters given
dataset Update Initialize: model parameters .i...... 0x: an empty list store loss per step and per model
2: for = 1 to I do
3: Compute next-step loss of K readout models: C := log (ue log p(ue (0:;; 0s)
4: Store C. .... LK to L.
5: for k 1 to do
6: Update parameters: 0g +- UUpuaeePraamtters(.s Dee)
7: end for
8: end for
9: return
Stage 2:
Require: Cross entropy results from stage L; initial model probability p({1) and switching strat-
egy (())
1: Initialize:
log p(E:) an empty list Q store posterior p(E.ID=c))
2: for f- 1 to T do
3: Compute log p(E,, - |zt-1 log Eς_, exp (log ((Ee/r-1))
4: Compute posterior (5, (D and store in Q:
p(. (D=t) log p - log Ee, p(E,, 1z"1)=
5: Get .!..... LK from L and combine K models to update p(E,, y (xt): s st -L +
6: end for
7: Compute total codelength Lswitch &- - log Ee, exp(s)
8: return Lewitch and Q
A.4 ACCURACY-BASED EVALUATIONS
For both linear and a combination of MLP evaluations, perform hyperparameter search learning rate, weight decay and EMA with batch size 1024 (see Table 4 for details). We split the dataset into training and validation where the validation contains 10% of the data. We choose the best performing hyperparameter on validation set and report top-1 accuracy of the test set in
16

---

Published as a conference paper at ICLR 2023
We treat the readout models type of the hyperparamettrs, choose the one with the lowest loss on validation set and report the test set accuracy.
Table La and Table lb
Parameter
Values
Batch size
1024
Learning rate
{ le-4, 3e-4, le-3, 3e-3 } 0.0, le-6, le-4, le-2 le-4, le-2, 1.0]
Weight decay
EMA step size (Polyak averaging)
Table 4: Hyperparameters for accuracy-based evaluations on VTAB.
Architecture
Obj-
Acc (%)
#
Rank
ViT/B16
DINO
76.01
2
2.26
ResNet-50
BYOL
77.93
2.37
2
ResNet-50
SUP
75.51
3.82
16
MAE
74.10
5
3.84
4
16
SUP
72.46
4.18
ResNet-50
SimCLR
74.35
4
4.53
Table 5: Comparison of average accuracy v.s. average ranking for summarizing performance on multiple datasets (19 datasets from VTAB) using linear evaluation protocol.
Average accuracy v.s. Ranking on multiple datasets. Table 5 shows the difference on model selection when using average accuracy and average ranking for aggregating performances on mul- tiple datasets. We see that the two summarization methods don't 't have the same results for model selection. The reason why it's problematic to average accuracy over multiple datasets is discussed in Demšar (2006). In brief, averaging over accuracy is only meaningful when the results on differ- ent datasets are comparable and averages are susceptible to outliers. Since the VTAB benchmark consists of 19 datasets for classification tasks of very different nature, averaging over the accuracy an ill-conceived practice and we choose to use average ranking for presenting our results.
A.5 IMPLEMENTATION DETAILS
A.5.1 VTAB
Frozen Encoders frozen encoders, we use the readout models: linear layer and MLP 1-
7 hidden layers. We use AdamW the optimizer and hyperparameters listed in Table 6 The switching probability for fix share a; = 10/t. Images are center cropped and resized to 224 x 224.
Parameter
Values
Batch size
32
Number of replay-streams
3, 10, 30, 100)
Learning rate
le-4, 3e-4, le-3, 3e-3
AdamW 8
0.5, 0.7, 0.9
Weight decay
0.0, 1e-6, lc-4, le-2
EMA step size (Polyak averaging)
le-4, le-2, 1.0
Table 6: Hyperparameters for computing MDL frozen encoders on VTAB.
Train from Scratch This section correspond to the training from scratch experiment VTAB presented in Section We use SGD with nesterov momentum for training from scratch. We take the center crop and resize the image 224 224 for preprocessing. No readout models is needed
17

---

Published as a conference paper at ICLR 2023
the encoder is trained on the relevant task. For each VTAB task, uniformly randomly select 300 configurations from the hyperparameters listed in Table 7
Parameter
Values
Batch size
32
Number of replay-streams
{ 50, 100, 150, 200
Learning rate
{ 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0
momentum
0,9
Weight decay
{ 1ee-5, 3e-5, le-4, 3e-4, 1e-3, 3e-3, le-2,
EMA step size (Polyak averaging)
1e-3, le-2,
Table 7: Hyperparameters for computing MDL with training ResNet-50 from scratch VTAB.
Fine-tuning This section correspond to the fine-tuning experiment on VTAB presented in Sec- tion 4 We use ResNet-50 encoder for the fine-tuning experiment. Readout model is linear layer. The encoder parameters are initialized with the pre-trained parameters of BYOL, but updated on the downstream task. We use SGD with nesterov momentum for fine-tuning pre-trained encoders. Images center cropped and resized to 224 224 for preprocessing. For each VTAB task, uniform randomly select 300 configurations from the hyperparameters listed in Table 8
Parameter
Values
Batch size
32
Number of replay-streams
{
10, 30, 50 }
Learning rate
{ lc-4, 3e-4, 1c-3, 3e-3, 0.01, 0.03, 0.1, 0.3, 1.
momentum
0.9
Weight decay
{ 0.0, 1e-5, 3e-5, le-4, 3e-4, 1e-3, 3e-3
EMA step size (Polyak averaging)
{ le-3, le-2, le-1, 1.0
Table 8: Hyperparameters for computing MDL with fine-tuning pre-trained encoders VTAB.
A.5.2 IMAGENET
Frozen Encoders Images preprocessed with spatial augmentation during training: randomly cropped, randomly flipped and resized to 224 x 224. We use the readout models: linear layer and MLP of 1-3 hidden layers. We use Adam W as the optimizer and hyperparameters listed in Table The switching probability for fix share if a, = 1/t.
Parameter
Values
Batch size
512
Number of replay-streams
10, 30, 50
Learning rate
3e-5, le-4, 3e-4, lc-3)
AdamW B
0.5, 0.7 }
Weight decay
le-2, 1.
EMA step size (Polyak averaging)
{ 0.01, 1.
Table Hyperparameters for computing MDL of frozen encoders on ImageNet.
This section correspond the fine-tuning experiment ImageNet presented in Sec- [ We use ResNet-50 encoder for the fine-tuning experiment. Readout model is linear layer. The encoder parameters are initialized with the pre-trained parameters from BYOL, but updated on the downstream task. We use AdamW W and spatial augmentation for preprocessing. Images are randomly cropped and resized 224 224. We use the same hyperparameters as the frozen encoder experiments listed in Table
Fine-tuning
18

---

Published as a conference paper at ICLR 2023
Train from Seratch We train standard ResNet-50 architecture from scratch ImageNet and follow the experimental setting described in (Bornschein et al., 2022). In particular we use replay- streams training with uniform replay, AdamW, use randaugment for data augmentation, and forward calibration. We run a random hyperparameter search by sampling 50 hyperparameter configurations from the intervals listed in appendix A.5.2
Parameter
Distribution
Values / Interval
Batch size
fixed
512
Number replay-streams
log-uniform
25 ... 100
Learning rate
log-uniform
le-4 3e-3
Adam W c
log-uniform
le-4
EMA step size (Polyak averaging)
log-uniig log-uniform
le-3 le-2
Weight decay
log-uniform
le-4 ... 1.
Label smoothing
fixed
0.01
A.6 COMPARISON OF SWITCHING STRATEGIES
We describe briefly other switching strategy we experimented with our evaluation framework:
Bayesian mixture: switching probability is after the first step.
(VITT))))) log Σ p(k) plye On (oca,
Elementwise mixture: at each step, it is allowed to switch to another model with a fixed probability which doesn't depend the current model used.
LEW Φ) log Σ[ pludor 0z. (d (cE))
E1/7 t=1
Where w(E)) fixed mixture weight which doesn't 't depend on E
Switch distribution: A sophisticated switching strategy which augments the transition with two latent variables u and for unstable and stable chain respectively. Once switching the stable chain, stay with the same model for the rest of the sequence. For more details, see Koolen & De Rooij 2208))
Figure compares comparess the above switching strategies with the fixed share with decreasing switching rate on ImageNet using ResNet50 pre-trained with BYOL. Bayesian mixture and elementwise mixture perform worse than fixed share by a large margin. Switch distribution is very close to fixed share performance.
6000
Fixhhar(a==1tt
Bayesian Mixture
Elementwise
4000
SwitchDist
3000
1000
Number of Examples
le6
Figure 5: Regret of Bayesian mixture, elementwise mixture and switching distribution compared to the fixed share baseline on ImageNet. Pretrained model is B BOL with ResNet50 backbone.
19

---

Published as a conference paper at ICLR 2023
A.7 MDL ON VTAB DATASETS
We report the final MDL score using fixed share switching strategy on all the VTAB datasets in Table 10 To better compare the numbers, average the MDL score by the number of examples in the datasets. For cumulative MDL score, we should multiple by the number of examples which is reported in Table 3
Natural
Special
Siructured
SmallNORBA Aziimath
SmalNOOBBlleatioo
DSpeited ISpertesOOrentaaiin Orientatioon
DSSrrite Locution
PanhhCaanelyn
Clevriistance
Architectur
Objective
Caltechioh
CHfar000
Flowers 102
SVIN
Sun395
EurOAAT
Rosise45
Retinopatty
ClevrCount
DMLab
KituDhaance
DTD
Pos
ResNet-50
SUPP
0.81
1.12
1.35
1.23
0.35
0.82
1.42
0.14
0.17
0.45
0.72
0.91
0.87
0.90
0.14
0.26
0.58
0.51
0.88
ReaNot-50
SimCLR
1.33
0.66
1.54
0.58
0.29
ResNet-50
BYOL
0.97
0.69
0.70
0.77
ResNet-101
SUP
0.76
1.02
1.27
0.34
0.95
1.38
0.14
0.18
0.48
0.73
0.89
091
0.9
0.13
0.23
0.59
0.52
0.92
ReaNet-101
BYOL
0.72
0.86
1.17
L04
0.43
0.78
1.25
0.13
0.17
0.38
0.70
0.77
0.79
0.77
0.11
0.16
0.55
0.24
0.63
ResNet-152
SUP
0.99
1.32
0.99
1.37
0.15
0.72
0.4
0.18
0.23
0.50
0.92
ReaNot-152
BYOL
0.71
0.84
1.15
L.0
0.42
0.79
1.22
0.13
0.17
0,70
0.77
0.77
0.78
0.10
0.15
0.55
0.25
0.66
VTT/16
SUP
0.83
0.95
1.45
L
0.38
1.23
1.38
0.19
0.20
0.56
0.72
1.06
L17
1.14
0.63
0.38
0.64
0.61
1.00
WT/S16
MAE
1.83
2.01
1.76
2.10
1.54
1.64
2.27
0.17
0.15
0.64
0.71
0.97
1.16
1.09
1.75
0.27
0.64
0.59
0.69
VT/B16
SUP
VT/B16
0.94
0.76
1.15
0.13
0.68
094
0.17
0.19
0.55
0.28
0.57
VT/B16
MAE
1.12
1.21
14
1.64
0.93
0.93
1.00
0.13
0.14
0.47
0.70
0.70
080
0.92
0.03
0.14
0.58
0.29
0.48
WTL16
SUP
0.81
0.88
1.52
1.4
0.37
1.06
1.40
0.18
0.18
0.52
0.72
0.96
100
1.04
0.32
0.27
0.61
0.59
0.91
WT/L16
MAE
0.82
1.36
0.09
0.76
Table 10: Fixed share codelength per example reported for each VTAB dataset.
ResNet50
VIT/B16
Rank
Supervised
SimCLR
BYOL
Supervised
DINO
MAE
Pretrain Objective
Figure 6: Ranking for representations pre-trained with different objectives (lower the better). The box plot is drawn from 25% 75% quantile with horizontal line denote the median and red dot to denote the mean. The whisker denotes 1.5 IQR values. The threshold of the ranking difference to be significant for hypothesis testing is 1.894.
In VTAB, datasets in the same group are similar to each other, but there is a significant distribution shift between groups. And natrual datasets are closer to the ImageNet which is used during pre- training. To investigate if the transfer performance differs with input distribution shift, we report the average rankings by dataset group in Figure Although the number of datasets in each group is lim- ited and, therefore, the result from hypothesis test conclusive, we can still get some idea the transfer performance from the ranking difference. We can see that self-supervised representations have a better performance than the supervised counterpart on structured datasets, despite the latter perform decently on natural datasets. MAE generalizes well with dataset shift, but the performance on natural datasets suffers compared to other methods.
20

---

Published as a conference paper at ICLR 2023
ResNet50
WT/816
VIT/816
Rank
Rank
Rank
Supervised
SinCLA
BYoL
Supervised
DWO
MiE
Supervised
SimcLa
BYOL
Supervised
DIO
MAE
Supervised
SimCLAt
BYOL
Supervised
DINO
MAE
Pretrain Objective
Pretrain Objective
Pretrain Objective
(a) Natural Datasets.
(b) Special Datasets.
(c) Structured Datasets.
Figure 7: Ranking on natrual, special and structured VTAB datasets.
The thresholds for ranking
difference be significant 3.12, 2.92 and 4. respectively.
ResNet
VIT
Supervised
BYOL
Supervised
MAE
12
Rank
101
192
101
152
516
B16
516
B16
L16
Figure 8: Rankings of ResNet and VIT models different sizes (lower the better). The thresholds for ranking difference be significant is 3.65.
Data ordering and Variance of MDL score. Theoretically, different permutations of the data sequence results in different codelengths. We reckon that this an area for future work. Rissa- nen (1986) suggests taking the permutation with the shortest codelength, but it is computationally too expensive. Hansen & Yu (200T) notes that the codelength invariant over orderings when a Bayesian predictor is used in prequential MDL.. De Rooij & Grünwald (201)) points out that while the ordering of data affects the codelength, does not appear to cause too much trouble in practice. Here, we provide an empirical evaluation of the variance of metric. We evaluate ResNet50-SUP representation on Cifar 1 00 with 5 permutations and obtain the MDL score of 55906 with a standard deviation of 134. To give an idea, the representations that are ranked immediately above and be- low achieve the MDL score of 50969 and 60549 respectively. Therefore, we believe our evaluation method is stable and reliable, We would also like point out that the current practice of comparing the test accuracy is not immune to randomness, e.g. initialization, dataset split and randomization during training.
the original VTAB datasets, as well as the average rank. The distribution shift results in changes of the MDL score (colored in blue and red in Table TT}. The datasets where the distribution shift occur consistent with our previous assessment in Appendix|A.2 However, the overall rank of the pre-training method doesn't it seem be affected. This because the number of datasets affected by the shift relatively small and the effect of the shift is similar to all pre-training methods.
MDL Results original VTAB Datasets Table repports the MDL
21

---

Published as a conference paper at ICLR 2023
Natural
Special
Siraclured
DSprite DSprhiesOnieeuiii Oriemutionn
SmalNORRDA Azimmthh
SmmlLNORLElevaii
DSyrlis D8priissocaaion Locatin
Pacck DiclChmelyee
ClewDistanc
Reinoppthy
ClewCount
KintDeetnce
Rank
Sma397
EDNSAT
DMLah
RosNes-00
RosNes-00
SimCT.R
VED816
DINO
0.54
VI3116
Table 11: Fixed share codelength per example reported for original VTAB benchmark where some of the datasets have distribution shift. The number is color coded the codelength changes more than le compared to the reshuffled version of the dataset. Blue: codelength is longer than reshuffled dataset; red: codelength is shorter.
A.8 POSTERIOR THE READOUT MODELS
Figure 9 shows the posterior of readout models p(ć, = kjD where k E [1.... K] on ImageNet for 4 different architecture and pre-training methods: ResNet50 + SimCLR, ResNet50 + BYOL, VTT/B 16 DINO and ViT/B16 MAE. We use 4 readout models: hidden layers.
linear layer and MLP with 1-3
Rashva55n-80
VITI816-DINO
10
PORERSo Prbbariiyy
Podtatce Prbbabikyy
MIFI
70
Number Essamplas
165
Number
360
Namber Fawngles
305
Number of Soumppie
500
Figure 9: Posterior probability of the readout models at each timestep on ImageNet. Pre-trained methods are ResNet50 SimCLR, ResNet50 BYOL, VIT/B16 DINO and VIT/B16 MAE
A.9 DATA EFFICIENCY AND FINE-TUNING
Figure 10 shows the regret plots, relative to training from scratch, for training a linear layer on top of the pre-trained and frozen BYOL backbone, training MLPs (1-7 hidden layers) on top of the frozen BYO backbone and for a linear readout but while fine-tuning all the encoder parameters. The network architecture ResNer-50 all cases.
22

---

Published as a conference paper at ICLR 2023
Tel
Cifr1100
e3
Caltech101
le1
3e
Rowers102
0-
Regeet
Regret
Regret
Regpet
4
15
Number Examples
Number Examples
Number of Fxumples le]
Number Fxamples
163
Pets
304
SVHN
125
Sun397
00
Supervised-FromScree
BYOL-Linear
-0.5
Regret
Negget
BYOL-ALL
BYOL-Finetune
-15
2
Number of Examples
Number of Examples 1o4
Number of Examples 1n4
1e3
EuroSAT
1e4
PatchCamelyon
1o4
Resisc45
163
Retinopathy
-0.5
Regret
Segret
Regret
Rearet
-10-
-1.-
-2
0.0
0.5
10
15
20
Examples
Examples
Examples 1e4
Examples I6l
1n4
ClevrCount
104
ClevrDistance
1n4
DMLLab
1e3
KittiDistance
25
0.0
0.5
20
1.5
Regret
0.0
Regret
Regret
Regret
-0.5
0.5
00-
-1.5
-0.5
-2.0
20000
60000
Number Examples
Number of Exemppes
Number of Exemples DSpritestocation
Number of Examples
SmallNoOBBAziuuhh
DSprntesOOrientttion
1.0
05
2.0
1.5
Regret
0.5
0.0
Regret
0.0
0.5
0.0
-15
-0.5
Number Examples 164
Number Examples 164
Number Examples 165
Number Examples 1e5
Data efficiency on 19 VTAB datasets. Four methods are compared: training from scratch, training a linear layer with frozen encoder initialized with BYOL, training MLPs with frozen en- coder initialized with BYOL and fine-tuning the encoder initialized with BYO.. We plot the cumu- lative next-step log loss relative to training from scratch as the baseline (in nats; lower is better).
Figure 10:
23


Reviews:
[{"review_id": "SzpkpdCw0I", "reviewer": "Reviewer_mVad", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a metric to compare representations in deep learning architectures. This metric is based on thinking of representations as a model selection problem and using the minimum description length principle. The authors also propose the intriguing idea of model switching where different architectures can be used depending on the task demands and characteristics. The metric is applied to traditional architectures for vision encoders across multiple tasks. ", "summary_of_the_review": "This is an interesting and fresh look at the problem of how to characterize and compare different models and their representations. An ideal metric for comparison would take into account not just how the models perform in one or a handful of tasks, but rather a more comprehensive view of the complexity of the architecture, their sample efficiency in training, performance, and generalization. This work is a step in the right direction by introducing new metrics that can satisfy many of these requirements.", "strength_and_weaknesses": "Strengths\n\nThe paper introduces a fresh look at how to compare different architectures and their associated representations of features. \n\nThe proposed metrics claim to take into account the model complexity and data efficiency, which are either not incorporated or only indirectly addressed in other metrics such as when merely comparing task performance.\n\nThe paper presents an extensive empirical demonstration of the use of the proposed metrics across multiple tasks, datasets, and interesting problems such as model scaling.\n\nWeaknesses\n\nThe main results are shown in somewhat unorthodox ways. The rank variable used throughout the figures is a bit hard to interpret. It would be great to expand on the second paragraph of the Experiments section to better explain how this is computed, given that this is critical to interpreting all the figures. Further, it would also be useful to understand how these values relate to other metrics such as accuracy in the datasets, performance for a given number of training examples, and even to show the MDL scores themselves.\n\nWhat is less clear to me about the model switching proposed here is who and when decides when to switch models. What would be particularly useful is a strategy to be able to switch across models for an arbitrary novel task and dataset. It is not clear to me that this is what the authors are showing. Given the results, it is less interesting to decide a posteriori, which models performed best and when. Perhaps the authors are showing predictability in model switching to new tasks/data but this was not clear to me.\n\nI love theorems and theorems can obviously be more valuable than empirical measures. However, it was a bit difficult to follow in this case, what the connection was between Theorem 1 and the empirical results shown in the subsequent figures (also given that some of the model assumptions are not followed in the architectures as the authors clearly spell out). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RgO4ezdHgWe", "reviewer": "Reviewer_pMpS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a method to evaluate the representations of deep learning models by combining predictions of readout models. The readout models are selected based on a switching policy according to the Minimum Description Length, which is an evaluation metric for probing models that consider both model accuracy and complexity. The switching policy is determined by a stochastic matrix that defines the probability of replacing the current expert model. With the proposed method, the author evaluates the representation efficiency of popular deep learning with different scales on a set of downstream tasks and investigates the data efficiency.", "summary_of_the_review": "This paper proposes an evaluation metric for representations of deep learning models that rely on switching within multiple read-out models based on MDL. The author has compared different models, i.e., ResNet and ViT, based on the proposed metric. However, a major concern is that: Is the proposed method could evaluate representations effectively, or at least better than the current methods? If the authors plan to show the stability and reliability of the proposed methods, there should at least be some experiments that show the failure of compared evaluation metrics, such as by a weighted average over the performance of some downstream tasks. Furthermore, if the authors want to propose an effective evaluation metric, it is necessary to show consistency when selecting different switching strategies.\n\n----Post Rebuttal----\n\nThe author has answered most of my questions and the revised version has motivated me to move my rating from 5 -> 6. For me, the author has proposed an effective (or stable) method to evaluate the representation by combining evaluation over different readout models. However, I still have some concerns about the efficiency of the switching policy, which may be left in future works.\n", "strength_and_weaknesses": "Strength: \nThe author proposes a new method to evaluate the representations of deep learning models based on a selection of different readout (or expert) models according to the MDL. If the author could adequately approve the advantage of the proposed metric, this could bring a new angle for model evaluation.\n\nWeakness:\n1. Although the authors have claimed that a rigorous method for representation evaluation is lacking for deep learning models, they didn’t show the advantage of their evaluation methods over others, especially in the experiments. Furthermore, for most unsupervised models, the representation quality is evaluated indirectly by multiple downstream tasks, where the model is required to show consistent improvements on multiple tasks to show its advantage. If the author claims that their method could be more accurate in evaluation, they should at least show some empirical comparisons, where the multi-task evaluation metrics fail but the readout switching method succeeds.\n\n2. The proposed switching method is based on Minimum Description Length, which is used as an alternative method for language model evaluation where the probing tasks fail to reflect differences in representations [1]. To adapt this in the vision task (as the models used in the experiment are vision models), the author should show that the vision model suffers from similar problems, or the MDL could provide a better evaluation solution than current solutions.\n\n3. Even if the author's claim is true, the representation could still be compared by a combination of the performance on several probing models without switching. If the author wants to show the advantage of the proposed switching method, there should be an experiment that compares the evaluation difference of using a) the proposed switching method and b) a combination of probing models without switching.\n\n4. The author claims that different probing models could bring differences during the evaluation of the representation of deep learning models. However, the selection of switching policy may also affect the evaluation metric. If so, what is the principle to find the best selection strategy?\n\n5. According to Algorithm 1, the switching happens at each time step when taking in a new pair of data (x_t, y_t). So what will happen to the proposed metric if the order of evaluation data changes?\n\n6. For the experiment in 4.1, the authors apply different supervision methods on the two models (looks like the “Supervised” is the same for both?). Why not just apply the proposed five supervision methods to both models for comparison?\n\n7. Although the author has listed the evaluation scores of different models and shown the rankings, there remains a question of whether the models behave consistently with this conclusion. For example, in Figure 4, the author shows that a deeper network could be worse than a shallow one (like ResNet, and ViT-B16 vs ViT-L16) in “Supervised”. This conclusion is quite questionable as shown in [2][3]\n\nWriting:\n1. For formulas, especially in section 2, it would be easier for the reader to review if the formula could be put in a separate row.\n2. In the last paragraph of Regret Bounds, DNN has been defined at the beginning.\n3. In 4.1, what does it mean by “pre-train on supervised losses”? Is the supervision method the same for ResNet and ViT?\n\nReference:\n\n[1] Voita, Elena, and Ivan Titov. \"Information-theoretic probing with minimum description length.\" arXiv preprint arXiv:2003.12298 (2020).\n\n[2] Gao, Shang-Hua, et al. \"Res2net: A new multi-scale backbone architecture.\" IEEE transactions on pattern analysis and machine intelligence 43.2 (2019): 652-662.\n\n[3] Liu, Ze, et al. \"Swin transformer: Hierarchical vision transformer using shifted windows.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[4] Khan, Salman, et al. \"Transformers in vision: A survey.\" ACM computing surveys (CSUR) 54.10s (2022): 1-41.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ORUL16EbWG", "reviewer": "Reviewer_4XSa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a metric for comparing learned representations, and furthermore doing it so that models can be switched based on data. That sounds useful for a variety of purposes. That said:\n\nThis paper was assigned to a completely wrong reviewer; an indication of some sort of a failure in the ICLR process. I don't have the necessary background to appreciate much of this, and reading the paper very carefully would have taken too long -- and probably wouldn't have been too useful for the review process anyway. I have informed the AC that I'll essentially skip the review by putting in a minimum confidence opinion that should not count towards the outcome.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "-", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FcXq04s_nA", "reviewer": "Reviewer_Jszx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors claim a new metric on the quality of representation, Minimum Description Length (MDL) with model switching. \nWith the representation model and readout model given, the proposed metric is the optimized upper bounds for the regret in switching readout model with respect to MDL loss. \nThe proposed metric can also evaluate data efficiency on online learning.\nFor the experiment, authors use ImageNet and VTAB database with ResNet-50 and ViT/B16 architectures and rank the representation quality with the claimed metric. ", "summary_of_the_review": "Although the authors provides a technical algorithm for obtaining a novel representation quality metric, it lacks theoretical justification and empirical evidence that the metric is feasible to assess quality of representation. \nThus, I reject this submission.\n\n**Post Rebuttal**\n\nThe revised script included discussions and materials that I expected for supporting the argument of the paper. Thus, I have updated my scores.", "strength_and_weaknesses": "## Strength\n\n* **Novel approach in evaluating representation quality.** Authors provide a new way of assessing representation quality, and their algorithm for obtaining this metric is straightforward.\n\n* **Adopting multiple readout model.** Good representations may not store the crucial information of data in a linear shape. Adopting multiple readout model can alleviate this concern. It was interesting to see data efficiency of each readout models.\n\n## Weakness\n\n* **Justification on using model switching process.** The switching model forces to change the status of $\\eta$ in after some time span, but the case shown in Figure 1, for example, the optimal selection should stick to one model after observing enough amount of data. The switching process is not likely to achieves the optimal sequence of readout model selection. This gap must be discussed. Instead of following a process with discrete status, using an ensemble method may relieve this problem, vaguely speaking.\n\n* **Justification on regret bound.** As acknowledged in the paper, the regret bound (2) holds for exponential family with MLE $\\theta$. Thus, this upper bound crucial for constructing Algorithm 1 may not hold when the readout models include MLP.\n\n* **The claimed metric needs an empirical evidence about the quality of the metric** to assert that the proposed method is feasible. E.g., comparing the MDL rank with quantitative performance metric of the represented data on practical tasks such as classification or domain adaptation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "05u2gBx9hg", "reviewer": "Reviewer_GYkn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this submission propose to use the Minimum Description Length (MDL) principle to evaluate the quality of the learnt representations in DNNs. In particular, they follow the prequential approach to estimate the description length, which assumes that *good* representations should allow for training prediction models for down-stream tasks in a data-efficient manner.\n\nConcretely, the authors train a set of K models in a prequential manner (with SGD) and use a switching strategy to choose the best model for a given number of training samples. The final code length is then computed as the cumulative prediction loss of this model ensemble. \n\nTo evaluate their proposed method, the authors perform experiments with various DNN architectures (ResNets and ViTs of different sizes) on a wide range of datasets. Further, the authors discuss insights obtained from their experiments, such as regarding the pre-training paradigm used to train the encoders (e.g., supervised vs. self-supervised), the model size, or the transfer learning dataset.", "summary_of_the_review": "As discussed above, I think the proposed approach is interesting and could be a valuable addition to the literature on evaluating DNN representations. However, in its current state (see weaknesses), I think the paper is not ready for publication.\n\nEDIT: After the authors' response, some of my concerns were addressed and I therefore raise my score to \"6: marginally above the acceptance threshold\".", "strength_and_weaknesses": "### Strengths \n- Given the importance of self- and unsupervised representation learning and the variety of methods to do so, evaluating the learnt representations fairly and thoroughly is of great interest to the community. With this submission, the authors provide an interesting approach for evaluating the model representations beyond linear probing.\n- The authors evaluate the proposed method on a wide range of experiments, which allows for discussing and analysing differences in the learnt representations with respect to different dimensions of interest (pre-training objective, model size, down-stream dataset.)\n- The idea of using MDL to evaluate the model representations is convincing, as it allows to do so simultaneously with models of different complexity whilst still capturing the representations quality in a single number.\n\n### Weaknesses\nWhile I find the idea of the proposed method intriguing, I have issues with several aspects of the current manuscript. \n\n- **Novelty and contextualisation:** The method is presented with an emphasis on the idea of using MDL for evaluating the learnt representations, which gives the impression that this is in itself novel. However, as the authors also state in the background section, Voita & Titov (2020), as well as Yogatama et al. (2019) have already investigated MDL for evaluating representations. Unfortunately, from the current discussion of the related work, the differences to prior work and the novel aspects and their relevance of the proposed approach do not become entirely clear. More generally, many statements lack appropriate references (e.g., the entire introduction has only 3 references), which will make it difficult for readers unfamiliar with the topic to follow.\n- **Benefits and evaluation of the proposed approach:** While the proposed approach is generally well-motivated, the practical benefits over linear readout models do not become clear. How do the rankings produced by the proposed approach compare to rankings obtained from simple linear readout models? In what sense are the rankings \"better\"? What are tangible benefits? \n- **Discussion of limitations**: One of the key advantages of linear probing or KNN evaluations of the learnt representations is their simplicity; i.e. the result is largely independent of any hyperparameter settings and easy to obtain and compare. The proposed approach combines many different DNN architectures with varying complexity, which all need to be trained and more heavily depend on the optimisation procedure and hyperparamters, which makes the results potentially less stable and requires careful parameter tuning. I would appreciate if the authors could extend the discussion on this. \n- **Organisation and state of the manuscript:** In its current form, the manuscript is difficult to read, mainly due to the following reasons. \n\t- The structure of the manuscript is not easy to follow and the relevance of some sections is unclear. For example, section 3 abruptly switches to regret bounds and introduces Theorem 1, without discussing why this is relevant. In fact, the theorem is introduced and discussed, only to then say that the assumptions of the theorem do not hold in the context of DNNs, i.e. the context of this work. Similarly, an online version for the MDL computation is introduced and discussed, only to then say that this is not used in this work (\"To facilitate experimenting with different switching strategies, we use this 2-stage implementation for all our experiments.\"); this statement conflicts with the description of the experimental details in section 4 (\"We use the online learning framework as described in Section 3\"). I would appreciate if the authors could clarify why these sections are relevant and if the discussion around these sections were improved.\n\t- The mathematical notation could be improved. For example, in the fourth line of the \"Read-out Model Switching\" paragraph, the formula contains a \"p_{\\xi_t}\", which is referred to as \"p_k\" afterwards. Later, a p(k) is used, which refers to p({\\xi_1}) as far as I understand (however, p(k) = 1 iff k=1, which should simplify the L_{BM} equation further), which adds unnecessary notational ambiguity and makes it difficult to follow.\n\t- Finally, as discussed above, I would highly appreciate if the authors could add citations when appropriate (especially in the introduction). \n- **Experimental protocol and ablations**: Some of the experimental details remain unclear. How long are the individual models trained on each time step? Is the prequential method stable w.r.t. the order of the data? Why are some of the datasets reshuffled? This seems to be non-standard and I fail to see why this is required. What happens without reshuffling?\n\nMinor: I would recommend to carefully check the paper for grammatical correctness for the final version.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster