SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data
Authors: Ching-Yun Ko
Pin-Yu Chen
Jeet Mohapatra
Payel Das
Luca Daniel
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learning, from task-centric model design to task-agnostic representation learning and task-specific fine-tuning. As the representations of pretrained models are used as a foundation for different downstream tasks, this paper proposes a new task-agnostic framework, \textit{SynBench}, to measure the quality of pretrained representations using synthetic data. We set up a reference by a theoretically-derived robustness-accuracy tradeoff of the class conditional Gaussian mixture. Given a pretrained model, the representations of data synthesized from the Gaussian mixture are used to compare with our reference to infer the quality. By comparing the ratio of area-under-curve between the raw data and their representations, SynBench offers a quantifiable score for robustness-accuracy performance benchmarking. Our framework applies to a wide range of pretrained models taking continuous data inputs and is independent of the downstream tasks and datasets. Evaluated with several pretrained vision transformer models, the experimental results show that our SynBench score well matches the actual linear probing performance of the pre-trained model when fine-tuned on downstream tasks. Moreover, our framework can be used to inform the design of robust linear probing on pretrained representations to mitigate the robustness-accuracy tradeoff in downstream tasks.

Under review as a conference paper at ICLR 2023
SYNBENCH: TASK- BENCHMARKING OF PRETRAINED REPRESENTATIONS USING SYNTHETIC DATA
Anonymous authors Paper under double-biee blind review
ABSTRACT
Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learn- ing, from task-centric model design to task-agnostic representation learning and task-specific fine-tuning, As the representations of pretrained models are used as a foundation for different downstream tasks, this paper proposes new task aggostic framework, SynBench, to measure the quality of pretrained representations using synthetic data. To address the challenge of task-agnostic data-free evaluation, we design synthetic binary classification proxy tasks with class conditional Gaussian mixtures to probe and compare model's robustness-accuracy performance on input synthetic data and their representations. Since the synthetic tasks spare access to real-life data, SynBench offers holistic evaluation and informs the model design- ers of the intrinsic robustness level of the model given user-specified threshold accuracy. Moreover, the use of class conditional Gaussian mixture allows us derive theoretically optimal robustness-accuracy tradeoff, which serves as ref- erence when evaluating the tradeoff on representations. By comparing the ratio of crea-under-curry between the raw data and their representations, SynBench of- fers a quantifiable score for robustness-accuren accurrac performance benchmarking. Our framework applies wide range of pretrained models taking continuous data inputs and independent of the downstream tasks and datasets. Evaluated with several pretrained vision transformer models, the experimental results show that our SynBench score well matches the actual linear probing performance of the pre-trained model when fine-tuned on downstream tasks. Moreover, our frame- work can be used to inform the design of robust linear probing on pretrained rep- resentations mitigate the robustness-aceurag tradcoff downstream tasks.
INTRODUCTION
In recent years, the use of large pretrained neural networks for efficient fine-tuning downstream tasks has prevailed in many application domains such as vision, language, and speech. Instead of designing task-dependent neural network architectures for different downstream tasks, the cur- rent methodology focuses on the principle of task-agnostic pretraining and task-specific finetuning, which uses neural network pretrained on large-scale dataset (often in a self-supervised supervised manner) to extract generic representations of the input data, which we call pretrained representations for simplicity. The pretrained representations are then used
un-
foundation (Bom- masani et al., 2021) to solve downstream tasks by training a linear head (i.e., linear probing) on the data representations with the labels provided by downstream dataset, or by simply employing zero-shot inference. Moreover, to handle multi-modal data, one can use a similar neural network architecture (e.g., transformer) for multi-modal data representation learning and alignment. Sue- cessful examples following this new machine learning paradigm include the GPT-3 language model (Brown et al. 2020). the vision transformer (Arnab et al., 2021). and the CLIP image-text model (Radford et al. 2021). to name few.
As large pretrained models are shown to achieve ofate-of-the-an performance on a variety of down- stream tasks with minimal fine-tuning, there is intensified demand for using pretrained represen- tations from a large model for efficient finetuning. When gauging the usefulness of pretrained model, it is convention to compare the accuracy selected real-life tasks. However, this ap-

---

Under review as a conference paper at ICLR 2023
proach has two possible drawbacks: (1) if the underlying pretrained model has hidden risks, such as lacking robustness to adversarial examples, the standard accuracy cannot inform the risk as it does not correlate well (even worse, sometimes has negative correlation) with adversarial robustness (Su et al. 2018). Therefore, the trending practice of pretraining and fine-tuning also signifies immediate damage all downstream tasks. (2) the implications suggested by any "better" results on specific datasets are subjective to the datasets used for evaluation and could be inconclusive when the eval- uation datasets change (e.g. ViT-L/16 is reportedly performing worse than ViT-B/16 on 4 out of 27 linear probing tasks according to Radford et al. (002T) D. and is incompetent to ViT-B/16 on finetumed medical tasks (Okolo et al.) [022] Tummala et al. 2022 Consequently, an ideal pretrained model should entail both good accuracy and adversarial robustness, and the level of goodness can be mea- sured a task-agnostic manner. To address this emerging challenge, we propose a novel framework named SynBench to evaluate the quality of pretrained representations, in terms of quantifying the tradcoff between standard accuracy and adversarial robustness to input perturbations. Specifically, SynBench uses synthetic data generated from a conditional Gaussian distribution to establish ref- erence characterizing the robustness-acee accurrac tradeoff based on the Bayes optimal linear classifiers. Then, SynBench obtains the representations of the same synthetic data from the pretrained model and compares them to the reference for performance benchmarking. Finally, we define the ratio of quantifiable metric of the quality of
area-under-curves in robustness-accuragg characterization pretrained representations. The entire procedure of SynBench is illustrated in Figure
Our SynBench framework features the following key advantages.
1. Soundness: We formalize the fundamental tradeoff in robustness and accuracy of the considered conditional Gaussian model and use this characterization as reference to benchmark the quality of pretrained representations.
2. Task-independence: Since the pretraining of large models is independent of the downstream datasets and tasks (e.g., through self-supervised or unsupervised training on broad data at scale), the use of synthetic data in SynBench provides task-agnostic approach to evaluating pretrained representations without the knowledge of downstream tasks and datasets.
Completeness and privacy: The flexibility of generating synthetic data (e.g., by adopting a dif- ferent data sampling procedure) offers a good proxy towards more comprehensive evaluation of pretrained representations when fine-tuned on different downstream datasets, especially the scenario when the available datasets not representative of the entire downstream datasets. Moreover, the use of synthetic data enables full control and simulation over data size and distri- bution, protects data privacy, and can facilitate model auditing and governance.
We highlight our main contributions as follows.
We propose SynBench, a novel task-agnostic framework that uses synthetic data to evaluate the quality of pretrained representations. The evaluation process of SynBench is independent of the downstream datasets and tasks and it applies to any model taking continuous data inputs.
Evaluated with several pretrained vision transformers, our experimental results show that the met- ric provided by SynBench well matches the model performance in terms of adversarial robustness and standard accuracy when finetuned on several downstream datasets. For example, SynBench- Score suggests that the Imagenet21k Ik pretrained network (ViT-B/16-ing 1k) improves with finetun- ing on Imagenet k (ViT-B/16), echoing with the higher CIFAR10 1 and CIFAR10-C linear probing accuracy of ViT-B/16.
We show that SynBench can be used to inform the design and selection of the hyperparameters in robust linear probing to mitigate the robustness-accuracy tradeoff when mee-tuned on downstream datasets. For example, conducting e-robust linear probing with selected by CeaeBench-coome gives ViT-B/16 0.6% increase in CIFAR10 accuracy and 1.3% increase in CIFAR accuracy.
2 RELATED WORK
Pretrained models in vision. In the past few years, much focus in the machine learning community has been shift to train representation networks capable of extracting features for a variety of down- stream tasks with minimal fine-tuning. Nowadays, many common vision tasks are achieved with the assistant of good backbones, e.g. classifications (Yu et al.) 2022 Wortsman et al. 20222 Foret
et al. 2020 Xie et al. 2020 Dosovitskiy et al. 2020
Chen et al. 2020a). object detection (Redmon
2

---

Under review as a conference paper at ICLR 2023
Generate conditional Gaussian
Representations
NICOI bouard
SynBench-Score
area A
area
area A area
netwox
SynBench
Figure 1: Overview of our SynBench framework. Step 1: generate class conditional Gaussian and form the inputs to the pretrained model: Step 2: gather rendered representations; Step 3: measure the expected robustness bound under a range of threshold accuracy for both input raw data and their representations according to equation and obtain the expected bound-threshold accuracy plot; Step 4: calculate SynBench score by the relative area under curve of the representations (area B) to the input data (area A area B) in the expected bound-thresher accuracy plot.
& Farhadi 2017: Liu et al. 2016 segmentation (Chen et al. 2017 Xie et al. 2021). etc. Among the popular backbones, vision transformers (ViT) (Dosovitskiy al. 2020) have attracted enormous interest. VITs stem from Transformers (Vaswani et al. (017) and split image into patches, which are then treated as tokens as for the original Transformers. We will exemplify the use of SynBench using several pretrained ViTs.
Benchmarking pretrained models. Since pretrained models are used as foundation for different
downstream tasks, it is central to transfer learning (Neyshabur et al.) 2020 Pruksachatkun et al.
2020), and also tightly related to model generalization (Qiao et al.) 2020; Carlucci et al. (019). To
benchmark the performance of pretrained model. it is convention apply the pretrained model for a number of popular tasks and conduct linear probing on the representations (Chen et al. 20206]
Dosovitskiy et al. 2020 Chen et al. 2020a [202]. Besides linear probing, evaluation frameworks have been proposed based on mutual information (Bachman et al.) (019) and minimum description length (MDL) (Blier & Ollivier 20188] Voita & Titov (020)), which are reliant on the label informa- tion of the downstream tasks and are hence task-specific. Moreover, recent work (Whitney et al.) 2020) also discussed the sensitivity of validation accuracy (nonlinear probes) and MDL to evaluation dataset size, and proposed variant of MDL and sample complexity based quantifier that depends on the data distribution.
It was not until recently that more fundamental questions are brought up related to the pretrained models [Bommasani et al., 202T Tran et al. 2022 Zhang Ré 2022). Lately, [Bommasani et al. (021) raised practical concerns about the homogenization incentivized by the scale of the pre- training. Although the homogenization might help in achieving competitive performance for some downstream tasks, the defects are also inherited by all these downstreams. On that account, more careful study of the fundamentals of pretrained models of paramount importance. Tran et al. (2022) was dedicated to explore the reliability of pretrained models by devising 10 types of tasks on 40 datasets in evaluating the desired aspect of reliability. Furthermore, it is pointed out by Zhang & Ré (2022) that pretrained models may not be robust subpopulation or group shift shown in 9 benchmarks. The ad adresarial robustness is benchmarked by authors of (Shao et al. 2021: Paul & Chen 2022), where Paul Chen 2022) demonstrated the superior robustness of ViTs through Im- agenet and SShoo et al. (2021) conducted white-box and transfer attacks on Imagenet and CIFARIO.
Optimal representations. In the seminal work of deep representation theory, Achille & Soatto (2018) depicted the desired optimal representations in supervised learning to be sufficient for down- stream task, invariant the effect of nuisances, maximally disentangled, and has minimal mutual information between representations and inputs. Focusing more on generalization than compres- sion, DDboos et al. (2020) gave the optimal representation based on V-informmtiinn (Xu et al.) 2019) and probed generalization in deep learning. More recently, Ruan et al. (2021) defined the optimal representations for domain generalization. In (Dubois et al.] (022), authors characterize the ide- alized representation properties for invariant self-supervised representation learning. Specifically, idealized representation should be well-distinguished by the desired family of probes for potential invariant tasks, have sufficiently large dimension, and be invariant input augmentations.
SynBench differs from the above quantifiers as it does not need knowledge of any downstream data and has controls over the evaluation set size since could draw arbitrary number of synthetic data.
3

---

Under review as a conference paper at ICLR 2023
With the assumed synthetic data distribution, could theoretically characterize the robustness- accuracy tradeoff that is independent to the downstream tasks. Therefore, SynBench provides a predefined standard of the tradeoff, which serves the reference for representations induced by pretrained models. It should be also mentioned that, recently sim-to-real transfer paradigm has been leveraged to test the quality of real data, by projecting those onto the space of model trained on large-scale synthetic data generated from a set of pre-defined grammar rules (Marzoev et al. 2020). SynBench, though conceptually similar very high level, is different from that line of work as the focus of this work is to quantify the accuracy-robusten tradeoff of pretrained representations using synthetic data from conditional distributions.
3 SYNBENCH: METHODOLOGY AND EVALUATION
Without the knowledge of the downstream tasks and data, we aim to develop task-agnostic frame- work to evaluate some fundamental behaviors of the representation network. As robustness is key desired property, we probe the network check how representation networks are preserving robust- ness in the original data. It is crucial to note that the probing method developed herein specifies the bbbustness-accurace tradeoff' in the pretrained representations, can be used for understanding (and possible ranking) different pretrained networks.
On the whole, we want to measure the idcalized robustness-accureee tradeoff using synthetic data. By propagating the Gaussian realizations through different representation networks, can also measure the robustness-aceue acceraac tradcoff for representations. We start this section by giving the synthetic data and the corresponding optimal linear classifier of interest.
3.1 SYNTHETIC DATA AND OPTIMAL LINBAR CLASSIFIER
We consider imbalanced) binary classification problems with data pair (x, v) generated from the mixture of two Gaussian distributions P ,2. such that
xly = 1 ~ N Σ), æly = --1~~-- Σ),
N E), 141 + 1/2 = M (A, E), 141 /1 -
or equivalently,
(1)
2
2
where y E (11, Ply +1) T,
-1] -1) and jú A1 #1 We
focus on the class-balancee balanced case (r = 3) and defer the imbalanced case to the Appendix eliminate the factor of data bias and can benchmark
D When
sampling from this idcalized distribution, the robustness degradation in an ideal setting.
Let 1pp denote the lp norm of vector for any p 1. For given classifier f and input with f(x = y. where y is the predicted label, it is not rational for the classifier to respond differently x + 8 than
for a small perturbation level measured by | i.e. inconsistent top-1 predic- tion (Szegedy et al.] 2013 Goodfellow et al. 2014). Therefore, the level of (adversarial) robustness for a classifier can be measured by the minimum magnitude of perturbation that causes misclassi- fication, i.c. || == mins:fiax-"" f(a)) |6] |0|pp- For a generic function f. solving the optimization problem exactly is hard (Katz al., 2017;; Sinha al. 2018). Luckily, one can readily solve for the
optimization if f is affine (Moosavi-Dezioon
et al. 2016).
In the following, we will exploit this point and consider the linear classifier that minimizes the robust classification error. An ideal candidate classifier for the class conditional Gaussian (equation T specified by the robust Bayes optimal classifier (Bhagoji et al., 2019; [Dbribann et al., 2020). Specifically, it is stated that the optimal robust classifier (with robust margin e) for data generated
from equation [ is a linear classifier. We derive the following result as a direct application of the fact. To simplify the exposition, we focus the & norm in the remainder of this paper. We refer
the readers to Appendix C for general fn-norm results. We use "bound" to denote the minimal perturbation of sample.
Theorem 1. For any sample x. the optimal robust classifier for P Pa gives gives
(x-- 21 y x- (p-2
(i) the bound (decision margin) ||
2 (ax y x-- (1 (x)()))
(ii) the scaled bound ||||
For sample x e P P So it further gives
4

---

Under review as a conference paper at ICLR 2023
60
60
brund conee
93-
0-
32-
-2-
Dirabuststt. Bayes
0.2-robust Bayie
0ud-robust Bayes
4
B-robust Bayos
B-robusl Bayes
0.5
0.5
accursty
(a) 2D Gaussian
(b) Theoretical couustesss-accurag tradcoff
Figure 2: Illustration of robustness-accuracy tradeoff suggested by 6-robust Bayes optimal classi- fiers. Figure (a) depicts class conditional 2D Gaussian case with decision boundaries drawn by e-robust Bayes optimal classifiers of varying values. Figure (b) draws the theoretically character- ized robustness- accuraacy tradeoff given in Theorem liv).
(iii) the standard accuracy 4b p E- (R -xy (+))
(M- (A))l
(iv) the expected scaled bound E ((()) -
||
225 a aa)
(v-1(a))
+1,
where zy is the solution of the convex problem arg min 1=l 2) 2) 2 2) and denotes the CDF of the standard normal distribution.
We note that for samples drawn from P Pa x and = Ia all c-robust Bayes optimal classifier overlap with each other. For a general covariance matrix E, the c of c-robust Bayes classifier specifies the desired size of margin and demonstrates the robustness accuracy tradeoff. We give an illustrative 2D class conditional Gaussian example in Figure 2(a) where different e-robust Bayes classifiers give different overall margins at the cost of accuracy. Concretely, increases, the robust Bayes optimal classifier rotates counterelockwise, leading to increased misclassifications, but also overall enlarged margins.
3.2 OBJECTIVE
For given representation network parametrized by 0, pected bounds on synthetic data and their representations, under a thresholding accuracy a, i.c. EnrP, P. 33-Py.=-laa--Wnu
interested in evaluating the
[| l2 [x) = y. a > ad) for A A. and A, respectively, where Py and Py characterize the probability density function of the synthetic data manifold of interest, and zi is translation vector allowing non-symmetric class conditional Gaussian. Here, without the prior of applications, uniform distribution and La/ Vd is the normalized all-ones vector. For simplicity, we let La- Formally, we define (a.c(at)
assume f
14/ Vd, where denotes random variable that follows
as
Fo.c(as) =B.. [| fe(z)= als, mr] = Σ Fu [||||| fe(r) = v] ((())))2
where e) is the indicator function specifying the si, e-dependent that surpasses the threshold accu- racy de. We put the detailed derivation in Appendix A the following sections, we will illustrate how to calculate the inner expectation term E., A f((r) for both the raw data and representa- tions.
3.2.1 RAW DATA
For raw data synthesized from Pon. according to equation
the inner expectation term is given (a))² + 1, where denotes the
by Theorem (iv) E [||A2|12 fc( = v] /an a (0) standard accuracy.
5

---

Under review as a conference paper at ICLR 2023
The subscript in the expected scaled bound E || fe(x) indicates the raw data space, to distinguish from the scaled bound to be derived for representations. We highlight that Theorem Tiv) tradeoff. We plot the expected scaled bound as function of accuracy in Figure 2(b This tradeoff holds true when the data follow the conditional Gaussian ex- actly. In the proposed SynBench framework, treat this theoretically-derive
directly gives vobustness-accuren
pooustness-accurnan tradeoff' as the reference, enabling fair comparison among representations induced by different pretrained models.
3.2.2 REPRESENTATIONS
Given a pretrained network, we gather the representations of the Gaussian realizations and quantify the desired bound induced by robust Bayes optimal classifier in the representation space. When deriving the robust Bayes optimal classifier, we model the representations by a general conditional Gaussian zly
N E),, 21y - (p Σ). By Theorem I [i)), consider the optimal robust classifier for the modeled conditional Gaussian in the representation space to calculate the scaled bound || (= - "p-''(00--ppppp
for poreetly-classifier samples and the inner expectation is estimated empirically. It is worthwhile to note that now the Bayes optimal classifier does not necessarily coincide with robust Bayes optimal classifier even when we synthesized the dataset with an identity matrix covariance in the input space.
b- [-===(]]]]
3.3 ROBUSTNESS-ACCUREE QUANTIFICATION OF REPRESENTATIONS
Recall that we aim to calculate
Eg,c (o,) BAloo-Wonnnea [|A|| fc(x) = 3) L)(s
for both raw data and the representations (i.e. ||كك and ||=))) We treat the expected bounds of the raw data under a threshold accuracy as the reference. Given a representation network, we compare the expected bounds of the representations rendered by representation networks with the reference.
We take ~ 5) under the guidance of Theorem ji..). Specifically, as Theorem Miii) gives an analytical expected accuracy for class conditional Gaussian, we can obtain the desired range of 8 by giving the accuracy. Now since we are interested in having the reference as a class conditional Gaussian that yields accuracy from 55% to almost 100%, we set the starting and ending by the fact that (0.1) ~ 0.55 and (5) ~ 1.0. We reiterate that with more accurate modelling of the data manifold of interest, SynBench can give more precise capture of the pretrained representation performance.
When the data is perfect Gaussian (e.g. input synthetic data), we calculate Eg,((a)) with the help of Section 3.2.1 We note that A₂ is independent of pretrained network parameters 0, and all the e-robust classifiers fe in the input space overlap with each other when 2 = La. We hereby denote the desired metric on the input synthetic data by E(a;), distinguish from that on the represen- tations E (az). For representations, we calculate (a.) with the help of Section 3.2.2 and the expectation is estimated empirically. We show an example of the probing results in Figure
To integrate over all the desired threshold accuracy, we use the area under the curve (AUC) and give the ratio to the reference by
Se (a,caa)da
SynBench-Score(4.c. a.)
So, E(a)da
which correspond to A B in Figure performance on pretrained representations.
area B
Larger value of SynBench-Score implies better probing
4 EXPERIMENTAL RESULTS
In this experiment, we exemplify the use of SynBench given pretrained representation network. In order to compare among network attributes, is desirable to control the variates. In Table we list
severeal pretrained vision transformers (VITs Dosovitskiy al. 2020 Chen et al. 202T Caron

---

Under review as a conference paper at ICLR 2023
Raw data w/ robust/std. Bayes
Rep, 0-robust/std. Bayes
area A
Rep, data 0.2-robust Bayes
Rep. data w/ 0.4-robust Bayes
Rep, 0.6-robust Bayes
expected sound En
0.8
Rep. data w/ 0.8 robust Bayes
area B
0.6
0.
area A
0.6
0.2
0.8
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
threshold accuracy d:
Figure 3: An example of the bubustmess-accuraae quantification of representations for ViT-B/16. (Left) The expected bound-threshold accuracy plot for the input data (E(a,) and representa- tions (Eg,((ae) with e = 0 ~ 0.8. (Right) The desired quantification SynBench-Score((,... ar) = area
(refer to equation 3 for c = 0 (top) and e = 0.6 (bottom).
A-raa B
Model
Arch.
pretraining
fine-tuning
patch
parameters (M)
VIT-I1/6
VTT-Tiny
Imgn21k
Imgnik
16
5.7
VTT-R/16
ViT-Base
Imgn21k
Imgn1k
16
86.6
VIT-B/16-in2Ik
VrT-Base
Imga21k
No
16
86.6
VIT-B/32
ViT-Base
Imgn21k
Imgnik
32
88.2
VIT-L/16
Var-Large
Imgn21K 1k
Imgn1k
16
304.3
WTT-S/I-DINO
VTT- Small
self-Imgn1k
No
16
21.7
VIT-S/-DIINO
ViT-Small
self--mgnlk
No
8
21.7
WITBB/1-DINO
VIT-
self- ImgnIk
No
16
85.8
WT-B/S-DINO
VIT-Base
self-Imgnlk
No
8
85.8
Resnet50-SimCL SimbLLV
Resnet50
self-Imgnlk
No
144.4
Resnet101 SimCLRv2
Resnet101
self lifgll
No
261.2
Variation::
Model size
[TI.(TT,,.)/16., VIT-(S.B)/I6-DINO, VTT-(S.B)/-DDIOO,
(50, R-8imC1.R22
Finetuning
ViT-B/16/.-nn2122
ViT patch size
VIT-B/(16.22),
VIT-S/(16.22-DDINO. VIIBB/(1..22-DDINO
Table 1: Model descriptions.
al. 2021) and ResNet& Chen al. 2020c), and make comparisons our best knowledge. We note that the performance of these models might be nuanced by scheduler, curriculum, and training episodes, which are not captured in the above table. To provide comprehensive evaluation, we give SynBench-ScoreeOO, 0t) with a, ranging from 0.7 to 0.9, and from 0 to 0.8. Due to space limit, some a, results are deferred to the appendix. The runtime of SynBench depends on the number of outcomes of the discrete uniform distribution 2({0.1,5) For one ~ (4(0.1,5) 5), it costs 59 seconds to generate 2048 Gaussian samples, 37 and 81 seconds to obtain the SynBench-Sooon for ViT-B/ 16 offers quantifiable score for
and VIT-L/16 on one GeForce RTX 2080 super. SynBench-Seoe aabustness-accurae performance benchmarking and is intrinsically a task-agnostic evaluation that characterizes general behaviors of the pretrained representations without the knowledge and use of any downstream data.
Apart from the task-agnostic metrics SynBench-Scome developed in this paper, we also report linear probing accuracy on CIFAR10//magent and CIFAR Ii-AIl0-//mageNete (Hendrycks & Di- etterich, 2019) validate the standard and transfer accuracy (use the probing layer trained on We note that evaluating pretrained repre- sentations on real-life tasks is sensitive to the choice of tasks and the results may be inconclu- sive. For example, CIFAR 10/mmaeNee suggest that VIT-L performs better than VIT-B, wheras KITTI/SST (Geiger et al. 2012: Socher al. 2013) (Radford et al. 2021 Table 10), Food- Seg103 @W al. 2021 Table 8), X-ray images (Okolo et al. 2022 Table 4-8), magnetic resonance
CIFAR10/Imagen to probe CIFAR (TFARll0-/ImagNee--00

---

Under review as a conference paper at ICLR 2023
= 0.7
<= 0
0.2
0.4
0.6
0.8
CIFAR10
CIFARTO-c
ImageNNtt
ImageNet-c
ViT-B/16
0.33
0.37
0.32
0.20
0.06
95.0
81.2
83.8
66.4
VII-B/16-in2Ik
0.20
0.23
0.18
0.07
0.01
89.6
71.4
82.6
63.6
Table 2: Comparisons on the finetuning procedure in pretraining. The SynBench- of ViTs with or without finetuning pretrained representations, and the linear probing accuracy CI- FAR I0lameeeeeee and transfer accuracy on CIFAR I0cc///eeeee...c......
Model
<=]
<
0.2
c
0.4
0.6
CIFARIO
CIFARIO-c
ImageNet
ImageNet-c
VIT-T1/16
0.01
0
0
0
81.9
59.1
74.8
43.3
0.7
WTT8/116
0.33
0.37
0.32
0.20
0.06
95.0
81.2
83.8
66.4
VIT-L/16
0.26
0.33
0.30
0.22
0.11
98.0
90.3
85.3
72.2
VITTI/6
0
(1
74.8
0.75
WTT-/166
0.26
0.30
0.25
0.11
0.01
95.0
81.2
83.8
66.4
VIT-L/16
0.19
0.27
0.24
0.16
0.04
98.0
90.3
85.3
72.2
VTT6
74.8
0.8
VIT-B/16
0.19
0.23
0.17
0.04
95.0
81.2
83.8
66.4
VIT-L/16
0.12
0.21
0,18
0.09
98.0
90.3
85.3
72.2
VTT75/16
59.1
74.8
0.85
VIT-B/16
0.10
0.15
0.09
0
95.0
81.2
83.8
66.4
VIT-L/16
0.05
0.13
0,10
0.03
98.0
90.3
85.3
72.2
VIT-TW16
0
0
81.9
59.1
74.8
43.3
0.9
VIT B/16
0.02
0.04
0.01
95.0
81.2
83.8
66.4
VIT-L/16
0.04
0.03
98.0
90.3
85.3
72.2
Table 3: Comparisons the model sizes. The SynBench-Scorre of ViTs of different sizes, and the linear probing accuracy on CIFAR 10R 1/e and transfer accuracy on CIFAR 10-c/ImageNet-c.
= 0.7
==
c 0.2
0.4
CIFARIO
CIFARI0-c
ImageNet
ImageNet-c
VIT-S/I6-DINO
0.48
0.47
0.42
0.32
0.17
95.3
75.9
75.3
47.5
VTT-B/16-DINO
0.55
0.58
0.53
0.46
0.35
96.5
78.9
76.4
52.1
VIT-S/8-DINO
0.40
0.42
0.39
0.34
0,26
96.2
78.0
79.0
VTT-B/8- DINO
0.50
0.56
0.50
0.40
0.30
97.0
80.6
79.5
53.7
Res50-Simc SimCLRv2
0.66
0.50
0.50
0.48
0.48
95.0
80.1
77.5
47.4
Res101-SimCLRyv
0.60
0.64
0.55
0.51
0.48
95.6
80.9
78.7
The SynBench-Score of self-supervised pretrained representations and the linear probing accuracy on CIFAR10//mageNet and transfer accuracy on CIFARIO-c/ImagNNttcc
Table 4:
imaging (Tummala et al. 2022 Table 2-3) suggest the opposite. In contrast, because SynBench- Score intrinsically task agnostic evaluation of the pretrained model, its result is independent of the choice of tasks.
Fine-tuned pretraining representation. When applying pretrained representation network the desired task, one can either only train a linear head on top of fixed pretrained model, or perform fine-tuning of both the representation network and the linear head. Thus, Table [ we investigate how the fine-tuning process is affecting the representation networks. Specifically, both networks in Table 2 is pretrained on Imagenet 21k with supervision. After the pretraining. ViT-B/16 further finetuned on Imagenet 1k. Interestingly, SynBench-Score shows that this finetuning is beneficial as improvements are witnessed across all e with SynBench-Score, which well match the empirical observation give by CIFARIO and CIFARIO- and prior results (Kumar al. 202I).
Model size. In Table 3, we compare ViTs of different sizes. Specifically, we perform SynBench on ViT-Ti, ViT-B, and ViT-L with patch size being 16. The model parameter 0 is provided by the pretrained model. It is noticeable that ViT-B/16 is generally on par with ViT-L/16. When we set the threshold accuracy to be higher values, ViT-L/16 starts to give slightly better evaluations especially with larger One interesting observation is that for each model, SynBench- score is not necessarily monotonic in e, which indicates standard linear probing (i.e., c = 0) may not be the most effective way to probing pretrained representations in terms of robustness-accurag consistent with recent findings (Fan et al. 202I). See the "Robust linear probing" paragraph below for detailed analysis. We also observe that larger models exhibit better resilience (slower reduction as e increases. On Self-supervised pretrained representations (Table 4 we
performance, which is
in SynBench- observe that bigger models have higher SynBench-scoree ViT-B/16-DINO, NTT-B/16-DING and Des101-SimLLyV have bigger SynBench-scue scores compared to ViT-S/16-DING ViT-S/16-DINO, and Res50-SimCLRve
8

---

Under review as a conference paper at ICLR 2023
= 0.6
c= 0
<=
0.2
==
0.4
<=
0.6
0.8
CIFARIÓ
CIFARTÓC
ImageNet
ImageNet-cc
ViT-B/16
0.45
0.47
0.44
0.36
0.25
95.0
81.2
83.8
66.4
VIT-B/32
0.02
0.03
0.03
0.01
92.2
76.6
80.5
61.4
Table 5: Comparisons on the VIT patch size. The SynBench- of ViTs of different patch sizes, and the linear probing accuracy CIFAR10 and transfer accuracy on CIFARIO-C.
arg max,
rbbust lincar probing
SynBench- Score
(UCAARI,C)))AR
VTT-7716
0.1
70,5-0.7
VIT-B/16
0.2
88.1+1.0
VIT-L/16
94.2+00.4
VTT-B/I6-in
0.2
80,5+0,4
VIT-B/32
84.4+0.8
Table 6: CIFARIO and CIFARIO-C accuracy changes using e-robust linear probing with e
arg max, httpmma..ymaaaamaa SynBench-Scooe.
VIT patch size. We also compare vision transformer patch sizes in Table 5 Specifically, we give VTT-B with patch size being 16 and 32, individually. SynBench-Scores show an consistent trend as the model performance on CIFAR1O and CIFARIO-C. From Table 4 we see that the SynBench- score of ViT-S/16-DINO is on par with that of ViT-S/8-DINO, and ViT-B/16-DINO has higher SynBench-score than ViT-B/8-DINO. Although linear probing on CIFARIO and ImageNet do not share the trend, bigger patch size model (VIT-S/I--DINO) does perform better than smaller ones (ViT-S/8-DINO) on the PASCAL VOC (Everingham et al. 2010 segmentation task (Caron et al.
2021
Figure bottom table).
According to Table 3] 0.2-robust Bayes classifiers consistently give better scores compared to 0-robust (standard) Bayes classifiers with ViT-B/16 and ViT-L/16. This offers us quick way of gauging the suitable downstream robust probing parameter for the given pretrained model. We stipulate that observing a 0.2-robust Bayes classifier to yield better SynBench-Scor than O-robust Bayes classifier may suggest the pretrained network to produce representations that have better overall performance with linear classifiers trained by 0.2-robust linear probing. We validate this by performing robust linear probing representations rendered by VITs for CIFARIO classifi- cations. Results are shown in Table For a given pretrained model, let f and g be the pretrained net- work and linear probing layer, solve the optimization problem min, max | L(g(f 6)), v) using the PyTorch library Torchattacks] and 10-step PGDL2 attacks (Madry al. 2018) for adver- sarial training. From Table we see that robust linear probing with = arg max, SynBench-Score generally gives decent abbustness-accuran obtain a 0.6% and 1.3% increase in CIFAR 10 standard and CIFARIO- 10-c transfer accuracy with ViT- B/16 (as in Table D). more complete table on e-robust linear probing results with different is given in the appendix.
Robust linear probing.
tradeoff. For example, with robust linear probing, we
DISCUSSION AND CONCLUSION
In this paper, we propose new task-agnostic framework SynBench for benchmarking the robustness-accut accuracy performance of pretrained representations. SynBench is fundamentally task- independent and provides quantifiable score that does not reply on any real-life data. SynBench ex- ploits an idealized data distribution, class conditional Gaussian mixture, to establish a theoretically- derived robustness couustees-acccuacy tradeoff, which serves as the reference for pretrained representations. Finally, quantifiable score SynBench-Scoree is provided that compares the ratio of area-under-curg between the reference and the pretrained representations. We validate the usefulness of Syn on several pretrained vision transformers in giving insightful comparisons different model attributes (e.g. model size, fine-tuned pretraining representations, VIT patch size, linear probing).
While we delved into the manusteess-accurae performance of pretrained representations of vision transformers, we envision the SynBench framework to be further extended to other trustworthiness dimensions such as privacy, faireess, etc. Moreover, as the popularization of pretrained representa- tions in various domains (c.g. vision, language. speech), we foresee SynBench to be generalized to more domains, and shed light on task-agnostic benchmarking designs.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. The Journal of Machine Learning Research, 19(1):1947-198,, 2018.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836 6846, 2021.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness from optimal transport. Advances in Neural Information Processing Systems, 32, 2019.
Léonard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint
2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervie vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint
2017.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 13-18 18 Jul 2020a. URL https: htl.http://ppptt //ttteee//prooeeinggg. mlr. wwtps//eee/eeaeeeeeeeeee
html
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information pro- cessing systems, 33:22243-22255. 2020c.
Xiangning Chen, Cho-Jui Hsich, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. International Conference on Learning Rep- resentations, 2021.
Chen Dan, Yuting Wei, and Pradeep Ravikumar. Sharp statistical guaratees for adversarially ro- bust gaussian classification. International Conference on Machine Learning, pp. PMLR, 2020.
Edgar Dobriban, Hamed Hassani, David Hong, and Alexander Robey. Provable tradeoffs in adver- sarially robust classification. arXiv preprint 161 2020.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Reprexentations, 2020.
Yann Dubois, Douwe Kiela, David Schwab, and Ramakrishna Vedantam, Learning optimal repre- sentations with the decodable information bottleneck. Advances in Neural Information Processing Systems, 33:18674-18690, 2020.
Yann Dubois, Tatsunori Hashimoto, Stefano Ermon, and Percy Liang. Improving self-supervised learning by characterizing idealized representations, 2022. URL htt.thttp///xiiiie.et
2209. 06235
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Lijie Fan, Sijia Liu, Pin-Yu Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in Neural Information Processing Systems, 34:2 14:21480-21492 2142, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awarr minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2020.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations, 2019.
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International conference computer aided verification, pp. 97-117. Springer, 2017.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuming can distort pretrained features and underperform ditto---istribution. In International Con- ference on Learning Representations, 2021.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Alana Marzoev, Samuel Madden, M Frans Kaashoek, Michael Cafarella, and Jacob Andreas. Un- natural language processing: Bridging the gap between synthetic and natural language data. arXiv preprint arXiv:2004. 13645, 2020.
Seyed-Mohsen Dezaovol,,,,, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- pufer vision and pattern recognition, pp. 2574-2582, 2016.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? Advances in neural information processing systems, 33:512-523, 2020.

---

Under review as a conference paper at ICLR 2023
Gabriel lluebe Okolo, Stamos Katsigiannis, and Naeem Ramzan. levit: An enhanced vision trans- former architecture for chest x-ray image classification. Computer Methods and Programs in Biomedicine, 226: 10:1411 2022.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2071-2081, 2022.
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 5231-5247, 2020.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556- 12565, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Yangjun Ruan, Yann Dubois, and Chris J Maddison. Optimal representations for covariate shift. In International Conference Learning Representations, 2021.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robust- ness of vision transformers. arXiv preprint arXiv:2103.15670, 2021.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. robustness the cost of accuracy?-a comprehensive study the robustness of 18 deep image classification models. Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint (20X::22070774:: 2022.
Sudhakar Tummala, Seifedine Kadry, Syed Ahmad Chan Bukhari, and Hafiz Tayyab Rauf. Classi- fication of brain tumor from magnetic resonance imaging using vision transformers ensembling. Current Oncology, 29(10))4488-511 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Elena Voita and Ivan Titov. Information-theonn -hherreticc probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183-196. 2020.
12

---

Under review as a conference paper at ICLR 2023
William F Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, and Kyunghyun Cho. Evaluating representations by the complexity learning low-loss predictors. arXiv preprint arXiv:2009.07368, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, Pp. 23965-23998, PMLR, 2022.
Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven CH Hoi, and Qianru Sun. A large-scale benchmark for food image segmentation. In Proceedings of the 29th ACM International Confer- ence on Multimedia, pp. 506-506660 515, 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077 2021.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 10687-10698, 2020.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference on Learning Represen- tations, 2019.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01997, 2022.
Michael Zhang and Christopher Ré, Contrastive adapters for foundation model group robustness. arXiv preprint arXiv:2207.07180, 2022.
13

---

Under review as a conference paper at ICLR 2023
A OBJECTIVE
(at) =Baull - (b |اله| = a 1
-Ea,r [| (z( (z) y. ((, ail
Ba ||| 2 Se y. ((), ar) P(s 8i)
Er [|| fx) = y. ala;(s c) 0o]
Ex
||||| 2 [((x) = y) [a(s))))a
71
B PROOFS
Theorem 1. For any sample the optimal robust classifier fe for P Pragg gives
(i) the bound (decision margin) || (2-- 21 y -- (B (x)2x
[i- 2p (x))
(ii) the scaled bound ||l|
(1-22(()) (1))
[p]]
For sample x D. it further gives
(a- (iii) the standard accuracy ( (1 j (B (x)))>]
22x ot (aj 2
(iv) the expected scaled bound E || f((r) = v)
+ 1,
where 2x; is the solution of the convex problem arg min min (ft 2) - 2) and denotes the CDF of the standard normal distribution.
Proof. (i) Following Bhagoji et (2019) Dan al. (2020), the Bayes optimal robust classifier for the general non-symmetric conditional Gaussians P, PuI.a specified in equation is
fe(x) = sign { x
[i + M2 (
T
Σ M- (ī 255 (i))
(4)
2
where sign(-) is the typical sign function and z is the solution of the convex prob- lem arg min =llas =>TE-(( (ī 2). The corresponding decision boundary is at
((x + 8) m12 E-1 (ā 2x( = 0.
T
A arg min |101|2 s.t. 8 (Øt 2x(2)))
111 + 112 x
E (E 2x (A))
2
| 2 (i Autue
-
2
( 23 (p)
(ii) Since the bound || is subject to the positions of two Gaussians, we scale the bound by the | (p- and obtain
distance from Gaussian centers to the classifier,
(x(1)))
A 2 = 1x
Pit 1 ( 23 (p))
(
[5- - 21)
(p 2
(x
Pi tu TT (ft 23 (A))
1 1 -2>(7))
14

---

Under review as a conference paper at ICLR 2023
(iii) For sample E. consider the Bayes optimal robust classifier in equation calculate the analytical standard accuracy by
we can
= P(y = 1)P (z) = = 1
=P -f(((()=
=P
(x M 1
(p (0- ())) 0
2
=P
[A + - - 2)())) > 0] w ~~ M(
PP [ m (p (x) > / 1--16 5-1 25. (+))
~ (0,
n x x 2y (p))
-- (p 2y (1)
w Eー 23 (x)
=P
(0,
||C (Ip 2Ο (1()))
[ 1 2④ (
1 (>))))>>
-- 2
E (is - 2x (x))i
(iv) For sample x Paiang E. let denote the accuracy, & denote x 2 and denote - 2y (ū)). From (iii), we have that the standard accuracy of conditional Gaussian samples with the Bayes optimal (robust) classifier is $ w
so a" |wls (a) Since for binary classification, only care about accuracy from 0.5 to 1, so should have j w 0.
Now consider the classifier in equation and the corresponding scaled bound from (ii,
1(=-
#1 yT' (it 2x>(Ai]
1tT w| 77 m
1t w jt w
||اه
-ー - 2x()))]
Since tly (yp Σ), we have t" w'wly N Nlyii" w, w TT I expected scaled bound of the
w). When we only want to get the
samples, we have that
E 12 fx(x)= = v]
[1t"
w|
fx( = y)
pF w
E [1TT
fx(x) = 1]
E [1(T w| [((x) y -1]
2 m
27 w
E [ w y = 1, 1 w 2 0] +
E
[- t w = -1, w < 0]
2 w
2T w
Recall that 1f wly ~ N (yjii w, nh D w)) then by the mean of truncated normal distribution, it is true that
0 p
E T w y 1, w N 0]
7 w +
y Vu 5:
w
1
T
0 i Tw
v 2 ST w
w6 2 w w +
1 1
7 Verge
f w + Vu " w
/ T
t w
T w 0 E T w y - w <
y
- vst w
VT
w
0+
Ve Fow
jt w + ITw T
Verge
V (1
V w 2
15

---

Under review as a conference paper at ICLR 2023
Therefore
[| fe(x) v]
1= @= w
v Vw E
5:7
_ w
Vin To7
VT
VuT
n w
V2T
1
Var
By replacing
n² / S
by Φ (a), we got
[| fe(x)= = v] =
2tT a (aa) c (())² +1.
C GENERAL Ep RESULTS
We note that our results in Appendix B can be straightforwardly generalized to lp. Given an 6p adversarial budget E:
Theorem 2. For any sample the optimal robust classifier fo for P pr.pe gives
|| ||p (x- P (x-0x(00)
(i) the bound (decision margin)
122 IT ((a-)x(A)) Ia E -(u-ex(()))
(ii) the scaled bound ||
For sample x Phi.pr, it further gives
(iil) the standard accuracy [D( 11 11i (x)))x())
11111-11 (x))l
(iv) the expected scaled bound E
|| (((x) = y)
v2 a (a))
+ 1,
where 2x, is the solution of the convex problem arg min - -) 2) and + denotes the CDF of the standard normal distribution.
Proof. We follow the proof of Theorem D and consider the classifier in equation
By Hölder's
inequality, we now have the corresponding lower bound and scaled lower bound
1x
(1 t1 " (ī 235 (a))
||
1Σ (īt 25 (1)))l1
u12 -T ax. (a))
[ - 23 (x) 1
|
|5 (īt - 2x. 2x)(1l))lo
- (ī ())
( Ni tog
2 - -7x-(x
- (ā- 2x()))
where + 1. The remainder of the proof will then follows in Theorem p
Remark. In general, in the case that Σ is singular, we can apply the economy-size (thin) decom- position with nonzero eigenvalues Σ = FAFT. Then, with a general non-symmetric conditional Gaussians
z|y = ~ M(4, Σ), = - -1 N4 E),
apply proper translation to symmetric conditional Gaussians
=1 M1, E1; F xlyy -1 N(FT 12: E); +
FT x.
FT MI
112
y = N( F FT 111
+ u1f
= = (-
2
2
where f FT 41
16

---

Under review as a conference paper at ICLR 2023
D CLASS IMBALANCE
Given an lg adversarial budget S ||e||2. consider the conditional Gaussian in equation with
Σ = La (d by dl identity matrix) and general class prior then the following theorem holds.
Theorem 3. For any sample the optimal robust classifier fe for Pa pr. gives
(2- 2122 a(1-e/11alaa 9/2
(i) the bound (decision margin) ||
|| </ 2
(ii) the scaled bound ||l|
p(1-c/ (plpa) 4/2
I /11-e/11122) 9/2
For sample x Pai further gives
(iii) the standard accuracy 7 ( w (0-g/2) + (1 -) 1012
"" aatt//2).
wwllz
(iv) the expected scaled bound E ||A||2 fc(x) = =y]
1 9/2
||
1
1= w 9/2
liwll
Twlla
1 - 9/2
# ul
1et w
9/2
12
where = (n((1-))/)), w = [i(1 e/ [lll))) o and ④ denotes the PDF and CDF of the standard normal distribution.
Proof. (i) Consider the Bayes optimal lg e-robust classifier
(Dobriban al. 2020
Theorem 4.1)
f((x) = sign (-- + U2
ai1 </llã|l2) 9/2
(5)
2
where y
- (n((11-)/r) For any x.
( #1 TT(1 e/llū(12) 9/2)
||a||2
(ii) Since bound || is subject to the positions of two Gaussians, we scale the bound by the distance from Gaussian centers the classifier, Ja" 2(1-c/1 Alla) - and obtain
[[ e/ 1alla
| 1|12 = 1 mutteyT e/ l21l 9/2l
[ e/1l
[1a(1 e/112|122]11
1 M(1 e/llA(l2) -//2
1x Hi #22 pT - //1l:/11122 9/21
j7"p((1 c/11A(12) 9/2)
17

---

Under review as a conference paper at ICLR 2023
(iii) For sample Pa pa..pa.la consider the Bayes optimal robust classifier in equation we can calculate the analytical standard accuracy by
Fly = -
=TP -=
=rP (z
Jn -
2
+(1- (1--)PPp (o M1 + M2 - -
2
=TP [(x + w) ii(1 - -
+(1 P)P [(- -11 - 9/2 < 0] w 2~ ((,
=rP [1
A11 - l/ > 9/2 i/ l
+(1 P)P A1
x M11 c/llA|la) ~~0((0,1))
"" n² - e2
9/2-17 -0 - 701-011lllllll -
=TP
[1 </\12i122111
17 //1121222
wx= ji(1 - 12)
9/2 + 1²" 7i(1 -e/l(All))
w - e/ (1ll))
+(1 P)P
| - 1 ~ N0,11
||a(1 2c
1/t(1 c/ c/(2/12)]1
Φ6 1- 711 e/11/|12) 8/2
1z j01 e/ 11
+ g/2
+ (1 - T)
[ </11u|12)ll2
[ </11i|12)112
Let w denote 7(1 - e/ l/11a the we got the accuracy
1lT 9/2
+ (1 -1)D a w + 9/2
-
|00 12
||w||2
(iv) For sample x P Wa let t denote Mu and denote ãa(1 e/IlØ|l2). According to
Theorem dii, when ji" [a(11111- (1l-=//l/lll))- 9/2 /2 0, the accuracy would be higher than 0.5. Therefore we consider / w 4/2 > 0.
Now consider the classifier in equation 5 and the corresponding scaled bound from (ii).
[x a1tt2 ST ai1 e/11Ai112 - 9/2l
11" w 9/2l
1 w 9/2]
|| 2
p(1 e/ (1 9/2)
11TT w 9/2)
n 9/2
Since tly N N Ia), we have thw g/2ly ~ N(yū w 9/2, w ww). When we only want to get the expected scaled bound of the correctly-classife samples, we have that
|| f((x) = y
1
E ||t 9/21 f((x) =
ji'w (Tw - 9/2
E 11ET w 9/21 f((x) = =
pTT 9/2
1
+
E [1e w 9/2) fx(x) =y- -1]
T w 9/2 T
E 1 w 1/2ly=11 1, q/2200
iF' w - 9/2
1 1-7
E
[ 9
11 2 <0
pT w 9/2
18

---

Under review as a conference paper at ICLR 2023
Recall that t 9/2/y ~ Nyyp" 9/2, w² w'w), then by the mean of truncated normal distribu- tion, it true that
db 0-
E
B w 9/2 1, 902 2/220 p² 9/2 + ||
1 T
0- A w
10l1
6 12
/= w - 9/2 + ||
||
4 1 10-90-- 9/2
|| ll
E [t- 2 w + 2/2//122 -1, 9/2 < 0
E 1 w q/2 y = 1,
0+ m
- 1tt w 9/2
||l||2
| 1l
cb
0+ w0t//
12
o( a= w-ta/2
p w + +g/2+ ||
|| lla
n uo 9/2
1| 1|2
Therefore
17" w/t//22
|| f((x)) = v]
7
ji'u - 002- ||l||2
iTT w 9/2
4
2 u //2
1
1 --T
1 w + 9/2 + 1|20112
||
17 w 9/2
n0
9/2
v0 |/
19

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL TABLES
Model
0.2
0.4
0.6
CIFARIO
CIFART0-c
ImageNet
ImugeNet-c
VIT-B/16
0.33
0.37
0.32
0.20
0.06
95.0
81.2
83.8
66.4
ViT-B/1G-in2Ik
0.23
0.18
0.07
0.01
89.6
71.4
82.6
VrT-
81.2
83.8
0.75
ViT-B/1G-in2Ik Ik
0.12
0.16
0.10
0.02
89.6
71.4
82.6
63.6
VTT-B716
0.19
0.23
0.17
0.04
95.0
81.2
83.8
66.4
0.8
ViT-B/16- -n1B
VTT-B/16
0.10
0.15
0.09
95.0
81.2
83.8
66.4
ViT-B/16-in21k
0.01
0.02
0
89.6
71.4
82.6
63.6
ViT-B/I6
0.04
0.01
81.2
83.8
06.4
0.9
ViT-B/16-in2Ikk
0
89.6
71.4
82.6
63.6
Table Full table of Table 2
02
Model
0.4
CIFAR10
CIFARIO-c
ImageNet
ImugeNet-c
ViT-B/16
0.33
0.37
0.32
0.20
0.06
95.0
81.2
83.8
66.4
VTT-B/32
0
0
0
0
0
92.2
76.6
80.5
61.4
VIT-B7166
0.26
0.30
0.25
0.11
0.01
95.0
83.8
66.4
0.75
VIT-B/32
92.2
76.6
80.5
61.4
VIT-B716
0.19
0.23
0.17
0.04
95.0
81.2
83.8
66.4
0.8
VIT-B/32
0
0
92.2
76.6
80.5
61.4
66.4
0.85
VIT-B/32
0
92.2
76.6
80.5
61.4
VTT-B/16
0.02
0.04
0.01
0
0
95.0
81.2
83.8
66.4
0.9
VIT-B/32
92.2
80.5
61.4
Table 8: Full table of Table 5
20

---

Under review as a conference paper at ICLR 2023
0.4-mobus linear probing
CIFARIO-C
76 76.96-10.0
92.43
CIFARIO
0.3-robus linear probing
CIFARID CIFARTO-
81.871143
71.43
77.14:0.18
95 35.01110.05
89 22-20.16
92.41:0.077
P means the numbers are similar.
98
0.2 no2-s linear probing
CIFARIO
58. 38.2110041
81 $1.99100118
90.66-0-226
71.55=00.29
76.9210227
CIFARIO
76. 06.1110.13
05.09100.00
89. 13-20.14
14
92.44
9K.
0.1 8.1-mbas linear probing
CIFARIC-
59.05 22
$1.55110.33
90.3
71.51=0.43
76.92:1030
Table 9: Full table of Table 6
CIFARTO
19.00:10.13
95. 13.17000044
18.20+-0.12
89.25 29.25-00.03
92.4810.04
CIFATION 10-2
58.22...3 30
$0.6510..15
90-28--114
70.84=0.37 0.37
75.75100.3
Dinear probing
CIFARIO
80.473017
94.47=0.05
97.97=0.03
88.92=0.36 36
92.02.0.155
arg tnxx, SynBench-Score
0.1/0.2
0.2/0.3
0.1
0.2
VYT-B/16-in2IK
VTT-1916
VIT-1/16
VIF-L16
VIT-B/32
21

---

Under review as a conference paper at ICLR 2023
F SYNTHETIC DATASET COMPLEXITY
SynBench can adjust the synthetic task complexity by statistically modeling the structure the co- variance matrix Py. In our previous experiments, have considered an identity covariance matrix, here we assume channel-wise band matrix covariance Σ (R,G.B channel entries are externally in- dependent, and internally Gaussians with band matrix covariance). Essentially, the assembled Σ is block-diagonal matrix with each block being a band matrix in the size of the image. For ViT with ImageNet size imputs, the blocks are 224 x 224 We let the band matrices be Pentadiagonal matrices (only main, first two upper, and two lower diagonals are nonzero).
From Table to Table we see that the trend suggested by SynBench generally consistent with our findings in Section Moreover, SynBench can well-capture the complexity of the task and give lower SynBench-scores when the task is more complex.
0.7
c 0
0.2
0.4
0.6
0.8
ViT-B/16
0.18
0.24
0.20
0.10
0.01
VIT-B/16-in2Ik
0.07
0.11
0.07
0.01
0
Table 10: SynBench- Score comparisons on the finetuning procedure in pretraining on synthetic data with Pentadiagonal covariance.
O. = 0.7
< 0
==
0.2
6 0.4
==
0.6
< 0.8
VIT-TV16
0
0
VIT-B/16
0.18
0.24
0.20
0.10
0.01
ViT-L/16
0.23
0.12
Table 11: SynBench-Score comparisons on the model sizes on synthetic data with Pentadiagonal covariance.
R,
0.2
0.8
VIT-S/I6-DINO
0.47
0.46
0.39
0.23
0.03
VTT-B/IG-DING
0.42
0.52
0.51
0.45
0.35
VIT-S/8-DING
0.36
0.38
0.36
0.30
0.20
DIT-B/8-DING
0.42
0.55
0.50
0.40
0.28
Res50-SimCLLy2
0.24
0.47
0.36
0.33
0.31
Res101-SimCIRR
0.30
0.32
0.30
0.29
Table 12: SynBench-Scoree of self-supervised pretrained representations on synthetic data with Pentadiagonal covariance.
0.4
0.6
0.8
ViT-B/16
0.18
0.24
0.20
0.10
0.01
ViT-B/32
0
0
0
0
Table SynBench-Scomd comparisons on the VIT patch size of ViTs on synthetic data with Penta- diagonal covariance.
G OTHER BASELINES
For completeness, report several baseline metrics for the synthetic conditional Gaussian classifi- cation task. We follow the implementation of Whitney et al. (2020 [ and set n.s (the training set size) be the length the synthetic dataset compute canonical results. is. In Table T5] We report valida- tion loss (val loss), minimum description length (MDL) (Voita & Titov 2020), surplus description length (SDL) and e-sample complexity (c-SC) (Whitney et al.) 2020). As reference, obtaining the metrics for VIT-B/16 costs 6807 seconds and VIT-1/16 costs 7373 seconds on one Tesla V100.
https://github. www.:/hhhhtteeeeereeeeeeeee
22

---

Under review as a conference paper at ICLR 2023
Name
VIT-B/16
ViT-B/16-in2Ik
2048
Val loss
3.10
3.37
MDL
6820.76
7114.12
SDL, s=l
4977.76
5271.12
SCC, c=1
1843.0
1843.0
SynBench
0.33
0.20
4096
Val loss MDL
1.77
1.41
10813.95
9412.53
SDL, c=1
7127.95
5726.53
eSC, s=1
3686.0
> 3686.0
SynBench
0.45
0.30
8192
Val loss MDL
0.73
0.77
9939.13
9773.16
SDL, s=1
3479.59
3153.33
eC, s=1
7372
7372
SynBench
0.52
0.38
16384
Val
0.85
0.86
MDL
20936.18
20899.58
SDL, c=1
7266.8
7136.29
SCC s=1
14745
14745
SynBench
0.56
0.41
32768
Val loss
0.68
0.70
MDL
30848.99
32944476
SDL, 7043.32
8611.49
=SC,
14265
14265
SynBench
0.59
0.44
Table 14: Baseline metries evaluating the representation quality on the conditional Gaussian syn- thetic data with [2048, 4096, 8192, 16384, 32768). For Val loss, MDL, SDL, and eSC, the smaller the better; SynBench, the bigger the better. Note that model ranking of SynBench consistent across different values of while other methods will change their rankings.
23

---

Under feview as a conference paper at ICLR 2023
Resnet 101 1--SimCLike Lkkv2
0.52
5443 133443
776.38
669
Reanet50 S--mimCC SimCLRve LR22
0.62
9646 09
37000.73
4045
Table 15: Baseline metrics evaluating the representation quality on the conditional Gaussian synthetic data with 7 = 8192.
VIT-B/8 VIT-B/8-DING DINO
0.64
6796.87
1185.31
2220
VIT-B/16-DINO
0.92
10535. 11
3432.28
7372
VIT-S/8-1 FEIS/--DINO DINO
0.70
8196.8
2056.69
404S
VIT-S/16- DIN6-IINO DINO
1:31
18536.93
11164.93 >>
7372.0 >>
VIT-W32
2.92
23332.98
15960.98
7372.0
WT--B/16-in WT-BB/66in221 ik2
0.77
9773.16
3153.33
7372
VIT-1/16
1.50
17672.6
10300.6
7372.0
V1T-3/16
0.73
9939.13
3479.59
7372
V1T-19/16
4.38
30071 64
22699.64
7372.0
>
SDL., s=1
Vallos loss
Name
MDL
eSC,
24


Reviews:
[{"review_id": "ntydbJixGC5", "reviewer": "Reviewer_REYd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at finding a *task-agnostic* and *model-agnostic* way of evaluating the quality of any pretrained model's extracted features. Specifically, authors develop a score that measures the accuracy-robustness tradeoff of a model's representation on a synthetic binary classification task. Authors asses the relevance of such score by studying how well it correlates with ViT model's ability to obtain a good in-distribution (CIFAR) / out-of-distribution (CIFAR-10C) accuracy tradeoff.", "summary_of_the_review": "The current manuscript does not appear ready for publication due to (i) lacking motivation on the important directions of the work, (ii) lacking experiments/evidence to properly support author's claims and (iii) lacking clarity and conciness in writing. Therefore, I cannot recommend acceptance at this stage.", "strength_and_weaknesses": "### Strengths\n\n**Interesting problem.** The problem of evaluating the quality of a pretrained model's representations in a *task-agnostic* fashion appears as an ambitious (perhaps a bit ill-posed?)/important problem in a context where pretrained/foundations models are becoming de-facto starting points for most existing specific applications.\n\n**Exhaustive related work section.** The related works part covers relevant lines of works, including most recent works on pretrained models benchmarking.\n\n### Weaknesses\n\n**Motivation.**\n\n*Use of synthetic data.* I'm having hard time wrapping my head around how the model's behavior on images synthetized from a Gaussian mixture (i.e probably very far away in the input space from both the source and target distributions) could inform the model's behavior on real-world images. Could authors provide additional motivation ?\n\n  *Adversarial perspective: * (See below for a more detailed comment) I'd suggest authors provide more motivation on why they chose to address the general problem of \"representation quality\" through the lens of adversarial robustness, as the link between adversarial robustness and generalization is not trivial (although I concede it is not unreasonable either).\n\n**Lack of comparisons.** Authors do not really compare to any other line of work. Although I could concede that other works have never tried to address this exact setting (i.e completly task-agnostic evaluation of representations), authors do not seem to have put any effort into adapting any technique to their setting, which makes the relevance of their score (on top of my next comment) hard to evaluate. For instance, cited lines of work that use unsupervised criterions (e.g. mutual information or Minimum Description Length) could be adaptated to the synthetic binary classification task that authors propose (e.g. comparing the mutual information between representations and labels in the input space vs in the representation space), and provide at least some point of comparison.\n\n**Hardly falsifiable hypothesis.** The contribution section claims that the score evaluates the quality of the representations. On the other hand, authors do not find any strong positive correlation between their score and actual tradeoff performances of models in Table 3. (e.g. standard probing for ViT-L, corresponding to $\\epsilon=0$, has smaller score but significantly higher OOD generalization). Authors comment on that part that \"although SyncBench-score may share trends with empricial real-life tasks, it is meant to characterize a general behavior of the pretrained representations.\" This feel a bit hand-wavy to me, in the sense that the definition of \"general behavior\" can mean everything and anything. I believe the paper would benefit from clearly defining what the score is meant to act as a proxy of, and explictly show positive correlation between their score and this metric. \n\n**Lack of experiments.** Echoing to my previous point, even if strongly positive correlation had actually been found, more experiments would be needed to support the initial goal/claims of producing a score that *applies to a wide range of pretrained models* and informs downstream accuracy. In particular, more architectures should be tried (at least 1 ConvNet), more datasets (CIFAR is a relatively outdated choice considering the myriad of more recent vision datasets), and at least one additional task (e.g. segmentation/detection ? or even anything in another modality?) to support the *task-agnosticity* part.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kVBsiwZ17BA", "reviewer": "Reviewer_xsCU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel way to evaluate the quality of a pre-trained representation. It requires no real data and downstream tasks for the evaluation. In the evaluation, it utilizes synthetic data generated from the mixture of two Gaussian distributions. The authors prove the soundness of the method using the assumption of Gaussian.\nThe synthetic data is used to evaluate the trade-off between accuracy and robustness.\nThe evaluation is done on ViT architecture with different sizes. The experiment shows the SynBench score can be used to predict the model performance on downstream tasks.\n ", "summary_of_the_review": "The paper presents an interesting idea to evaluate pre-trained representation. The idea is conducted with Gaussian synthetic data. It is unclear to me whether this way of evaluation can really transfer to a meaningful metric for downstream tasks. The experiment in the paper is somewhat limited to support the claim of the authors. If authors can apply this evaluation strategy to more model architectures and show the same discovery, it will be very impressive. I think its current status is below the threshold of ICLR.", "strength_and_weaknesses": "Strength: \n- The idea of evaluating a presentation without using a real dataset and downstream tasks is interesting. The idea is novel and the theory seems to be well proved under the Gaussian assumption.\n\nWeakness: \n- The theory's assumption is too strong and unclear whether the result can be transferred to more complex real-world scenarios.\n- The experiment can be an effective way to validate whether the assumption holds true for more challenging scenarios. The experiment done in the paper is insufficient to get a conclusion. The comparison is done using variants of ViT and does not include other architectures. Since the evaluation method is a universal approach, it is important to see whether it is effective for other models.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YWpWM_SFO_x", "reviewer": "Reviewer_Lj97", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a task-agnostic framework to measure the quality of pretrained representations.  This approach uses generated data from a conditional Gaussian mixture to evaluate the learned representation.  The usefulness of this metric is evaluated by comparing to results from linear probing on CIFAR 10/10-c", "summary_of_the_review": "Creation of a task-agnostic measure of a learned pretrained representation has significant value.  \n\nHowever, by evaluating this metric only on a single task and dataset (classification for CIFAR 10/10-c), the impact of the paper is strongly limited.  How this would hold on other tasks (e.g. segmentation, surface normals, etc.) as well as larger, real world datasets where such approaches are needed, is not explored.  The assertion is that this metric is more useful than linear probing because it can be examined at the time of pretraining (not downstream task formulation) to anticipate how well that representation will perform across a number of tasks.  With that logic, this metric should enable the user to select the \"best\" representation from a set of representations which have been learned.  This needs to be demonstrated over a wide range of pretraining datasets, pretraining methodologies, downstream tasks, donwstream datasets, etc.", "strength_and_weaknesses": "Strengths:\nDetermining a robust and straightforward measure of the quality of learned representations (whether pretrained or otherwise) is an important endeavor.  Current approaches use linear-probing, but this is naturally task specific.  Having a task-agnostic measure of representation quality would be valuable.  The author's state the main tradeoff is between accuracy and adversarial robustness.\n\nWeakness:\nIt's not clear whether a task-agnostic representation is meaningful or valuable.  Is the best representation the one that transfers to the most tasks, transfers the best to a specific task, is most extensible to an expanse of the domain, etc?  The authors have clearly stated what they deem as \"good\", but it's not clear that this is a general definition of a \"good\" representation.   \n\nThe theoretical results derived all assume a balanced dataset, which is uncommon in real wold data.\n\nThe evaluation is only done on image classification (CIFAR 10/10-c).  While it is true that this dataset is used, it doesn't seem to be a good test dataset alone for this.  Vision transformers and large scale pretraining are most beneficial on massive datasets (both in the number of samples and when the images themselves are larger than CIFAR-10).  Therefore it's unclear how these results would hold on real world scenarios where such approaches would be actually used.  It's fine to start with CIFAR-10, but demonstration on a larger dataset (even Imagenet) and especially on a dataset where such approaches are more appropriate (e.g. medical imaging) would be much more powerful.\n\nTo demonstrate the usefulness of this approach, I believe the authors would need to more systematically demonstrate how the scores obtained from different pretraining procedure (e.g. MOCO, DINO, SimCLR, etc.), pretraining dataset (e.g. Imagenet, COCO, something entirely different like medical or remote sensing), downstream task (e.g. classification, segmentation, surface normals), and downstream dataset change and how that corresponds to the final downstream task performance (both fine tuning and linear probing).  The authors are implying that this metric is more general and therefore more useful than something like linear probing, but they have not demonstrated it over a wide enough range of representations and tasks, in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject