Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts
Authors: Jonathan Crabbé
Pau Rodriguez
Vaishaal Shankar
Luca Zappella
Arno Blaas
Venue: Accepted by TMLR ()

Abstract: What distinguishes robust models from non-robust ones? While for ImageNet distribution shifts it has been shown that such differences in robustness can be traced back predominantly to differences in training data, so far it is not known what that translates to in terms of what the model has learned. In this work, we bridge this gap by probing the representation spaces of 16 robust zero-shot CLIP vision encoders with various backbones (ResNets and ViTs) and pretraining sets (OpenAI, LAION-400M, LAION-2B, YFCC15M, CC12M and DataComp), and comparing them to the representation spaces of less robust models with identical backbones, but different (pre)training sets or objectives (CLIP pretraining on ImageNet-Captions, and supervised training or finetuning on ImageNet). Through this analysis, we generate three novel insights. Firstly, we detect the presence of outlier features in robust zero-shot CLIP vision encoders, which to the best of our knowledge is the first time these are observed in non-language and non-transformer models. Secondly, we find the existence of outlier features to be an indication of ImageNet shift robustness in models, since we only find them in robust models in our analysis. Lastly, we also investigate the number of unique encoded concepts in the representation space and find zero-shot CLIP models to encode a higher number of unique concepts in their representation space. However, we do not find this to be an indicator of ImageNet shift robustness and hypothesize that it is rather related to the language supervision.

Published in Transactions on Machine Learning Research (10/2024)
Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts
Jonathan Crabbé
1mathhnncr13000@pppp com
University of Cambridge (work done while Apple)
Pau Rodríguez Apple
pwn.rodrigucce@pppe..cm
Vaishaal Shankar Apple
vaishaal shankar@@pple.coooo
Luca Zappella
taapella@apple.com com
Apple
Arno Blaas
ablaas com
Apple
Reviewed OpenReview: https: /tttp:://pperrevieet openreview. net/
net/forun? forum? id= 1S0p tFFtmV
Abstract
What distinguishes robust models from non-robust ones? While for ImageNet distribution shifts it has been shown that such differences in robustness can be traced back predomi- nantly to differences in training data, so far it is not known what that translates to in terms of what the model has learned. In this work, we bridge this gap by probing the represen- tation spaces of 16 robust zcro-shot CLIP vision encoders with various backboncs (ResNets and ViTs) and pretraining sets (OpenAI, LAION-40MM, LAION-2B, YFCC15M, CC12M and DataComp), and comparing them the representation spaces of less robust models with identical backbones, but different (pre)training sets objectives (CLIP pretraining on ImageNet-Captions. and supervised training or finetuning
ImageNet). Through this analysis, generale three novel insights. Firstly, we detect the presence of outlier features in robust zero-shot CLIP vision encoders, which to the best of our knowledge is the first time these are observed non-language and non-transformor models. Secondly, we find the existence of outlier features to be an indication ImageNet shift. robustness in models, since we only find them in robust models in our analysis. Lastly, we also investigate the number of unique encoded concepts in the representation space and find zero-shot CLIP models to encode higher number of unique concepts in their representation space. However, we do not find this to be indicator of ImageNet shift robustness and hypothesize that it is rather related to the language supervision.

---

Published in Transactions on Machine Learning Research (10/2024)
1 Introduction
Large pretrained multimodal models, such as CLIP (Radford et al., 2021), have demonstrated unprocedented robustness to distribution shifts around ImageNet'. When used as zero-shot image classifiers, their perfor- mance on ImageNet (Deng et al., 2009) translates remarkably well to performances on natural shifts of ImageNet, such as, e.g., ImageNet-V2 (Recht et al., 2019). This led many works analyzing what actually causes this remarkable robustness of CLIP to shifts around ImageNel, with Fang et al. (2022) establishing that the root cause of CLIP's robustness lies in the quality and diversity of data it was pretrained on. In particular, they find the robustness of CLIP to shifts around ImageNet to disappear when it is pretrained on ImageNet-Captione a modification of ImageNet suitable for unsupervised language-vision pretraining.
While the cause of CLIP's robustness to ImageNet shifts is thus known, we set out to establish how exactly robustness manifests itself in features learned by the model. Finding feature patterns in the representation space that only appear in robust models is the first step towards a better understanding of the emergence of robustness. It is also key diagnosing robustness in times when only limited knowledge about the (pre)training distribution or the shifts is available. To find robustness patterns in robust CLIP models, we leverage the various models provided by Ilharco et al. (2021) in the OpenCLIP repository, non-robust CLIP models pretrained on ImageNet-Caations provided by Fang et al. (2022). and models we train in a supervised way on ImageNet from scratch. We analyze the visual features in these models by probing their last layer activation vectors with quantitative interpretability tools, such as kurtosis analysis of activation vectors (Elhage el al., 2023), singular value decomposition (SVD) of classifier weight matrices and concept probing of the representation space (Bau et al., 2017). Through this analysis, we distill insights distinctive characteristics of CLIP model features and CLIP ImageNet distribution shift robustness.
well as
Our contributions. (1) We show that many robust CLIP models have outlier features. These features stand out their activation is typically several orders of magnitude above the average activation of features in the same layer. Interestingly, this observation also holds for robust CLIP models with ResNet backbones. To the best of our knowledge, this is the first time that outlier features are observed in non-language and non-transformer models (Dettmers et al., 2022; Elhage et al., 2023; Sun et al., 2024). We also show through an SVD analysis that outlier features are propagated to the logits of downstream classifiers, which results in what we call privileged directions that are crucial to model predictions. (2) Through comparative analysis we find that outlier features are robustness indicators that distinguish robust models from their non-robust counterparts. Since none of the non-robust models exhibit them, we find outlier features to be an indicator whose presence can indicate that a model will be robust to distribution shifts around ImageNet. Privileged directions on the other hand, are not such an indicator, since they can also be found in some of the non-robust models. (3) We show that robust zero-shot CLIP models all encode a high number of unique concepts in their features. As a consequence, the features of robust models are highly polysemantic, which means that they superpose a large set of concepts. Surprisingly, we find through our comparative analysis that representations that are rich in concepts not necessarily more robust, as this property can also be found in non-robust Captions raited CL.IP models. This suggests that language supervision tends to enrich visual representations in human concepts.
2 Background on CLIP models and notation
In this section, we summarize the CLIP paradigm introduced by Radford et al. (2021) and introduce the notation used in later sections. We explain how CLIP models are trained and used for image classification.
CLIP architecture. A CLIP model consists in an image encoder fo Rdx + Rds and a text encoder Rd + d mapping to the same representation space dda of dimensions du E N. With our notations, dx C. W for images with C E N channels, height H E N and width W E N. Similarly, dr = L. V for lexts with context length E N and vocabulary size V E N. Given a batch of (image, text) pairs
Similar Radford et al. (2021), use CLIP as a name for the general training technique of unsupervised langnage-vision pretraining, not only for the specific models obtained by OpenAT.
2

---

Published in Transactions on Machine Learning Research (10/2024)
B [1x00 _x(") E IRdx xR | [B]}, where N denotes the batch size and [B] = {n
E S B).
the encoders are trained to minimize the symmetric contrastive loss
fu. LB fur fe) + LE (BB, fi)
(b)
exp
T 008
fo 1o
St
It
C,
--
log
b=1
b==
B
exp
r- cos
fo (1(0 )
ft
B
exp
7--10s So (I((0 St.
(b)
L. ==
Σ b
log
B >b== exp
7 cos
So
(b)
Se
(b')
with cos denoting the cosine similarity. Ly and L are InfoNCE losses from Oord et al. (2018) with learnable temperature parameter T E Rt. These losses induce the encoders to align the image and text embedding pairs in the representation space dd through the nominator, while separating image and text embeddings of distinct samples through the denominator, which differs between Co and Lt.
Building zero-shot classifiers. Once the model is trained, we have access to an image and text encoder that tend to align images with their text description. This can be used to build zero-shot image classifier that discriminates between set of K E N classes & [K]. The typical approach is to combine the name of the class k together with a template, to create a! For instance, the class lion can be combined with the template an image of <class> yicld the text dcscription an image input image Zy Rdxx, one can assign logits to each class & [K] as follows:
lion. To assign a class
logit (x = cos (xo)) x (
-[w [m(()))))
where || denotes the euclidean norm in Rda and the matrix W E Rxdd has elements
r-1 fe
Wkj
11
llfe
for each & E [K] and [dæ]. A zero-shot image classifier can thus be built from CLIP as the composition between the linear classification head W and the CLIP image encoder (with normalization). For case of notation, we use f = fv, x = etc. in the remainder of this work, unless otherwise specified.
3 Robustness of CLIP models
In this work, we focus on the robustness of models to shifts from ImageNet (Deng et al., 2009) to the five natural distribution shifts considered by Fang et al. (2022), namely ImageNet-V2 (Recht et al., 2019), ImageNet-R (Hendrycks et al., 2021a), ImageNet-Sketch (Wang et al., 2019), ObjectNet (Barbu el. al., 2019) and ImageNet-A (Hendrycks al., 2021b).
Measuring robustness. There different ways to measure robustness to specific distribution shift from source distribution A to shifted distribution B. In some works, especially in the literature invariant and robust learning, simply performance on both source and shifted distribution are reported (Arjovsky et al., 2019; Makar et al., 2022; Jiang & Veitch, 2022; Kumar el. al., 2022), and their quotient can be straightforwardly computed to report robustness in a single metric ( (What fraction of the original performance is maintained on the shifteed d.ta?). On the other hand, in the literature investigating the robustness of CLIP to ImageNet shifts, typically slightly more involved metric called Effective Robustness (ER) is reported (Taori et al., 2020; Fang et al., 2022). We choose to include both, and will find that similar insights about
3

---

Published in Transactions on Machine Learning Research (10/2024)
the robustness of models can be derived from both of them when it comes to distribution shifts around ImageNet.
Lastly, we note that recently Shi et al. (2023) have investigated the robustness of models to the five natural distribution shifts of ImageNet, using additionally a different data distribution than ImageNet (YFCC-15M Thomee al. (2016)) as the source distribution. In that case; none of the models investigated (including CLIP models) is significantly more robust than the other models. Therefore, we keep the focus of our analysis the shifts from ImageNet to its five natural distribution shifts, where most zero-shot CLIP models are without doubt significantly more robust than their finetuned supervised counterparts, we will recap in the following.
Model pool. We run our analyses across five backbone architectures: ResNet50, ResNet101, ViT-B-16, ViT-B-32, ViT-L-14 (He et al., 2015; Dosovitskiy et al., 2020). For each architecture, the OpenCLIP repository (Ilharco et al., 2021) contains robust pretrained CLIP models on various pretraining datasets: the original (unreleased) OpenAT pretraining set (OpenAT, Radford et al. (2021)): YFCC-15M (Thomee et al., 2016; Radford et al., 2021), CC-12M (Changpinyo et al., 2021), LAIÖN-400M, LAION-2B (Schuhmann et al., 2022); and DataComp (Cherti et al., 2023). Furthermore, Fang el. al. (2022) provided us with checkpoints of the non-robust CLIP models pretrained on ImageNet-Captii Captionss for three different versions of ImageNet-Captions: a first version for which only the original titles from the internet associated to the images are used as text (anageNee-Captionn--)) a second version for which a concatenation of original titles and description was used (ImageNet- of original titles, description, and tags was used SimageNet-Captionn---e et al. (2022)). We load the pretrained vision encoders of all available combinations of architecture and pretraining dataset, and construct a zero-shot classification model for ImageNet using the methodology described in Section 2. By finetuning each robust zero-shot model on ImageNet, we obtain classifiers with lower robustness than their robust zero-shot counterparts (Andreassen et al., 2021; Kumar et al., 2022; Wortsman et al., 2022a). Lastly, since the non-robust SrageNee-CCaptinnn a ResNet55 backbone, we obtaín further non-robust models for the remaining architectures by training them classification models on ImageNet from scratch, and obtain models with far lower robustness than the finetuned ones (Fang et al., 2022) (details on model finetuning and training can be found in Appendix H).
and a last version for which a concatenation
for more details see Fang
models were only trained for
Results. For all models, we compute the test accuracy on ImageNet, on the five shifted datasets (averaged), their quotient, as well as the ER (for details on the computation of ER, see Appendix A). We report the two metrics of robustness (quotient of test accuracies %ace, and ER) in Figure la and Figure (for the individual test accuracies that these metrics are calculated from, see Appendix B). We see that according to both metrics, for each type of backbone the zero-shot CLIP models from OpenCTIP have the highest robustness ('Robust zero-shot CLIP"), which decreases but remains significant when fine-tuned ImageNet ('Fine-tuned CLIP"). On the other hand, ImaageNet-Captions CLIP models (NNon-robust zero-shot CLIP') and ImageNet trained supervised models ("Supervised'² do not exhibit any ER and much lower robustness according to %acer typically loosing more than half of their ImageNet accuracy on the shifted datasets (%ace dropping below 50%).
In agrecment with the literature, we find that for each architecture, zero-shot CLIP models that were pretrained on OpenAl, YFCC-15M, CC-12M, LAION-400M, LAION-2B, or Data- Comp, are the most robust to ImageNet distribution shifts. This robustness, while decreased, remains significant after fine-tuning on ImageNet for almost all pretraining datasets. On the other hand, zero- shot CLIP models pretrained on MmageNet-Captionss and supervised models trained on ImageNet, are non-robust to ImageNet distribution shifts.
Take-away 1.
Foor the ViT-I=11, we were unable train a supervised version to convergence from scratch on ImageNet (Dnsovitskly et (2020) and He et al. (2022) comment the difficulties of training such an overparametrized model on TmageNet).
4

---

Published in Transactions on Machine Learning Research (10/2024)
ahot Fine-tured
Supervised
Non-robust zero shot
40%6
ResNet50
ResNet101
VIT-8-16
VIT-B-32
VITL-14
30%
Ellecive Robuetnes
20%
10%
NA
Opanni
YFCC- 19M
CC-10 12M
Imageh Mww.ee----llann Caglione-
Imagehhes Cagtitnnett Rd
ImageNos Cap.ioneinneei Idht
Imagehett
Oipenjii
IFCC-11M
ImageNet
Opanáu
LAION-ADDE 40011
LAION-21
OataCoamp
Imagebeer
Opannji
LAION- LAINN-AOORA
LAION-20 20
Imagebler
Oppnn
LAION-on LAION-R00R -ogo
LAON-20
DataCome
Imagetter
Training Detaset
(a) FRR calculated according to Equation (5) in Appendix A.
Robust zero shot CLIP Fine-tuned CLIP
Supervised
Non-robust zaro-shot CLIP
100%
VIT-B-32
VIT+-14
75%
maintained
50%
25%
NA
Opanal
XPCC-
CC 12tt
Imagee Capptonne 1
Imaget Capttoree
Imagehone Copttooro
ImageNott
VPCc 10M
Imagehet
Ohannii
LAOO 400M
LAOON-20
Dat
Imagehiet
Openti
LAIO -000
LAOON-2 20
Imagueher
Opennii
LAOO 4ooM
LAJON-28
DataCrema
Imageeer
Training Dataset
(b) %ace calculated as the ratio between the average accuracy on the five ImageNet shifts the ImageNet test accuracy.
Figure Robustness metries for the models in our pool. Higher values indicate higher robustness. We see that for each backbone and pretraining data, robustness decreases from Robust zero-shot CLIP Fine-tuned CLIP and reaches mēnimumm for ImageNel supervised ('Supervised") and Imogenet-Caplion trained ("Non- robust zero-shot CLIP") models.
4 Outlier features and privileged directions
In this section, we explain how we detect outlier features in zero-shol CLIP vision classifiers, and how we find them to be an indication of model robustness. We start by explaining what outlier features are and how they are surfaced, and then proceeed to analyse how they propagate class logits in the form of privilegod directions. We analyze both outlier features and privileged directions for each of the models in our model pool.

---

Published in Transactions on Machine Learning Research (10/2024)
Approach. We aim to analyze what models with high robustness have learned in comparison to models with lower robustness. To this end, we compare the features, i.e. the representation space spanned by the image encoder fo described in Section 2. Like Goh et al. (2021), we focus on the output of the encoder only since it is used for downstream classification by a lincar head computing the ImageNet class logits. Furthermore, a central kernel aligument (CKA) analysis in Appendix C reveals that. robust models diller from less robust models most consistently in this last layer, making it the most relevant layer to focus on from comparative standpoint as well. We use the ImageNet test set to produce activation vectors h(m)= fo(z(") E Rd for each image z(1) E Ha fed to the encoder.
Preliminary observations. Qualitatively observing the distribution of activation vectors, one thing that immediately stands out is the fact that some components E [drr are much larger than the average activation: 1(t) > d Ee h (()) A similar phenomenon has recently been observed by Dettmers et al. (2022) in large language models (L.LMs). Such features, whose activation is substantially more important than average, were coined as outlier features. Subsequent work by Elhage et al. (2023) introduced a simple way to surface these outlier features, through metric called activalion kurlosis. We now use this criterion to quantitatively analyze the features of CLIP models.
Activation kurtosis. Following Elhage et al. (2023); we measure the activation kurtosis to quantitatively evaluate the presence of outlier features in a model. The activation kurtosis is computed over all the components of an activation vector h(m,, and averaged over N activation vectors:
da ((t)) po (H(ml))
Kurtosis = N
(1)
o (h(m))
where m(h) == d Enn h; and (h) == def
[he m(h))² As explained by Elhage et al. (2023), Kurtosis >> 3 indicates the presence of outlier features (3 being the kurtosis of an isotropic Gaussian).
dy =
We report the average kurtosis over the ImageNet test set in Figure 2a for cach architecture and across the various levels of robustness. This leads to three insights. Firstly, across all archilectures, there are robust zero-shot CLIP models with outlier features, as indicated by their Kurtosis >> 3. Secondly, the kurtosis, like the robustness, drops when finetuning on ImageNet. In Appendix I we include an additional analysis showing that Kurtosis closely tracks the ER metric when interpolating between robust zero-shot CLIP models and fine-tuned CLIP models in weight space. Lastly and most importantly, the values Kurtosis R 3 obtained for the non-robust supervised, and non-robust zero-shot CLIP models reveal the absence of outlier features in non-robust models, suggesting that the presence of outlier features is an indicator of robustness that can only be found in robust models.
Privileged directions in representation space. The strong presence of outlier features in the most robust models does not necessarily explain the performances of these models. Indeed, it is perfectly possible that outlier features are ignored by the linear head computing the class logits based on the activation vectors, e.g. if they are part of ker W, the null space of the weight matrix W defined in Section 2. Thus, to assess whether outlier features are of importance, we now introduce the notion of privileged directions of the representation space ddd as an instance of generalized form of outlier features. While outlier features are studied in the canonical basis fei..... Caw I, since we can write h; = Proj., (h), they can be generalized to be any set of directions of the representation space that receive a projection substantially above average (for more details and an illustration, see Appendix F).
We locus on the direct that are important for the compulation of logits by the linear head W, namely right singular vectors of W. These can be identified by performing a singular value decomposition (SVD) of W, which can be written as W = _rank(W) a, unoi where a, E IR+, u4 E Rdy and e; E Rdæ respectively correspond to singulor values (SV), left singular vectors (LSV) and right singular vectors (RSV) of W. In this decomposition, each RSV e; corresponds to a direction in representation space that is mapped to the logits encoded in the LSV Ug. Since both of these vectors are normalized ||u+]| ||>:|| 1, the importance of the direction Us for W is reflected by the SV a+- Note that the SV a, by itself does not refer to the model's
6

---

Published in Transactions on Machine Learning Research (10/2024)
Robust Fine-tuned
Supenweed
Nonrobuus zero-shot
RedNeed0
ReehMrr101
VT--8-16
VIT-B-322
WT--14
67.6
Adlivaton Kunoode
45
Opmayy
CC 10M
Inage Capllonaaaon Caplooee
Intgghol Castiche Lift
Ai
YCCC H0M4
Laca cota
Inozelaat
LAION 20
Datebrerp
Traning
Activation kurtosis calculated through Equation (1),
Superieed
ReoNat101
VIT-L-14
Largost Importamccc////oooo Inportance
YTCC 1932
CC
Imarg ImargphiennC Cupanare
Imnagghii Conpitone fut Imag Carpiorre tld
YFCE 1ots
Irteppoket
Libaa
LAcal-20 200
Dattd samp
Imagghhee
Openna
LAlca
LAual-22e 20e
Opendi
LAC
LACK--
DanD pand-ma
Imorgehies
Trahinng Dataset
(b) Direction importance ratio calculated through Equation (3).
Figure 2: Outlier features and privileged directions. Most robust zero-shot CLIP models have outlier features (high kurlosis) and privileged divections (high direction importance oullierness). Fine-tuned models have no outlier features but still exhibit privileged directions, although those are noticeably less privileged. Supervised models and non-robust zero-shot CLIP models have no outlier features. The full distribution of importance scores can be found in Appendix E.1
encoder activations. How measure if the direction is typically given an important activation by the encoder? We propose to measure the average cosine similarity of activation vectors (()) with this direction. This leads us to a unified importance metric for each direction E [rank(W)] of the representation space:
N cos(u: (1, (x0)
Gi E (aa)
Importance(1) =
(2)
N
Classificution head importance
Encoder importance
Note that this metric defined so that 0 < Importance(1) < 1. With this metric, we can measure what extent the presence of outlier features induces privileged directions in representation space. such privileged directions exist, we expect some singular directions v; to have Importance()) substantially
7

---

Published in Transactions on Machine Learning Research (10/2024)
higher than average, i.c. with Importance(t) > rank -ソ=1 Importanco((. We thus can identify privileged directions as the RSVs associated to outlier values in the importance scores. Indeed, we observe the existence of such privilege directions in Figure 2b, where we plot the quotient of the largest and the average importance score lor each model
max==
Fank(W)
Importance(i)
Importance Ratio
2j=1 (
Importance(/)rankk(V))
We notice that all the robust zero-shot CLIP models have such privileged direction. For the less robust finetuned models, these privileged directions still exist, but not they are not as strong. This indicates that finetuning de-emphasizes privileged directions. Finally, the importance distributions of non-robust models have no privileged directions with only one exception: the ImageNet-Captions CLIP models. These models have privileged directions in the absence of outlier features, which might appear surprising at first. By analyzing Equation (2); we deduce that these models should have some singular values that are substantially larger than average. As explained in Section 2, these singular values correspond to weight matrix obtained by stacking activations of the language encoder from the CLIP model. Since the non-robust SongeNel-Caplicne CLIP models show little to no robustness, we deduce that privileged directions can not serve as an indication of robustness.
Many robust models exhibit outlier features and privileged directions in their represen- tation spaces. Since we do not find outlier features in non-robust models, outlier features appear to be an indication of model robustness to shifts around ImageNet. Privileged directions can however also be found in non-robust models and are thus not an indication of robustness.
Take-away 2.
Emergence of outlier features. We have observed that outlier features are an indication of robustness ImageNet distribution shifts. We would like to suggest hypothesis to explain why they distinguish robust models from their less robust counterparts. In our analysis, they can only be found in robust zero-shot CLIP models that were typically trained on datasets with size and diversity that is several orders of magnitude above that of ImageNet. Similarly, Sun et al. (2024) observed 'massive activations' (closely related to outlier features) only in transformers which were pretrained on datasets that were large and diverse in comparison to ImageNet, but not transformers trained on ImageNet itself. Combined together, these two facts suggest that outlier features result from training on larger and more diverse datasets (in comparison to the evaluation dataset), and that nosustness and outlier features thus share a common root cause in the type of pretraining data.
Relationship of outlier features to pruning. Note that previous work on LLMs found that outlier features also have positive effects on model pruning: Sun et al. (2023) found that LLMs with outlier features can be efficiently pruned by retaining features with larger activations. We also found some weak evidence of this kind when pruning latent directions (see Appendix G).
5 Concept probing
In this section we find that robust. CLIP models encode more concepts than non-robust supervised models. However, also non-robust CLIP models encode similarly high numbers of concepts. Our analysis thus suggests that this stems from language supervision in CLIP, rather than being related to robustness. We first describe our approach, and then discuss the concepts encoded in the privileged directions identified in the previous section. We then show that robust and non-robust. CL.IP models encode more unique concepts than fine-tuned and supervised models. Lastly, we explain how this leads to polysemanticity in CLIP models.
Approach. With the discovery of privileged directions in the representation spaces of models with ro- bustness to Imagenet shifts, is legitimate ask what type of information these directions encode. More generally, are there differences in the way robust models encode human concepts? To answer these questions, we use concept probing. This approach was introduced by Bau et al. (2017), along with the Broden dataset.
8

---

Published in Transactions on Machine Learning Research (10/2024)
This dataset consists of 63, 305 images illustrating C' = 1, 197 concepts, including scenes (e.g. street), objects (e.g- flower), parts (e.g. headboard), textures (e.g. swirly), colors (e.g. pink) and materials (e.g. metal). Note that several concepts can be present in each image. For each concept E [C], we construct a set of positive imagos p C R (images that contain the concept) and negative images N C R (images that do not contain the concept). In the following, we shall consider balanced concept sets: |P = [ Concept probing consists in determining if activations a given direction of the representation space discriminate between pe and N=.
Assigning concepts to directions. We are interested in assinning concepts to each RSV v of the linear head matrix W. To determine whether representation space direction enables the identification of concept E [C], we proceed as follows. For each activation vector h(em) = f=(2*) associated to positive images g(0,)) E pe, we compute the projection Proj., (f(ca)) on the RSV. We perform the same computations for the projections Proj.. (h(-)))) of negative images (((6))) E N If the direction 2 discriminates between concept negatives and positives, we expect a scparation between these projections: Proj., (h(em)) * Proj.. (poji,(((T))) (N)))))) In other words, we expect the projections on v; to be good classifier to predict the presence of c. Following Suau et al. (2022), we measure the average precision AP; of this classifier to determine whether the concept is encoded in direction e4- We set a threshold AP; 2 0.9 to establish that the concept is encoded in v,3.
Interpreting privileged directions. We look at the concepts with highest AP in the privileged direction of each robust model represented in Figure 2b (i.e, robust and fine-tuned CLIP). In the following, we illustrate our insights about which concepts are encoded in the privileged direction on the OpenAl pretrained models. Note that the following discussion generalizes well to the other robust models, for which we report the top-3 concepts in Appendix E.3.
Interestingly, the most privileged direction of both zero-shot Open ViTs encode the same top-3 concepts: meshed, flecked and perforated. The most privileged direction of the ResNet50 also encodes concepts related to textures, with the knitted and chequered concepts. The most privileged direction of the ResNet101 encodes concepts with high colour contrasts, with the moon bounce, inflalable bounce game and ball pil concepts. We note that all these concepts describe regular alternating patterns, either through the presence absence of holes or through the variation of colours.
Let us now discuss the concepts encoded in the privileged directions of finetuned models. We find that finetuning replaces the above texture-related concepts by less abstract and more concrete concepts. After finetuning, the concept that are best encoded in the privileged directions are martial art gym for the ViT-B 16, tennis court for the ViT-B/32, mountain pass for the ResNet50 and flight of stairs for the ResNet101. All of these concepts are substantially less generic than the ones encoded in the zero-shot models.
Take-away 3. We qualitatively observe that privileged directions of robust zero-shot CLIP models tend to encode rather generic texture information, while fine-tuning tends to replace these generic concepts in privileged directions by less abstract and more concrete concepts.
Number of unique concepts. Let us now discuss the representation spaces of various models beyond privileged directions. A first way to characterize a representation space as a whole is to simply count the number of unique concepts they encode. In other words, for the representation space of each model, we evaluate
Nunique- == |C]
C: (cE [C] AP; 2 0.9 for some E [rank(W))]-
(4)
We report the number of unique concepts encoded in each type of model from our pool in Figure 3.
We notice that robust zero-shot CLIP models encode substantially more concepts than their fine-tuned and supervised counterparts, which thus at first might appear to be another indicator of robustness. However, also some of the non-robust zero-shot ImageNet-Caatioon pretrained CLIP models encode a relatively high number of unique concepts, namely mageNett-Cationss and Captiont- On the other hand,
3 All the below conclusions still hold the same way if we change the threshold other values such as 0.8, 0.85, or 0.95.
9

---

Published in Transactions on Machine Learning Research (10/2024)
Robust zero- shot CLIP
Fine-tuned CLP
Supervised
Non-robust zero- shot CLIP
800
ResNet50
ReeNet101
VIT-B-16
VIT-B-32
VITL-14
800
Concepts
400
200
NA
Openni
VPCC- 18M
CC 12M
ImageNestCCo Captionts
Imazgehls Capalone Rd
Imargehing Caglione Adl
Imargehien
Oipannu
YFCC 18h4
Imargehhet
Opanni
LADD 4D0M
LAI
CalaComm
Imargeheet
OpannJu
LAIDN--ADa
LAIDN-28
ImargeNit
Optanni
LAION-ADDOR
LAION-28
Datacoom
ImageNet
Training Dataset
Figure 3: Resulls of the unique concept analysis, showing lotal number of unique Broden concepls encoded in last layers, as in Equation (4). Zero-shot CLIP models encode substantially more concepts than supercised models.
which also includes (potentially less diverse) image tags in the text associated to each image, does not encode that many concepts, in fact only barely more than the supervised ResNet50. Thus, the number of unique concepts in the model features seems to be more strongly influenced by the type and diversity of the text associated lo images during training and their inclusion into the training objective, rather than by the model robustness or a root cause thereof.
SraageNet-Captionee
We can further compare the set of concepts encoded for the other architectures, by producing their Venn diagrams in Figure 4 (ImageNet-Captions models omitted here for bet ter comparability among backbones). In each case, the most significant section of the Venn diagrams is the overlap between all model types (this ranges from 237 concepts for RN-101 to 516 concepts for ViT-B/16). This suggests that all 3 models share large pool of features that are useful for each respective task the models were trained on. This is confirmed by Figure 5, where we track the size of overlap of the finetuned concepts with all sections of the Venn diagram (normalized by the number of finetuned concepts (Canel) during finetuning. As we can see, finetuning increases the amount of concepts that are shared by the zero-shot, finetuned, and supervised models, since Cine n Curro Csup increases. Interestingly, the overlap with concepts specific to the supervised models (C Cine)) \ Czeru) does not always increase substantially, which suggests that finetuning does not necessarily encode concepts that are exclusively learned in a supervised setting. In agreement with Figure 3, the Venn diagrams show that zero-shot models encode many concepts that are unknown to tne-tuned and supervised models (this ranges from 77 concepts for VYT-B/16 to 105 concepts for RN-50).
Connection to polysemanticity. A large number of encoded concepts can come at the cost. of inter- pretability. As explained by Olah et al. (2020), superposing many concepts in a given representation space creates polysemanlic features. Those features correspond to directions of the representation spaces that en- code several unrelated concepts, which makes the interpretation of such features challenging. Polysemantic features are typically identified by using feature visualization to construct images that maximally activate the unit ceeuron/represenaatinn space direction) of interest (Olah et al., 2017). A manual inspection of these images permits to identify that several concepts are present in the image maximizing the unit activation.
A manual feature visualizations for each RSV i/ of cach model our pool would be prohibitively expensive. For this reason, we use a prozy for polysemanticity based on the Broden dataset. For each RSV v; we count the number of concepts encoded in the corresponding direction of the representation space Neoncepo (1) ==
10

---

Published in Transactions on Machine Learning Research (10/2024)
OpenAl
Zeroshot Concepts Czoro
Zeroshot Concepts Caoro
Finetuned Concepts Crns
Finetuned Concepts Cin
105
101
246
257
11
104
51
30
Supervised Concepts Caup
Supervised Concepts Coup
OpenAl VIT-B/16
OpenAA VIT-B/32
Concepts Ceero
Zeroshot Concepts Cyera
Finetuned Concepts
Finetuned Concepts Cane
52
91
616
Supervised Concopts Coup
Supervised Concepts Csup
Figure 4: Overlap between the concepts encoded in the representation space of different models for each OpenAl models. Zero-shol models encode many concepls nol encoded other models.
He [C] AP; 0.9)|. The higher this number is, the more likely it is that the feature corresponding to v; is polysemantic. As measure of polysemanticity for the model, simply average this number over all singular vectors: polysemanticity
dn dd Neamecpt (t). By measuring this number for all zero-shot. models, we found that this ranges between polysemanticity = 3 for the OpenAl ResNet 50 and polysemanticity = 16 for the LAION-2B ViT-B/16. By looking at the complete results in Appendix E.4, we also note that most CLIP models are on the higher side of this range, with typically more than 10 concepts per direction on average. Since the Broden datasct has no duplicate concepts, we deduce that these models are highly polyscmantic.
Take-away 4. A larger number of concepts encoded in a model's representation space is not an indication of robustness since the number concepts encoded by some non-robust zero-shot ImageNet- Captions CLIP models can be as high as for some robust zero-shot CLIP models. However, both types of zero-shot CLIP models encode higher number of concepts than their supervised counterparts, and among the TaageNet-Captionn CLIP models, the amount of concepts varies depending the type of language supervision used. This supports the idea that the amount of concepts encoded is related to the language supervision of CLIP models. The large number of concepts encoded in CLIP models makes these models polyscmantic.
6
Related work
Interpretabilityy and CLIP models. number of works studied CLIP from a model-centric or inter- pretability perspective. We can broadly divide these works into two categories. (1) The first body of work, like ours, uses interpretability methods to gain better understanding of CL.P. For instance, Li et al. (2022)
11

---

Published in Transactions on Machine Learning Research (10/2024)
OpenAi
RN-101 OpenAi
1.0
1.0
0.8
0.8
Nomelieedhe bel size
sel size
Concept ovarlap
Concept averlap
0.6
L6ksa Guurll Youp
0.8
(Gre O Garol Guu
(Caaa Gup) San
(Ge fual
Give n Goew fi
Oua Glen Cap
Gie Curra Sep
0.4
Cass Cure Guu
02
0.0
10
Epoch
Epoch
VIT-B/16 OpanAl
VIT-B/22 OpannA
1.0
0.8
Norallzed set vize
Norwalood wetelze
Concept overlap
Concept averfap
0.6
1chee (1 Gorl Vor
0.8
(Gre R faee) Gue
Clp) 1/Gm
(Grv B Goal Gen
Care
Gres Gup
0.4
Gve Cora 1 Cop.
0.4
Gua Guee Gao
0.2
0.2
0.
0.0
10
Epach
Epoch
Figure 5: Overlap of the finetuned model concepts with zero-shot and supervised models during finetuning, normalized al each epoch by the number of fineluned concepts |Cinol- The overlap with the zero-shol-only (i.e. not overlapping with supervised) concepts decreases (blue curve), while concepts shared with zero-shot and supervised models increases (green curve).
analyze saliency maps of CLIP and found that they tend to focus on the background in images. Goh et al. (2021) analyzed CLIP ResNets and found mullimodal neurons, that respond to the presence of a concept in many different settings. (2) The second body of work leverages CLIP to explain other models. For instance, Jain et al. (2022) use CLIP to label hard examples that are localized as a direction in any modcl's repre- sentation space. Similarly, Oikarinen & Weng (2022) use CLIP to label neurons by aligning their activation patterns with concept activation patterns on a probing set of examples. To the best of our knowledge, our work is the first to leverage interpretability to better understand the distribution shift robustness of CLIP.
Outlier features in foundation models. Outlier features in foundation models were first discovered in LLMs by Dettmers et al. (2022). Those features are found to have an adverse effect on the model quantization. The reason for which outlier features appear in LLMs is yet unknown. Elhage et al. (2023) investigated several possibles causes (such as layer normalization), but found no conclusive explanation. They conclude that the emergence of outlier features is most likely relic of Adam optimization. Bondarenko et al. (2023) found that outlier features in transformers assign most of their mass to separator tokens and that modifying the attention mechanism (by clipping the softmax and using gated attention) decreases the amount of outlier features learned during pretraining. To the best of our knowledge, our work is the first to discuss outlier features outside of language and transformer models. We can also offer an alternative explanation for their emergence (see end of Section 4). Overall, our work shows that outlier features are more universal phenomena than previously known, and motivates further research to understand the mechanisms at play.
12

---

Published in Transactions on Machine Learning Research (10/2024)
7 Discussion
The goal of this work was to generate a better understanding of what models that are robust to distribution shifts around ImageNet have learned from data that distinguishes them from non-robust models. To this end, we conducted a thorough investigation of the representation spaces of robust CLIP models and their non-robust counterparts, analyzing a total of 39 models.
We found outlier features (Dettmers et al., 2022; Elhage al., 2023) be an indication of robustness, since they only be found in models that robust to ImageNet distribution shifts. To the best of our knowledge, this is also the first time that outlier features were observed outside of language and transformer models. Since the presence of outlier features can be detected without access to the shifted datasets, we believe that they could be a useful tool for practitioners to get feeling for the distribution shift robustness of pretrained model during deployment, when the exact form of distribution shift is typically unknown. Interestingly, we can also validate this indicator on robust non-CLIP models (CoCa, Yu et al. (2022)) Appendix D.
Lastly, found that larger numbers of encoded concepts in the reprosentation space rather related to the lype of language supervision than to be indication of robustness, since they can also be found in non-robust ImageNet-Captions CLIP models and their number varies with the type of text used for language supervision. It would be interesting to further investigate this hypothesis by creating different types language supervision for identical images, potentially leveraging state-of-the-art multimodal models, to create even richer signals.
In general, we believe an interesting avenue for future research would be extend the analysis of this work to dataset shifts beyond the ImageNet family, see if our analysis remains relevant beyond the much investigated ImageNet shifts.
Acknowledgments
The authors thank Hadi Pour Ansari, Dan Busbridge, Miguel Sarabia del Castillo, Barry Theobald, and Nicholas Apostoloff for their helpful feedback and their support.
13

---

Published in Transactions on Machine Learning Research (10/2024)
References
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca. Roelofs. The evolution of out-of- distribution robustness throughout fine-tuning. arXiv preprint arXiv:2106.1588 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprinl ar trXio:1907.02893 2019.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenen- baum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Aduances neural information processing systems, 32, 2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. arXiv preprint brXh::2306.1299. 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-lext pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive aanguag--maage learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 28299, 2023.
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248 255. leee, 2009.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXio preprinl urXiv:2208.. 07339, 2022.
Benjamin Devillers, Bhavin Choksi, Romain Bielawski, and Rufin VanRullen. Does language help general- ization in vision models? arXie preprint arXiv:2104.08318, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arx:o:2010. 11929, 2020.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprinl ar Xiv:2200. 10652, 2022.
Nelson Elhage, Robert Lasenby, and Christopher Olah. Privileged bases in the transformer residual stream, 2023. URL
www.raansfooreerrrrruutttteeeeee www.ciitt..ub/eeee.e.. httb/2223/privilegeee bas.sgedddbassiiideeee.eeeeaeeee html.
circuits.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pro-training (clip). In International Conference on Machine Learning, pp. 6216 6234. PMLR, 2022.
Alex Fang, Simon Korublith, and Ludwig Schmidt, Does progress on imagenet transfer to real-world datasets? arXiv preprint XrX:::2301.04644 2023.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Diställ, 6(3):e30, 2021.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Mcasuring statistical depen- dence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Com- puler Vision, pp. 1134-1144, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arxiv e-prints. XrXiu preprint ar 1512 053880888, 10, 2015.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. CVPR, 2021b.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL ptttss//doo..ogg 10. http://eeoooo..14777..
Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as dircctions in latent space. arxiv preprint arXiv:2206. 14754, 2022.
Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts. Advances in Neural Information Processing Systems, 35:20782- 25220922-0079, 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519 3529. PMLR, 2019.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 2022.
Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xiaomeng Li. Exploring visual interpretability for contrastive image pre-training arXiv preprint or :xXx::229.000446 2022.
Maggie Makar, Ben Packer, Dan Moldovan, Davis Blalock, Yoni Halpern, and Alexander D'Amour, Causally motivated shortcut removal using auxiliary labels. In International Conference on Artificial Intelligence and Statistics, pp. 739 766. PMLR, 2022.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift question answering models. In International Conference on Machine Learning, pp. 6905-6916. PMLR, 2020.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Thao Nguyen, Gabriel Ilaarco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. arXie preprinl arXiv:2208.053: 2022.
Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron representations in
deep vision networks. arXiv preprint arXiv:2204.10966 2022.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11)ee7 2017.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3)=000024-001, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMT.R, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? orXiv preprinl arXiv:1806.0044 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International conference on machine learning, pp. 5389 5400. PMLR, 2019.
Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. Xixiv preprint orXic:2210.0111 01899, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. orXiv preprint arXiv:2210.08/4 000.22000,840,,, 2022.
Zhouxing Shi, Nicholas Carlini, Ananth Balashankar, Ludwig Schmidt, Cho-Jui Hsich, Alex Beutel, and Yao Qin. Effective nosustiness against natural distribution shifts for models with different training data. arXiv preprint XXie:2332201381 2023.
Xavier Suau, Luca Zappella, and Nicholas Apostoloff. Self-conditioning pre-trained language models. In International Conference on Machine Learning, pp. 4455 4473. PMLR, 2022.
Anand Subramanian. Torch cka, 2021. URL
com Poatiiikk/Pyoorc-oooolll
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint 2rXiv:2306.1100 2023.
Mingjie Sun, Xinlei Chen, Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint 2rXiv:2402.1776. 2024.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mea- suring robustness to natural distribution shifts in image classification. Advonces in Neural Information Processing Systems, 33:1 33:18583-18599 2020.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59 (2):64-73 2016.
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://githus: com/pytorch/viision, 2016.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advonces in Neural Information Processing Systems, pp. 10506-10518, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongsook Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- PMLR, 2022a.
tional Conference Machine Learning, pp.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7959, 2022b.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung Mojtaba Scyedhosscini, and Yonghui Wu. Coca: Contrastive caplioners are image-texi foundation models. Xiv preprinl arXin:2205.0191 2022.
17

---

Published in Transactions on Machine Learning Research (10/2024)
71
Baseline Models
61
Baseline Models Fit
ImageNet Shifts Accuracy [%]
51
y y=x
xx x
CLIP Backbone
41
RN-101
31
RN-50
VIT-B/16
21
ViT-B/32
Model Type
11
Zeroshot
Finetuned
10 20 30 400060 70 80 90
ImageNet Accuracy [%]
Figure 6: Accuracies of baseline models and OpenAI CLIP models ImageNet and (average) accuracies its five natural shifts (ImageNet- V2, ImageNet- R, ImageNet- SmageNeetSSketch, ImageNet-A, ObjectNet). The zero- shot OpenAl CLIP models accuracies are substantially above the baseline fit, i.c., they have high ER. The fine-tuned models are closer to the baseline fil, i.e., they have lower, but still significant ER.
A Computing Effective Robustness
In this section, define Efffective Robustness (ER).
Context. ER. has emerged as natural metric to measure how the performance of model reference distribution (in-distribution) generalizes to natural shifts of this distribution (Fang et al., 2022). When plotting the in-distribution accuracy (X-axis, logit scaling) against the average shifted-distributiin accuracy (Y-axis, logit scaling) of various architectures trained on ImageNet, Taori et al. (2020) found that most of the existing models lie on the same line. They also found that models trained with substantially more data lie above this line, showing a desirable gain in shiftod-distribbttion accuracy for fixed in-distribution accuracy. They coined this vertical lift above the line as Effective Robustness.
Computing ER. To quantify ER, following Taori et al. (2020), gathers the ImageNet test accuracy ACC(T) and the average accuracy over the ImageNet shifts ACC(S) of set of reference models trained on ImageNet and fits a linear model on this pool of accuracies to map logit(ACc(/)) to logit(ACC(SS),, with the logit. function logit [o, 1] R defined as & - In(x) - In(1 - œ)) The resulting line can be used to predict what (logit) accuracy we would expect to see on the ImageNet shifts, given a (logit) accuracy on the original ImageNet. Given a new model that has accuracy ACC(T) on ImageNet and average accuracy ACC(S) on the canonical ImageNet shifts, ER computed
ER(ACC(S)) ACC(!)) ==
ACC(S) logit [88 logit [ACC(/)) + 8o]
(5)
By fitting line on the bascline accuracies collected by Taori (2020), we get slope of 81 .66 and an intercept of By -149 with Pearson corrclation .99. This line, along with the bascline models, be observed in Figure 6.
18

---

Published in Transactions on Machine Learning Research (10/2024)
B Model Accuracies on ImageNet and its shifts
In Table we show the top-1 accuracies that the models analysed in the main part of the paper achieve on the ImageNet test set, and in Table 2 the test accuracies on the five shifted datasets (averaged).
Table 1: ImageNet test set accuracies for the models under investigation
Backbone
Pretraining data
Zero-shot CLIP
Finetuned CLIP
ImageNet supervised
OpenAl
60
%
YFCC-15M
32 %
69 %
CC-12M
36
69 %
ResNet50
70 %
Captiont
25 %
ImageNet-Captions-tt
27
N.A.
28 %
OpenAT
62 %
78 %
ResNet101
71
YFCC-15M
34
72 %
OpenAT
68 %
81 %
LATON-100M
67
%
ViT-B-16
80 %
LATON-2B
70 %
81 %
DataComp
63 %
78 %
OpenAI
63 %
78 %
LATON-400M
60 %
76 %
75 %
ViT-B-32
LAION-2B
66
76 %
OpenAT
75 %
85 %
LATON-100M
73
%
VIT-L-14
N.A.
LAION-2B
74 %
84 %
DataComp
79
85 %
Table 2: Average test set accuracies ImageNet- V2, ImageNet-R, ImageNet-Sketch,, ImageNet-A, ObjectNet for the models under investigation.
Backbone
Pretraining data
Zero-shot CLIP
Finetuned CLIP
ImageNet supervised
OpenAl
44 %
43 %
YFCC-15M
18
30 %
CC-12M
27 %
36 %
ResNet50
30 %
Cappiontt
9 %
ImageNet-CCaptionstt
%
N.A.
GrageNet-Caations-tee
10 %
OpenAI
49
%
ResNet101
31 %
YFCC-15M
21 %
33 %
OpenAl
60
55 %
LATON-400M
56 %
56 %
VIT-B-16
41 %
LATON-2B
60 %
58 %
DataComp
52
%
OpenAI
50 %
15 %
LAION-4000 400M
48 %
48 %
35
ViT-B-32
LAION-2B
54 %
48 %
OpenAl
72 %
67 %
LATON-400M
64
64 %
VIT-I-14
N.A.
LATON-2B
66 %
66 %
DataComp
76
%
C Zero-shot and finetuned models' differences are localized
In this appendix, we apply central kernel aligmment (CKA) to identify where changes between robust zero- shot CLIP models and their less robust fine-tuned counterparts occur. Kornblith et al. (2019) introduce
19

---

Published in Transactions on Machine Learning Research (10/2024)
the CKA metric to quantify the degree of similarity between the activation patterns of two neural network layers. It takes two batch of activation vectors and b, it computes their normalized similarity in terms of the Hilbert-Schmidt Independence Criterion (HSIC, Gretton et al. (2005)):
(SIC b) CKA(a, b) = /HSTC(a, a) / HHSTC(b, b)
We use the PyTorch-Moddd-ompp package (Subramanian, 2021) to compute this metric between the activation vectors of zero-shot models and their finetuned counterparts for each layer in the backbone. The results are shown in Figure 7 and Figure &. Across architectures and pretraining sets, we find that there is often large drop CKA between zero-shot and finetuned models occurring the last layer. This makes the activations in the last layer particularly interesting layer to analyse when investigating ER, fine-luned models typically have only half the ER of their zero-shot counterpart (see Figure la).
D Robustness indicators in non-CLIP models
In addition to the CLIP models investigated in the main paper, below investigate CoCa models (Yu et al., 2022) pre-trained LAION-2B as another set of robust multimodal models which do not fall into the CLIP lamily. We see that our findings extend to these non-CLIP multimodal models as well.
First, Table 3, we confirm that these models have high effective robustness when used as zero-shot classifiers. As for the other models in the main part of the paper, we observe that finetuning on ImageNet decreases the effective robustness of these classifiers.
Table 3: ER for the models of the CoCa family, calculated by Equation (5) (accuracies on ImageNel shoun in brackets). We see that also for these models, ER decreases from Zero-shot to Finetuned.
Backbone
CoCa Zero-shot
CoCa Finetuned
VIT-B-32
24% (64%)
14% (76%)
ViT-I--14
34% (76%)
21% (84%)
Next, in Table 4, we show that all the zero-shot models have high kurtosis, which implies the existence of outlier features in their representation space. Additionally, show that finetuning again decreases the kurtosis.
Table 4: Results of the kurtosis analysis showing outlier features present also in robust models with ViT-L-14 backbone or of CoCa family, but disappearing as soon as models are finetuned. Values calculated according to Equation (1) over all ImageNet test examples.
Backbone
CoCa Zero-shot
CoCa Finetuned
ViT-B-32
12.0
3.6
VIT-L-14
15.5
4.6
Finally, in Table 5, we see that zero-shot model encodes more concepts. Again, we see that finetuning removes some concepts from the model's representation space. However, from our experiments with ImageNet- Captions CLIP models, we know that this does not correspond to robustness indicator.
Table 5: Results of the unique concept analysis, showing total # of unique Broden concepts encoded in last layers, as in Equation (4). Zero-shot models encode substantially more concepts.
Backbone CoCa Zero-shot.
CoCa Finetuned
VIT-B-32
674
530
ViT-I-114
747
629
20

---

Published in Transactions on Machine Learning Research (10/2024)
RM-E0 OpenN
RN-1D OpenA)
OSA
04
04
0000
1000
Iapere
poo
0000
gerd
cunstraatinn 200d
Layor Name
Layor Nome
VIT-P115 OpenAl
WT-2032 OpenA)
04
03
04
(bianne rame
32
imogenet-shifb imagenet
10000000
no00000
reb 1okk1
laye: porm poot
red tedrock
syee somm pook
wasatsooioo pood
Syye poim
vol
Layer
Name
RN-50 VF0C-15M
Riu5o DC-12M
10
C3
08
64
doloseL/nzme Litme
datonet, hamre imagenal-shhrt whitt imagenet
03
imagevet
imapenet-attts
opte
Nayor
abenton posl
Jeys7
poot
Layor
Name
Figure 7: Result of layer by layer CKA comparison between zero-shot CLIP and its counterpart that was fine-
buned on IetgeNet for various backbones and pretraining sets (Part 1). In orange, CKA between activation vectors on ImageNet test set. In blue, CKA between activation vectors on shifted ImageNet sets (average as solid line, standard deviation in shaded blue). Typically, we see large drops of CKA in the last layer.
21

---

Published in Transactions on Machine Learning Research (10/2024)
FN 104101 YFCO 15M
VIT 3/16 LAJON 400M
D5
04
03
6000
2003
beyer webooo0
contrraation 2o0d
Layor Namo
Layor Nomo
VIT-EUIG LAIDN-211
VIT-802 LAION-
chrisnet
013
85
CAA 02
95
04
184
C2
Spyee poim 2ie
10000000
no00000
reb 10tk1
laye: strm, poot
red modt
wpei som pook
sopsaa weaasoaaee Pood
Layer
Name
WT-892 LAION-2B
10
C2
0.8
37
DRA
010
05
04
S0ttooo Lntose_namma nome
irsgeestt-diits imsggnnst
52
500
mibbock
redioou
dosxitaaionn badd
love pom
Layor
Figure 8: Result of layer by layer CKA comparison between zero-shot CLIP and its counterpart that was fine- buned on IetgeNet for various backbones and pretraining sets (Part 2). In orange, CKA between activation vectors on ImageNet test set. In blue, CKA between activation vectors on shifted ImageNet sets (average as solid line, standard deviation in shaded blue). Typically, we see large drops of CKA in the last layer.
22

---

Published in Transactions on Machine Learning Research (10/2024)
E Further experiment results
E.1 Importance score analysis for each model
In Figure 9 and Figure 10 we show the the full distribution of importance scores (Equation (2)) over all representation space directions for all models.
We see that robust zero-shot CLIP models and finetuned CLIP models have one strongly privileged direction, typically orders of magnitude larger than the bulk of importance scores. For the non-robust models trained Iet and ImageNet-Caption, this single outlier value in importance scores does not exist.
E.2 Pruning analysis for each model
Figure 11 and Figure 12 show the effect of gradually pruning the least important SV of W on ER and ACC for all models that were not pretrained on the OpenAl pretraining dataset, similar to the analysis shown in Figure 13 in the main paper.
Qualitatively, our finding from the main paper is confirmed across the remaining models we investigate: A small subset (typically around 20%) of privileged directions in representation space explain the high ER of zero-shot models. The remaining directions can be pruned without significantly impacting neither nor ACC.
E.3 Top-3 concepts in most dominant direction for each model
We look at the concepts with highest AP in the most privileged direction of each model represented in Figure 9, similar to what we did in Section 5 for the most privileged direction of each model in Figure 2b. The results shown in Table 6. Again, we observe that for the majority of cases, the robust zero-shot models encode concepts related to textures their top concepts encoded along the privileged directions (e.g., scaly, meshed, or matted), while the less robust finetuned models encode more concrete concepts (e.g.. carrousel, book stand, or pantry).
Table 6: Top-3 concepts with highest AP encoded in the most privileged direction of each model. For each. concept, the AP is included in brackets.
Cuucapt
Stup Pootroiatig
Plactuaed
Zenozst
Fustiand
Zenoist
Flactuao
Zendar.
Backhhoe
Right vockaries inber 3.90)
gime (0.90)
pit (000))
R0C.00
Segended Traegellee
N.A.
kolvichh
(8.k)
Y20C--3M
chepe af
pezloe (156)
peairy (497)
Iespll (LSS)
teaple
9.56)
CC-124
Wiazz 3022 (LLR))
Dansal dianee [CP)
apoand ILCS
jedlo dolied (100)
15N-50
OperA1
Trageelie
nomrtaalo pos 3:99)
Iniited (8)
Vegetable
Tritte (191)
sooee will st)
otlinct tebi (.87)
YP0C-USM
back
3331
22l (USS))
Tamgaa [091)
slapp TUSS.
Ecc innling (1200
LARIN-21 201
Bood (.9)
enatted (1MD)
catooak [ (U47)
eelly BAS
roloce (1.5)
etacteed (.00)
VIL-B/16
LARON-
OpeLA]
book staud 3007)
axaied
bosgle 677
Loodstonn' ((44)
Cou toadss 0.55)
nalier's 563
posforteed (.00)
calaune (.0k)
V213C-13M
Soratale supd)
A.A
bark (U57)
N.A.
air have (16))
N.A.
fra 73M)
creat perter (1D))
dkiinen: [111)
killingg (TD)
27t 2agg (135)
LAMOX
staltct 3:03)
stalted ( 61)
ovigo overalse Intevicr [191
wall (O (859)
fabyraec (..0)
vurted
VIL-B/33
lanosed
axde (1.54)
anbbege interter (.00)
jul adia (1)))
kind kedda 30300)
sanaiade (3.00
Supordtdd trogyhlee
beleekdra stafoo
Terherer
Tned
N.A.
E.4 Polysemanticty for each model
We report the polyscmanticity metric computed as per Section for all zero-shot models in Table 7. As claimed in the paper, this ranges from polysemanticity for the OpenAI ResNet 50 to polysemanticity 16 for the LAION-2B ViT-B/16, with typically more than 10 concepts encoded in direction on average.
23

---

Published in Transactions on Machine Learning Research (10/2024)
RN-50
16.7
10
Importatin
1271
Model Calegary Recunt CLF
Finekrred Cup
Iaae lad CappiCalloinn CLP
Ihage Net Suparies)
Pratnaring
RN--111
10"
Finwsined CUIP
magellen Caplone
magatter Supariee)
Opanti
VPOC-19V
Supendeed Imagalis!
10.
Iree cou/tano soMe dalbadion oraral Srookoo
10"
Model Calegory
OpesAl
LAIGN-IEN
LAUN-20
VP00-15M
Siperveed Imagetet
DelaCorm
Phetaerig
Figure 9: Distribution of importance over all RSVs in the representation space for all models with ResNet50, 10s, and ViT-B-16 backbones. Robust zero-shot CLIP models (blue) have one strongly privileged direction. Fine-tuned models (green) still exhibit one strong privileged direction, but with lower importance than the robust. zero-shot models. The supervised models (orange) do not have them at all. The non-robust ImageNet-Capticn models (red) do have few directions that are little bit larger than the bulk, but not separated by orders of uagnitude like for the robust and fine-tuned CLIP models.
24

---

Published in Transactions on Machine Learning Research (10/2024)
VI-000
Moveel Catagery
Resud CLP
*ea-ared CUP
maagtter Supaveel
OpenAi
LAION--220 4A00N--40M
Superias Imagehir
DetaComp
Preturring
10
10
Satepory
Ratual
imgaltee Superited
LAIGN400M
LAK7422
Openii
Protorrig
Figure 10: Distribution of importance over all RSVs in the representation space for all models with VIT-B- and VIT-I-14 backbones. Robust zero-shot CLIP models (blue) have one strongly privileged direction. Fino-tuned models (green) still exhibit one strong privileged direction, but with lower importance than the robust zero-shot. models. The supervised models (orange) do not have them at all.
25

---

Published in Transactions on Machine Learning Research (10/2024)
Effecivv Robusiness
Accuracy [%]]
Backbone
VIT 8/32
30
VIT-8/16
Backbone
WIT-8/32
ImageNet
VTT-0/16
ImageNel- Shills
Number of Ablated SV (%)
Number of Ablated SV [%]
(a) LAION-40MM pretrained
(b) ACC LAION 400M pretrained
28
26
Eftoctive Robustness
124
Apcuracy [%6]
22
Backbone
VIT-B/32
VIT-8/16
Backbone
Dataset
VIT-8/32
ImageNet
WI-B/166
ImageNet Strfts
Number of Ablated SV [%]
Number of Ablated SV [%6]
(c) LAION-2B pretrained
(d) LAION-2B pretrained
20
Backbone
MTTB/332
Accurasy
B/16
Dataset
Effective
ImageNet
ImageNet Shifts
Backbone WIT
VIT-B/18
10
5
100
60
Number of Ablated SV (%)
Number of Ablated SV [%]
(e) ER DataComp pretrained
(f) ACC DataComp pretrained
Figure 11: Effect of gradually pruning the least important SV of W on ER and ACC for LAION-40OM, LAION- 2B, and DataComp pretrained models.
26

---

Published in Transactions on Machine Learning Research (10/2024)
Backbone RN-60
32.5
7.3
30.0
Rebustness
Backbone
RN-50
Acxuracy
Dalasel
22.5
ImageNer
ImageNei-Shitt
7.0
20.0
17.6
20
40
80
80
100
20
40
80
100
Number Ablated SV [%]
Number Ablated [%]
(a) ER YFCC-15M pretrained
(b) ACC YFCC-15M pretrained
14.2
14.0
13.5
13.6
Accurscy [96]
Eftecivv
Backbono
13.0
12.8
24
Dataset
Backbona
ImageNet
12.6
22
ImageNee-Shiii Shifts
40
80
100
20
40
60
80
100
Number Ableted [%]
Number Ablated SV
(c) ER CC-12M pretrained
(d) ACC CC-12M pretrained
Figure 12: Effect of gradually pruning the least imporlant SV of W on ER and ACC for YFCC-15M and CC- 12M pretrained models.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Polysemanticity of zero-shot CLIP models, showing average # of concepts encoded in RSV of last layer (i.e. with AP 0.9 on Broden dotaset concepts).
Backbone
Pretraining data
Polysemanticity
OpenAI
3.0
ResNet50
YFCC-15M
6.4
CC-12M
5.3
ImageNet-CCapiionst
10.5
ImageNett-Capiionsst
10.5
ImageNet-Captions-ee
4.5
OpenAI
3.5
ResNet1011
YFCC-15M
7.8
OpenAI
14.5
LATON-400M
11.9
ViT-B-16
LATON-2B
16.0
DataComp
14.1
OpenAT
14.1
LAION-400 4000
11.5
ViT-B-32
LAION-2B
11.1
F Intuition behind generalization of outlier features
Below, we give more details and an intuition why outlier features can be generalized from the canonical basis {e1,...: edg (we can write h = Proje, to be any set of directions of the representation space that receive projection substantially above average:
Let us assume, for instance, that two of the elements in the canonical basis and e2 correspond to outlier features. This means that an activation vector h related to an input image has projections h1 = Proje, (h) and h2 = Projeg (h) substantially above the average hı, h2 > n-1 Σ h; Now let us define a new unit vector ej 2- (el
22). We deduce that the projection onto this vector is also substantially higher than
average h1 = Proj.; (h) = 2 2-112112 1/2(1 + h2) > n Ei= h4. Hence, the unit vector ei can be considered as an outlier feature in a new non-canonical basis. In general, we can extend the notion of outlier features to any vector in the span(ei....... R").
G Pruning results
Pruning non-privileged directions. Given that we have established that outlier features induce privileged directions in representation space, it seemss interesting to check their role in model performance. To that aim, we gradually prune each RSV y; by increasing order of o; by setting o, < the singular value expansion* W = ii=1 ouuuy. By pruning variable proportion of the singular vectors, we obtain the results in Figure 13. We see that the 80% least important RSV of the representation space can be pruned without a substantial effect on performance, i.c. that the robust models are low-rank in their last layer where they have privileged directions.
When extending the pruning experiment to finetuned CLIP models and supervised models trained only with ImageNet, we make the two interesting observations from these new results (see Figure 14):
All models are low-rank. For all the models (zero-shot, finetuned and supervised), the performances are not substantially affected if we remove the 80% least important singular directions of their representation space (compare to Table 1). This shows that many existing models admit good low-rank approximations. This also demonstrates that the fact that these models are low-rank is not necessarily an indicator of robustness.
NNote that sorting the RSV 8n by increasing di similar sorting the RSV by increasing Importance()), since these two variables are related by Spearman rank correlation p = 96%
28

---

Published in Transactions on Machine Learning Research (10/2024)
70
60
Effective Robustness
50
Accuracy [%)
20
40
Backbone
RN-50
RN-101
16
Backbone
VIT-B/32
RN-101
Dataset
VIT-8/16
ImageNet
10
VTT-B/32
ImageNet Shifts
20
40
80
80
100
20
40
60
80
100
Number [%]
Number Ablated [%]
Figure 13: Effect of gradually praming the least imporland SV of W on ER and ACC. The least 80% important SV can be pruned without any substantial effect.
Variant
70
Firetuned
Supervised
Zeroshol
Datasel
ImageNet
Ascuracy [S0
Acturacy [8]]
ImagoNet-Shifts
Variant
Finetuned
Supervised
Zeroehot
Dataset
10
InsggeNet
ImageNot-S sitts
75
85
96
100
75
85
95
100
Number of Ablated SV (%)
Numbar of Ablated SV [%]]
(a) ResNet50
(b) ResNet101
Figure 14: Extension of pruning results to finetuned and ImageNet supervised models. Zero-shot models obtoined with OpenAl pretraining set.
Faster drop for supervised models. When the number of ablated singular values ranges between 80% - 100%,
that the ImageNet accuracy of supervised models drop substantially faster than the accuracy of the finetuned and the zero-shot models. In fact, for the ResNet50, the ImageNet accuracy curves even cross. This implies that the most important direction of the zero-shot model's representation space better discriminate between ImageNet classes than the most important directions of the supervised model's representation space. In the former case, these directions correspond to the zero-shot model's privileged dircctions. We believe that this new result further reinforces the importance of privileged directions understand the performances of robust models.
H Details on experiments
H.1 Finetuned CLIP models
To obtain the finetuned CLIP models, we proceed follows. We start from building the zero-shot CLIP models as described in Section 3. As Wortsman al. (2022b), then finetune these models for 10 epochs
29

---

Published in Transactions on Machine Learning Research (10/2024)
the ImageNet training set, using a batch size of 256 and learning rate of 3- 10-0 with cosine annealing learning rate scheduler and warm-up of 500 steps. We use the AdamW optimizer and set the weight decay to 0.1.
H.2 Supervised ImageNet models
We note that the ResNets used by Radford et al. (2021) have small modifications, such the usage of attention pooling. Unfortunately, we are not aware of any public weights for such modified architectures trained on ImageNet from scratch. We thus train these modified ResNet models from scratch for 90 epochs on the ImageNet training set, using a batch size of 1024. We use AdamW, and a learning rate schedule decaying from 10-3 to 10 after 30 epochs and 10-5 after 60 epochs (with a warm-up period of 5,000 steps). We set weight decay to 10 2, We use the standard augmentations of horizontal flip with random crop well as label smoothing.
For the VIT models, loadable checkpoints with identical architectures were available from torchvision (TorchVision maintainers and contributors, 2016), and we thus use those directly.
Analysis of Wise-FT models
In this appendix, we use the approach of Wisc-FT (Wortsman et al., 2022b) to obtain continuous spectrum of ER. Given a zero-shot model Sa, with weights Og E O and finetuned model fa, with weights 0, E O Wortsman et al. (2022b) propose to interpolate between the two models in weight space. This done by taking combination 0g == (1 - a) -0) + a 02 for some interpolation parameter a E [0, 1]. One then defines a new model fe_ based on the interpolated weights.
Surprisingly, interpolating between zero-shot CLIP models and finetuned CLIP models produce models with good performances. To illustrate that, we perform the Wise-FT interpolation with all the models from our pool. We report the ImageNet & shift accuracies of these models in Figures 15 and 16. For the OpenAI and LAION models in Figure 15, we observe that the shift accuracy of interpolated models often surpass both the zero-shot and the finetuned models. The YFCC-15M and CC-12M models in Figure 16 exhibit a different trend: both ImageNet & shift accuracies increase monotonically as sweeps from zcro-shot finetuned. This likely due to the low accuracy of the corresponding zero-shot models.
By analyzing the ER of interpolated OpenAl and LAION models in Figure 17, see that the ER gradually degrades a sweeps between the zero-shot and the finetuned models. Interestingly, the ER of YFCC-15M and CC-12M models in Figure 18 peaks at a = .4 and then decreases monotonically.
Let us now look at how our robustness indicators cvolve as we sweep a between zcro-shot and finctuned models. Ideally, if these indicators are good ER proxies, they should exhibit similar trends as the ones described in the previous paragraph. For the OpenAl and LAION models, we indeed observe in Figures 19 and 21 that the kurtosis and the number of unique encoded concepts gradually decrease as sweeps from zero-shot to finetuned models. Similarly, substantially after = 0.4 for the YFCC-15M and CC-12M models. This suggests that these two metrics constitute a good proxy to track how the ER of a given model evolves.
observe in Figures 20 and 22 that these two metrics start to
Note that the Wisc-FT idea has since been generalized to a combination of several finetuned models by Wortsman et al. (2022a) with model soups. We leave the investigations of model soups for future work.
30

---

Published in Transactions on Machine Learning Research (10/2024)
R1-50 OownAl
RN-101 OpenAl
0.60
0.75
0.70
0.60
catasot
magensl
magenst
iaagenebents
magenehehits
0.45
0.50
0.0
0.2
0.4
Appha
0.6
0.8
Finduned
20r00not
0.2
0.4
Aspin
0.6
0.8
Finokned
VIT OpenAi
OpenA)
0.75
0.
0.60
2.65
0.55
0.50
iaggnee
magenee
Imagenebents
magenehettts
2510shot
0.2
Fincluned
20: 22100700
Fincunedd
VIT NTT-BM1 LAION
LAION 400M
0.90
0.75
Acor33
2.
0.65
dalaeeel pame
maganel
maaenstiihifs
0.59
Friduned
Finouued
A LAION 28
VIT-B-I2
LAIOON LAION 23
0.80
1ocubryy
0.65
2.00
0.60
dstseel, name
datasre name
3.90
magentt
magenst
megenstenfa
magensishnls
0.0
02
04
Aphha
06
0.8
Fineured
Zeroent
0.2
04
Alpha
06
08
10
Figure 15: Accuracies ImageNet & shifts for Wisc-FT models (1/2).
31

---

Published in Transactions on Machine Learning Research (10/2024)
RN-50 YFCC-15M
RN-101 101 YFCC-15M
0.7
dataset
0.7
set.
Imaganat
imagenel
imagenef-ahifft
imagenel-shiits
0.8
0.8
Accuracy
0.5
0.4
0.0
0.2
0.4
0.6
0.8
Finetuned
0.0
0.2
0.4
0.5
0.8
1.
Alpha
Alpha
RN-50 CC- 12M
0.7
datasel_ name
Imagenet
imagenet-shiits
0.5
Accuracy
0.3
0.0
0.4
Alpha
0.8
Finetuned
Figure 16: Accuracies on ImageNet & shifts for Wisc-FT models (2/2).
32

---

Published in Transactions on Machine Learning Research (10/2024)
F81-50 OpenAi
RN--01 OpenAi
0.24
EMoctive RIb0ST100
0.16
0.10
0.14
0.12
20roohot
0.2
0.4
Appha
0.6
0.8
Finduned
20r00not
0.2
0.4
Appin
0.6
0.8
Findkned
OpenAi
OpanA)
0.24
Etfectve Bosusteees
0.80
0
0.255
0.16
0.10
2510shot
0.2
Apha
Fincuund
20106700
Fincuned
VIT LAION 400M
VIT 400M
0.29
0.22
Eftctto Robusinse
0.20
0.22
0.14
3.16
Prlduned
Finouned
Apthw
VIT-BK16 LAVON 28
VIT VIT-B-22 LAION LATB- 28
0.28
0.24
3.28
0.24
0.20
0.22
3.20
0.0
0.
0.4
Appha
0.6
0.8
Fineured
Zeroenet
0.2
044
Alpha
0.6
0.
10
Figure 17: ER for Wisc-FT models (1/2).
33

---

Published in Transactions on Machine Learning Research (10/2024)
RN-50 YFCC-15M
RN-101 YFCC-15M
0.09
0.07
0.08
0.06
0.06
0.07
Effeclive
0.04
Effeclive
0.00
0.05
0.02
0.04
0.01
0.0
0.2
0.4
0.6
0.8
Fineluned
0.0
0.2
0.4
0.5
0.8
1.
Alpha
Alpha
RN-50 CC-12M
Effectvv Robustress
0.12
0.11
0.10
0.09
0.0
0.4
0.8
Finetuned
Alpha
Figure 18: ER for Wise-FT models (2/2).
34

---

Published in Transactions on Machine Learning Research (10/2024)
FN 02 Opensi
RN 101 OpenAi
chiosel 2WmT iegentt
99
diteovel name
120
magenst
maagenshenta
30
103
Adiistion Humostt
80
N2hrstton
10
15
10
0.0
0.2
0.4
Aptha
0.6
0.8
Finetred
0.0
0.2
0.4
Apphx
0.6
0.8
1.0
VIT-8/16 OpanAA
VIT-B/S2 OpanAA)
100
dataset name
iagenne
100
didateel name
iaaeentt
megensiintss
magendisinla
83
80
Actission Kurtore
Achuntion Nartocik
40
20
20
Zwroehot
0.2
Aspha
0.5
Finsureed
0.2
0.4
Alphra
Finwtreed
VIT-8/16 L000N
VIT-B/22 LAION 400M
dutasot,
datasat, namo
imagenel
imagnnel
14
25
Activaion Kirbols
12
Activtton Kuroots
20
10
15
10
Filduned
Finduned
VIT LAION 28
VIT LNION 29
namo
datasot,
megerit
inagenct
35
inagenstanta
16
iaaennsiaints
Actictionn Harbals
30
Achhation Kumbotts
14
25
12
10
10
0.0
0.2
0.4
Appha
0.6
0.8
Filoured
0.2
0.4
Alpha
0.6
0.8
Findured
Figure 19: Activation kurtosis for Wisc-FT models (1/2). The activation kurtosis is computed for both ImageNet tests and ImageNet shifts.
35

---

Published in Transactions on Machine Learning Research (10/2024)
YFCC-15M
YFCC-15M
10
10
imagenal
imagenel-shitss
Actvaaion Kunosis
Adivatiin Kufoolls
8
5
dataset_name _name imagenet
imagenet-shilts
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Alpha
Alpha
RN-50 CC-12M
Activation Kirtosis
dataset_ name
imagenet
imagenet-shills
0.0
0.2
0.4
0.8
Finetuned
Alpha
Figure 20: Activation hurlosis for Wise-FT models (2/2). The activation kurlosis is computed for both ImageNet tests and ImageNet shifts.
36

---

Published in Transactions on Machine Learning Research (10/2024)
F81-50 OpenAi
RN-101 OpenAi
100
Number Lnigue cOniCAT deledoor
475
800
K25
420
375
225
20roohot
0.2
0.4
Appha
0.6
0.8
Finduned
20r0snot
0.2
0.4
Aipin
0.6
0.0
Finkkned
OpenAl
OpanA)
Mimberd unique contagt dedeckre
800
660
600
840
1230
600
000
660
2510shot
0.2
Apha
Finclundd
22106met
Fincuned
VIT 400M
VIT 402M
080
Number untuu dereciees
Nimber unieue concect delecto
890
6:0
620
600
Prlduned
Finduned
Apthw
B LAVON
VIT 8-12 LAION LATON 28
delectors
720
Number wwu.ggu eorront delector
700
10:0
600
820
800
580
0.0
0.2
0.4
Appha
0.6
0.8
Fineured
Zeroenet
0.2
044
Alpha
0.6
08
10
Figure 21: Number of unique concepts encoded in Wise-FT models (1/2).
37

---

Published in Transactions on Machine Learning Research (10/2024)
RN-50 YFCC-15M
RN-101 YFCC-15M
850
Number unioue conaap detectors
Number of unique concegt concept deteectors
820
840
610
830
803
820
880
810
800
0.0
0.2
0.4
0.6
0.8
Fineluned
0.0
0.2
0.4
0.5
0.8
1.
Alpha
Alpha
RN-50 CC-12M
Number concapt delecloos
850
840
630
820
810
600
0.0
0.4
Alpha
0.8
Finetuned
Figure 22: Number of unique concepts encoded in Wise-FT models (2/2).
38

---

Published in Transactions on Machine Learning Research (10/2024)
J Further literature
Defining CLIP ER. The definition of ER, crucially relies on the observation in multiple works that the model performance on natural shifts is linearly related to its performance in-distribution when both quantities are plotted with a logit scaling (Recht et al., 2018; 2019; Miller et al., 2020). We note though that there are known exceptions to this, e.g. considering out-of-distribution generalization on real-world datasets that substantially differ from the in-distribution dataset (Fang et al., 2023).
Explaining CLIP ER. A first. intuitive explanation for the surprisingly high effective robustness of CLIP might be the fact that the learned embeddings are endowed with semantic grounding through pretraining with text data. This hypothesis was refuted by Devillers et al. (2021), who demonstrated that the embeddings in CLIP do not offer gains in unsupervised clustering, few-shot learning, transfer learning and adversarial robustness as compared to vision-only models. In a subsequent work, Fang et al. (2022) demonstrated that the high robustness of these models rather emerges from the high diversity of data present in their training set. This was achieved by showing that pretraining SimCLR models Chen al. (2020) on larger datasets, such the YFCC dataset by Radford et al. (2021), without any language supervision matches the effective robustness of CLIP. Shi et al. (2023) reinforced this data-centric explanation by showing that the performance on the pretraining set also correlates linearly with the out-of-dist on the importance of data-quality for effective robustness, Nguyen et al. (2022) showed that increasing the protraining set size does not necessarily improve the effective robustness of the resulting model. Rather, it suggests that it is preferable to filter data to keep salient examples, as was done, e.g., to assemble the LAION dataset (Schuhmann et al., 2022).
performance. To put the emphasis
Other indicators of ER. By comparing pretrained models with models trained from scratch, Neyshabur et al. (2020) demonstrated that these models exhibit interesting differences, such as their reliance on high- level statistics of their input features and the fact that they tend to be separated by performance barriers in parameter space. Guillory et al. (2021) found observable model behaviours that are predictive of effective robustness. In particular, the difference of model's average confidence between the in and distr--dist correlates with out-of-distribution performance.
Polysemanticity in foundation models. Polysemantic neurons were coined by Olah al. (2020) in the context vision model interpretability. These neurons get activated in the presence of several unrelated concepts. For instance, the InceptionV1 model has a neurons that fires when either cats or appear in the image. These neurons render the interpretation of the model substantially more complicated, as they prevent lo atlach unambiguous labels to all the neurons in a model. This will limit the insights gained by traditional interpretability techniques. For example, producing saliency maps for a polysemantic neuron could highlight many unrelated parts an image, corresponding to the unrelated concepts this neuron is sensitive to. A qualitative analysis of the neurons in CLIP by Goh et al. (2021) showed that a CLIP ResNet has substantial amount of polysemantic neurons. The emergence of polysemantic neurons is complex phenomenon. It is not yet well-understood for models at scale. The latest works on the subject mostly focus on toy models, see e.g. the works of Elhage et al. (2022) and Scherlis et al. (2022). To the best of our knowledge, our work is the first to explicitly discusses the link that exists between polysemanticity and robustness to natural distribution shifts.
39


Reviews:
[{"review_id": "axPWFPdvB0", "reviewer": "Reviewer_aZBL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The empirical observations provided in this paper are insightful, including:\n  - All robust models have **outlier features**;\n  - **Privileged directions** are crucial to model predictions, but they can be also found in non-robust models;\n  - Robust models have a high number of **unique concepts**.\n- This paper is well written and contextualized.\n\n## Weaknesses\n\n- Some statements were expressed too vaguely. My understanding is that the author empirically showed that \"for all models, being robust $\\to$ having outlier features\" and \"for all models, not being robust $\\to$ not having outlier features\" hold. However, \"there exists a model, not being robust $\\to$ having privileged directions\" is also true. It would be nice if the author could clarify what \"being a signature of robustness\" precisely means.\n- Some claims were not supported by sufficient evidence. For example, the evidence of the takeaway message 3 seems too weak. I'm aware that training large models is computationally costly, but the presented evidence seems insufficient to draw a conclusion. Anyway, the difference between generic/abstract concepts and concrete concepts is subtle and too subjective. Further, I may have missed something, but the connection between the experimental results and the claims in the takeaway message 4 is unclear to me. How is the statement \"high number of concepts stems from language supervision\" supported and verified?\n- The post hoc analysis and insights are valuable, but this paper would be more impactful if the author could discuss more how we can improve the robustness and generalization ability of language-image models based on the given insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigated what makes robust language-image models different from non-robust ones, particularly in handling ImageNet distribution shifts. The author probed the representation spaces of 16 robust CLIP vision encoders with various backbones and pretraining sets, comparing them to less robust models with identical backbones but different (pre)training sets or objectives. This paper provided empirical observations and studied the relationships between robustness and various concepts, such as outlier features, privileged directions, and the number of unique concepts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please further clarify how the takeaway messages are supported by empirical evidence.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GeVZtshFnH", "reviewer": "Reviewer_SLo8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Novel findings: The paper presents novel insights into the relationship between feature characteristics and robustness in CLIP models. \n2. Comprehensive analysis: This study provides a comprehensive analysis of the robustness of CLIP models by utilizing various models.\n3. The paper is well-written and easy to follow. The analysis is presented logically.\n\nWeaknesses:\n1. Limited explanation of outlier features: While the paper identifies outlier features as a key finding, it doesn't delve deep enough into their potential causes or implications. Further investigation into what these outlier features represent and why they emerge in robust models would significantly strengthen the paper.\n2. Limited dataset: The study primarily focuses on ImageNet distribution shifts. Exploring the generalizability of the findings to other datasets would be better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies what differentiates robust models from non-robust ones. The authors analyze a range of CLIP models with varying backbones, pretraining datasets, and robustness levels using different interpretability tools. They identify three key findings:\n1. Outlier features in robust CLIP models: Robust CLIP models exhibit outlier features with significantly higher activations compared to other features. \n2. Outlier features as robustness signatures: The presence of outlier features consistently distinguishes robust models from non-robust ones. This suggests that outlier features can serve as an indicator of robustness.\n3. Concept richness is not solely linked to robustness: While robust CLIP models encode a high number of unique concepts, this characteristic is also observed in non-robust models trained on ImageNet-Captions. This implies that language supervision plays a significant role in enriching visual representations with human concepts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adding an analysis of what these outlier features represent.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XIZjpSoeoX", "reviewer": "Reviewer_ZEJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## **Strengths**\n\n1. The paper is nicely structured and well-written: it studies multiple different concepts, but they are well-organized with sections and in particular take-away boxes, which would help readers/practitioners.\n2. The experiment suite of the paper is extensive and thorough. The authors test 5 backbone architectures (ResNet50, ResNet101, ViT-B/16, ViT-B/32, ViT-L/14) and many different variations (clip pretraining, fine-tuning, pretrained with ImageNet captions). This provides a very thorough picture of different attributes of robustness of all these models. To the best of my knowledge, this is the most comprehensive study of all these models. While lacking novelty, the paper can serve as a nice survey of all these pre-trained models and their robustness attributes to practitioners working on similar domains.\n3. The set of observations, especially that language supervision leads to outlier features and more robust models encode more abstract concepts in comparison to fine-tuned/less robust models, is very interesting. There is potential for future research based on these observations on how to maintain robustness while fine-tuning a pre-trained model for a specific task, and how to get signal for robustness without having a test set, by measuring other attributes such as outlier features.\n\nIn short, **I am happy with accepting the paper with minimal changes**, provided all my questions are answered.\n\n\n## **Weaknesses**\n\n**(Section 3)**\n\nFirst, this way of measuring effective robustness (by comparing the performance of the model to the predicted performance based on accuracy-on-the-line) is a bit alien to me. The reason we call in-distribution performance “in-distribution”, is because this is the distribution that the model has been trained on. If the only way to break out of the linear trend is to use more diverse data, then we should consider performance on the more diverse data as “in-distribution performance”. In other words, if a model is trained on something other than ImageNet (possibly a superset, but even if it is not, the point stands), then performance on ImageNet cannot be considered as “ID performance.” For zero shot models (no fine-tuning), if our training distribution is completely unknown, we can as easily call the natural shift data as “ID” and the original ImageNet data as OOD. What does it say about the effective robustness metric? How do the numbers look in this case? **However, I note that prior work has also done it the same way, and the authors would point to [1], and so this is not a weakness of the current work.**\n\nWhat is a weakness of the current work is mentioning that zero-shot clip models are more robust than their fine-tuned counterparts. This has been studied, both conceptually and empirically, by [2]. They also compare other methods such as linear-probing and LP-FT. Prior work such as [3, 4] also showed effectiveness of tuning particular layers instead of the entire network. I suggest the following experiments:\n\n1. comparison to linear-probing and LP-FT \n2. Add a linear head on top of the visual encoder with C number of classes, and fine-tune only that with the ImageNet images\n\n\n\n## **Questions**\n\n**(Contributions)**\n\n> To the best of our knowledge, this is the first time that outlier features are observed in non-language and non-transformer models.\n\nCould the authors cite papers that have observed this for language or transformer models? This citation is added later, in section 4, but should be added to the introduction as well.\n\n**(Nitpick, activation vectors)**\n\nDoes the image encoder $f_v$ output the activation vector directly? Or does it output an embedding $e$, that passes through an activation function $\\sigma$, to get the activation vector? This does not make much difference to the analysis, I am just curious about the notations.\n\n# References\n\nReferences\n\n[1] Measuring Robustness to Natural Distribution Shifts in Image Classification, https://arxiv.org/abs/2007.00644\n\n[2] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, https://arxiv.org/abs/2202.10054\n\n[3] Surgical Fine-Tuning Improves Adaptation to Distribution Shifts, https://arxiv.org/abs/2210.11466\n\n[4] Less is More: Selective Layer Finetuning with SubTuning, https://arxiv.org/abs/2302.06354", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Robustness is a key property for machine learning models to be useful in practice. In short, it means that a desirable property in machine learning models is that when they see slightly different data compared to what they have been trained on, they should not lose too much of their performance on the original dataset. \n\nImageNet is a popular benchmark in image classification, with many natural shifts/slightly different variants of it. This gives us the opportunity to test if a model maintains its original performance on this slightly different variants, thereby measuring its robustness. From prior work, we have a large suite of models that work/do not work on this task. This paper asks the question: what is different between models that are robust and non-robust in this task?\n\nThe authors run a rigorous study of many different models and show the following:\n\n1. **Robustness of clip models**: Pretrained clip models used in a zero-shot way are more robust than their fine-tuned counterparts. \n2. **Outlier features**: features that have a significantly high magnitude compared to the average feature magnitude. Robust models have this more than non-robust models.\n3. **Privileged direction**: Robust models have strong privileged direction compared to less-robust counterparts.\n4. **Concept encoded**: Robust models encode more abstract concepts than non-robust ones, whose top-3 encoded concepts are more granular/fine-grained. Robust models also encode more concepts than their non-robust counterparts.\n\nThe paper uses prior methods to analyze these differences between robust and non-robust methods. While there is no causal relationship established (e.g., robustness implies existence of outlier features and vice versa), these observations can serve as easy to check conditions to estimate a model’s robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer the weaknesses/questions mentioned above.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As I mentioned above, some of the conclusions in the paper are not presented very clearly, and may be lacking evidence to support them in the current version. I believe the paper should be accepted, but I request that the authors update the wording in the paper, especially the take-away boxes to clarify that (1) no causal conclusions can be made and (2) to incorporate the suggestions of Reviewer aZBL for takeaway box 4.
Decision: Accept (poster)