Do better ImageNet classifiers assess perceptual similarity better?
Authors: Manoj Kumar
Neil Houlsby
Nal Kalchbrenner
Ekin Dogus Cubuk
Venue: Accepted by TMLR ()

Abstract: Perceptual distances between images, as measured in the space of pre-trained deep features, have outperformed prior low-level, pixel-based metrics on assessing image similarity. While the capabilities of older and less accurate models such as AlexNet and VGG to capture perceptual similarity are well known, modern and more accurate models are less studied. In this paper, we present a large-scale empirical study to assess how well ImageNet classifiers perform on perceptual similarity. First, we observe a inverse correlation between ImageNet accuracy and Perceptual Scores of modern networks such as ResNets, EfficientNets, and Vision Transformers: that is better classifiers achieve worse Perceptual Scores. Then, we examine the ImageNet accuracy/Perceptual Score relationship on varying the depth, width, number of training steps, weight decay, label smoothing, and dropout. Higher accuracy improves Perceptual Score up to a certain point, but we uncover a Pareto frontier between accuracies and Perceptual Score in the mid-to-high accuracy regime. We explore this relationship further using a number of plausible hypotheses such as distortion invariance, spatial frequency sensitivity, and alternative perceptual functions. Interestingly we discover shallow ResNets and ResNets trained for less than 5 epochs only on ImageNet, whose emergent Perceptual Score matches the prior best networks trained directly on supervised human perceptual judgements.

Published in Transactions on Machine Learning Research (09/2022)
Do better ImageNet classifiers assess perceptual similarity better?
Manoj Kumar
mechcoder@google.com
Neil Houlsby
neilhoulsh@@google.cm
Nal Kalchbrenner
nalk malk@google.com
Ekin D. Cubuk
cubukligoogle.om com
Google Research, Brain Team
Reviewed on OpenReview: https: // openreview. net/ forum? id= grCKGZZuHO
Abstract
Perceptual distances between images, as measured in the space pre-trained deep features, have outperformed prior low-level, pixel-based metrics assessing perceptual similarity. While the capabilities of older and less accurate models such AlexNet and VGG to capture perceptual similarity are well known, modern and more accurate models are less studied. In this paper, we present large-scale empirical study to assess how well Ima- geNet classifiers perform on perceptual similarity. First, we observe a inverse correlation between ImageNet accuracy and Perceptual Scores of modern networks such as ResNets, Efficient.Nets, and Vision Transformers: that better classifiers achieve worse Perceptual Scores. Then, examine the ImageNet accuracy Cecuracy/Perceppee PPerceppul Score relationship on varying the depth, width, number of training steps, weight decay, label smoothing, and dropout. Higher accuracy improves Perceptual Score up to a certain point, but we uncover Pareto frontier between accuracies and Perceptual Score in the mid-to-high accuracy regime. We explore this relationship further using a number of plausible hypotheses such as distortion invariance, spatial frequency sensitivity, and alternative perceptual functions. Interestingly we discover shallow ResNets and ResNets trained for less than 5 epochs only on ImageNet, whose emergent Perceptual Score matches the prior best networks trained directly on su- pervised human perceptual judgements.
Introduction
ImageNet (Russakovsky et al. 2015 is the cornerstone of modern supervised learning and has enabled significant progress in computer vision. Features learnt via training on ImageNet transfer well to a number of downstream tasks (Carreira & Zisserman, 2017 Zhai al. 2019 Huang al. 2017), making ImageNet pretraining standard recipe. Further, better accuracy on Image usually implics better performance diverse set of downstream tasks such robustness to common corruptions OOhan 2019: Xie et al.
2020b Taori et al. 2020 Radford et al.) 2202]), adversarial robustness (Cubuk et al.
2017 Xie et al.) 2020a
out-of-distribution generalization (Recht et al. 2019 Andreassen et al. 2021 Miller et al. 2021 ) transfer
learning on smaller classification datasets (Kornblith et al.] 2219)), pose estimation (Mathis et al. 2021
domain adaptation (Zhang & Davison 2020), object detection and segmentation (Zoph et al. 2020) and for
predicting neural recordings and behaviors primates on object recognition tasks (Schrimpf et al. 2020
As remarkable side effect, ImageNet models can also capture a notion of similarity identical humans, known perceptual similarity. Designing distance metrics that correspond to human judgements well

---

Published in Transactions on Machine Learning Research (09/2022)
established problem in computer vision, and number of low-level metrics (Zhang et al. 2011 Wang et al, 2004 Mantiuk et al. 2011) have been introduced for this purpose. The first generation of ImageNer Net classifiers:
Alex (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014), and SqueezeNet (landola et al,
2016) can all measure perceptual similarity termed as Perceptual Scores (PS), as an emergent property, in a way that outperforms all prior pixel-level metrics and correlates better with human judgement (Zhang et al. 2018).
In this paper, we are motivated by the following questions: Considering ImageNet classification has pro- gressed significantly since then, fier directly? Since modern neural network training involves a large mumber of hyperparamcters, are there design choices that can improve a classifier's perceptual similarity? Are there latent factors that govern the relationship between ImageNet accuracy and perceptual similarity?
obtain better perceptual similarity metric by using better classi-
We perform a suite of experiments on BAPPS, a large dataset of human-cvaluated perceptual judgements (Zhang et al.] 2018). To the best of our knowledge, our work is the first empirical study to present system- atic investigation and rigorous deep dive into the relationship between ImageNet accuracy and perceptual similarity. We study the ImageNet accuracy/PS. interplay of wide variety of networks across many combi- nations of architectures and hyperparameters. Given the increasing interest in analyzing how representations of ImageNet classifiers transfer to other domains, our work adds another direction to this literature.
Fig. displays the ImageNet accuracy and PS every ImageNet classifier in our study. While Zhang et al. (2018), show a positive correlation between PS and ImageNet accuracy, we observe that this holds only in the low-accuracy regime. In the mid-to-high accuracy regime, we uncover counter-intuitive Pareto frontier between accuracy and PS. Modern networks, EfficientNets, Vision Transformers and ResNets, lic to the right, obtaining high accuracies and low PS. Contrary to prevailing evidence that suggests models with high validation accuracies on ImageNet are likely to transfer better to other domains, we find that representations from underfit ImageNet models with modest validation accuracies achieve the best PS. Our experiments further suggest that attaining the best PS is somewhat architecture agnostic. For example, ResNets trained with large amounts of weight decay or early stopped within a few epochs of training can match or outperform the PS of AlexNet and VGG.
Finally, investigate if there are latent factors that govern the relationship between ImageNet accuracy and PS using the following hypotheses. Does the inverse- persist with global perceptual functions? Are low PS models less sensitive to distortions? What are the contributions of skip connections in modern architectures to decreased PS, any? Do lower layers of better classifiers have a higher PS than higher layers? What is the impact of ImageNet class granularity on PS? While the causatory latent factor that governs the relationship between accuracy and perceptual similarity remains unclear, our paper opens the door to further understanding of this phenomenon.
A summary of our experiments are:
We systematically evaluate the PS of modern "out-of-the-box" networks, ResNets, Efficient and Vision Transformers.
We study the variation of PS (and accuracy) function of width, depth, number of training steps, weight decay, label smoothing and dropout. Our large-scale study consists of 722 different ImageNet networks across the cross product of these 7 different hyperparameters and 5 architectures.
We explore the relationship between ImageNet accuracy and PS further using spatial frequency sensitivity, invariance to distortions, class granularity, and improved global perceptual functions.
Our empirical study leads to the following surprising results:
While modern classifiers outperform prior pixel-based metrics in PS, they under-perform moderate classifiers like AlexNet.
2

---

Published in Transactions on Machine Learning Research (09/2022)
Perceptual Score
69
67
66
10
20
30
40
50
60
70
Validation Accuracy
Figure 1: We discover a Pareto frontier (marked in dark blue) between Perceptual Scores (Zhang et al. 2018 on the 64 x 64 BAPPS Dataset (y-axis) and ImageNet 64 64 validation accuracies (x-axis). Fach blue dot represents an ImageNet classifier. Better ImageNet classifiers achieve better Perceptual Scores up certain point. Beyond this point, improving on accuracy hurts Perceptual Score. Modern networks, EfficientNets (A), Vision Transformers (A) and ResNets lie far to the right of the point of optimal Perceptual Score, achieving high accuracy and lower Perceptual Scores. The best Perceptual Scores are attained by classifiers with moderate accuracy (20.0-40.0). The lowest and highest Perceptual Score observed in prior networks are 64.3 (using random weights, i.e. an untrained network) and 68.9 (AlexNet) respectively.
Modern ImageNet models that are much shallower, narrower, early-stopped within a few epochs of training and trained with larger values of weight decay attain significantly higher PS than their out-of-the-box counterparts.
In all of our hyperparameter sweeps with the exception of label smoothing and dropout, we discover an unexpected and previously unobserved tradeoff between accuracy and PS. In each hyperparameter sweep, there exists an optimal accuracy up to which improving accuracy improves PS. This optimum fairly low and attained quite early in the hyperparameter sweep. Beyond this point, better classifiers achieve worse PS.
Perceptual functions that rely on global image reprosentations such style achieve better PS than per-pixel perceptual functions.
We do not find a correlation between PS of models and their sensitivity to distortions. Further, low Perceptuall Score models are not necessarily more reliant on high-frequency spatial information for classification as compared to high PS models.
High-level features found in the latter layers of residual networks, have better PS than low-level features found in the earlier layers.
Finally, we find particularly shallow, early-stopped ResNets trained only ImageNet that altain an emergent Perceptual Score of 70.2. This matches the best Perceptual Scores across prior networks which were trained directly on BAPPS to match human judgements.
2
Related Work
There has been rich body of work dedicated to analyzing the transfer of pretrained ImageNet networks to various downstream tasks. Kornblith et al. (2019) show that transfer learning performance. is highly correlated
with ImageNet top-1 accuracy. Taori et al. (2020): Miller et al. (2021) show that improvements in accuracy on
ImageNet consistently results in improvementts on test datasets with distribution shift. Djolonga et al. (2021) show a positive correlation between ImageNet transfer performance and out-of-distrins distribution robustness. While Geirhos et al.) (2018) demonstrate that ImagoNet-traincd models have a stronger bias towards using texture cues compared to humans, Hermann al. (2019) show that among high- performief models, shape-bias is
correlated with ImageNet accuracy. [Schrimpf et. al. (0200) found that CNNs that achieve higher accuracy
3

---

Published in Transactions on Machine Learning Research (09/2022)
(a)
(b)
(c)
Figure Three sample triplets from the BAPPS Dataset corresponding to the traditional (left), super- resolution (center) and color (right) distortion families. Each human rater provides a binary label, that indicates which of the two patches closer to the center patch, that is label 0 if the left patch is closer. The mean rating for these three triplets across 5 raters 0.0, 1.0 and 0.8, respectively.
on ImageNet are better at predicting neural recordings of primates when executing object recognition tasks, but the positive correlation was weaker for high-accuracy models. While better ImageNet classifiers transfer better on the above tasks, we are the first to observe a negative correlation between ImageNet classifiers and their inherent ability to capture image similarity, While we do expect that larger models with more capacity can attain better PS when trained directly on BAPPS, we use BAPPS solcly evaluate the emergent perceptual properties of ImageNet-brained classifiers.
An orthogonal line of work, involves incorporating "perceptual losses" that minimize the distance between intermediate features of a ground-truth and generated image to drive photorcalistic synthesis in style transfer
Galys et al. 2015 Li et al. 2017). super-resolution (Johnson et. al. 2016), and conditional image sy synthesis
Dosovitskiy & Brox 2016). Chen & Koltun 2017). Following the investigation done in (Zhang et al.
2018
state-of-the-art models in image-synthesis domains such as super-resolution (Lugmayr et al., 2020
2021
Bhat et al.
2021), image inpainting (Nazeri al. 2019). [Zhao al. 2021), high-resolution image synthesis
Easer et al. 2021), (Karras et al. 2019), (Karras et al., 2020), deblurring (Kupyn al.) 2019) ZZhang
et al. 2020 image-to-imase
translation (Richardson et al.) 2021 (Lee et al. [0188), video generation [Babacizadeh et al. 2021). (Franceschi et | (2020). Ww et al. 2021), (Lee et al. 2018a). neural radiance
feels (Martin-Brunl Brualla et al. 2021), (Mildenhall et al. 2020). view synthesis (Wang et al. 2021b), (Riegler & Koltun 2020), (Tucker & Snavely 2020). (Wiles et al. 2020), and others [Lai ct al. 2018 (Liu et al. 2021))
use perceptual distance as either an auxiliary training loss to improve image synt hesis or as an evaluation metric assess synthesized image quality. While there has been some anecdotal evidence regarding the greater efficacy of VGG features over ResNet-50 for style losses (Nakano. 2019), there has been systematic investigation assessing the relationship between ImageNet accuracy and perceptual similarity. In this paper, we exclusively focus uncovering the relationship between the accuracy of ImageNet models and their inherent ability to capture perceptual similarity and not on applications of perceptual losses to downstream tasks.
3 Background
3.1
The BAPPS Dataset
The BAPPS Dataset (Zhang et al. level distortions the Alobe 5k dataset (Bychkovsky al. 2011]h for training and the RAISEIK dataset (Dang-Nguyen al. 2015) for validation. (Zhang et al.] 2018) consider 6 distortion familics namely:
2018) is a datasel of 161k patches derived by applying exclusively low-
Traditional Distortions: Random noise, blurring, spatial shifts, corruptions and compression arti- facts. (1 family.)
CNN-basod Distortions: Distortions created by CNN-based autocncoders trained on autocncoding, denoising, colorization and superresolution. (1 family.)
Outputs of Real algorithms: Outputs from stato-of-the-art frame interpolation, video deblurring, colorization and superresolution models. The distortions created by each class of models is treated separate family. (4 families.)
4

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2 in Zhang et al 20188 contains comprehensive list of distortions. The train set consists of the traditional and CNN-based distortions and the validation set contains all 6 families. Given family of distortions and a set of reference images, ZZhang et al. (2018) generate the BAPPS dataset as follows. They select a reference patch I and then apply two distortions at random to generate the target patches 2o and #1- They record the binary response of a human, indicating which of the larget palches closer to the reference patch. For given image triplet, (r0,411 p the average of and human responses on the train and validation set respectively. Fig. 2 displays 3 sample image triplets from the BAPPS Dataset.
3.2 Perceptual Score
We first define the PS of network, for which we adopt the "2AFC" scoring protocol (Zhang al. 2018 First, let and 21 denote two images. Let 50' and 5ú be the feature maps for Zo and at the 7th layer of a network, normalized across the channel dimension. The perceptual similarity function d(zo, 21) ) is defined as:
d( x1)
Σ IEC
Ellwoh. vo12
(1)
H.W,
where Hi and Wi denote the height and width of the feature maps at layer 1, respectively. denotes the subset of layers which used in the perceptual similarity; this subset is architecture specific. Given reference image and two target images zo and 21, BAPPS (Zhang et al. 2018 provides ground truth soft-label p. can be interpreted as the probability a human rater would rale xı more similar to than 20.
For a noural network with distances do = xo) and d1 = d(x, x1), from Zhang ct al. 2018 we define the
PS s(do, di) to be the following value times 100:
plo > di] + (1- > dol 0 di]
(2)
Dynamic Range of PS. Unlike accuracy that can vary from 0 to 100, PS has narrow dynamic range. To provide intuition on this dynamic range, we plot the PS for ground truth ratings p from BAPPS and simulated combinations of (do, d1) de- pendent on p. Let ğ = 1p > 0.5), i.e we convert the human ratings into binary labels. If p 1, we sample do and d, from truncated normal distribu- tions, ie do o(1, 0, 1), d1 6(0, a, O, 1) where and are the lower and upper bounds. Conversely if ğ ==0 O, we sample do ~ @00, a, 0, 1)  o(1, a, 0, 1). In Fig. 3 a models the noise in predicting d, as a is increased, the distances (do, d,) have a higher chance of being misaligned with p, and as expected PS smoothly decreases from the upper bound (~ 0.8) to random choice (0.5).
80
Perceptual Score
75
65
Tamen Deand Medid:
60
0.4
0.6
1.0
The blue line depicts PS on simulated dis- tances and real labels from BAPPS. models the noise in predicting the simulated distances from the real labels. The green and red horizontal lines denote the best (AlexNet) and worst PS (untrained) obtained with ImageNet networks. The orange line denotes the best PS obtained with low-level metrics (FSIMc). Farly-stopped Res Nets obtain the best PS score this study.
Figure 3:
On the actual BAPPS datasct, the previous best PS for low-level metrics was attained by FSIMc (Zhang et al. 2011), a low-level hand crafted matching func- tion with a value of 63.8. The previously reported lower and upper PS bounds for ImageNet networks (Zhang al. 2018 values of 64.3 and 68.9.
5

---

Published in Transactions on Machine Learning Research (09/2022)
69.0
Perceptual Score
68.5
68.0
67.5
depth
67.0
ResNet
Efft cientNet
66.5
VIT
10
20
30
40
50
60
70
Accuracy
Figure 4: Perceptual Scores (Zhang al. 2018) on the 64 64 BAPPS dataset as function of ImageNet 64 64 validation accuracics. Each point is the average across runs. Error bars on both the axes the corresponding standard errors (variance in accuracy is sufficiently small that error bars in the x-direction are barely visible). Across the out-of-the-box ResNets, Efficient.Nets and ViTs (circles), there exists negative correlation between Perceptual Score and accuracy. On reducing the depth even further, (ResNet-reduuced depth, triangles) we uncover an "optimal accuracy threshold". Below this threshold, Perceptual Scores correlate with accuracy and above Perceptual Scores decrease. Among the architectures in this plot, ResNet-6 attains this optimal accuracy threshold and has the highest Perceptual Score.
These were achieved by a randomly initialized net- work and AlexNet (Krizhevsky et al., 2012) respectively. VGG obtains a PS of 67.0. The human upper bound is 73.9, reflective of the entropy across human raters. Across all our experiments, we cover almost the entire prior dynamic PS range of trained networks, with the PS of our networks ranging from 65.0 to 69.7.
4 Experimental Setup
Architectures. We study two of the most popular classes of vision architectures: Convolutional Neural Networks (CNNs) and Transformers. As representative CNNs, we train ResNets (He et al., 2016) and Ef- ficientNets (Tan & Le 2019). For EfficientNets, we use model variants B0 to B5 (Tan & Le 2019). For ResNets, we train networks with depths ranging from 6 to 200 layers. For Vision Transformers (ViT) (Dsoo vitskiy et al. 2020
the Base and Large variants, with patch sizes of 8 and 4, leading to four models (ViT-B/8, VIT-B/4, ViT-L/88 and VIT-I./4). We use smaller patch sizes than is common because use lower resolution images (see the following Training section). However, any patch size smaller than 4 leads to poor generalization.
Training. We train our networks on ImageNet at a resolution of 64 x 64 and report their accuracies on the ImageNet validation set and PS on the BAPPS validation set. ImageNet 64 x 64 provides a cleaner test bed instead the standard high-resolution (224 x 224 and above) for the following reasons.
BAPPS is constructed using 64 x images so that human raters can focus on low-level local changes as opposed to high-level semantic differences. ImageNet models pre-trained high-resolution images classify 64 x 64 images poorly. Such models are sub-optimal for analysis, because we interested in the perceptual properties of classifiers that generalize reasonably well. An alternative option would be to resize 64 x 64 images to high-resolution images using linear interpolations. However, this can have the adverse side effect of blurring out or removing distortions from BAPPS images. Nonetheless, in Appendix A we observe similar phenomena on models trained with high resolution images, with significantly worse PS.
For all networks, we apply only random crops and flips and disable all other augmentations. We use the recommended hyperparameter settings given by the corresponding open-sourced training code. For the

---

Published in Transactions on Machine Learning Research (09/2022)
models where validation accuracy decreases during training, Afffeet-B4 and EffNettB5, we stop training just before this happens.
Representations. In prior work, Zhang et al. (2018) compute the PS of VGG using the outputs of every 2x2 max pooling layer, leading to 5 leatures in total. For the shallower AlexNet architeclure, they employed the output of each of the 5 convolutions. Here, we describe the representations we use to compute the PS of modern networks, in this work.
Similar to VGG, for ResNets and EfficientNets, we use the outputs of the reduction stages, where the spatial dimensions are reduced 2x via strided convolutions. Specifically, we obtain four 2D representations of resolution 16 16, 4 4, and 2 x 2.
For Vision Transformers, we use the global CLS representation of the image at the output of every encoder block. Our initial experiments on using features at the output of every layer instead of every block or reduction stage, as computed in AlexNet lead to worse PS across all architectures.
5 ImageNet accuracy versus Perceptual Score
We train networks with their default hyperparameters five times and report the mean accuracy and PS (64 x 64) with error bars in Fig. 4 (Appendix M contains a list of default hyperparamcters). All modern networks (red, green, and black) in Fig. A obtain a higher PS than FSIMc (63.8) and randomly initialized networks (64.3). Surprisingly though, representations of better ImageNet networks perform consistently worse. For example, ResNet-18 which achieves the best PS of 68.5 has a modest accuracy of 52.0% (64 x 64). EfficientNet B5 which has the worst PS of 67.2 achieves a much higher accuracy of 66.3%. Additionally, all networks perform worse than AlexNet (68.9).
Next, we push this observation to the limits. Our next experiment hopes to identify much smaller network that achieves improved PS at the expense of extremely low accuracy. In Fig. 4 we report the accuracies and PS obtained by shallow ResNets with depths from 2 to 10 (ResNet, reduced depth). ResNet-10 and ResNet-6 obtaín improved PS with reduced accuracies. However, the ResNets with a depth smallor than 6 incur losses in both accuracy and PS. Our results indicate that better classifiers produce better perceptual representations up until a certain "optimal accuracy". Above this optimal accuracy, ImageNet classifiers trade off better accuracies with worse PS. This phenomenon leads to some interesting observations: 1) ResNet-6 happens to achieve this optimal accuracy with a PS of 69.1, outperforming AlexNet. 2) Beyond the optimal accuracy, there is a strong inverse correlation between accuracy and PS (coelficient -0.84). 3) ResNet-200 and ResNet-3 achieve similar PS, while having a an accuracy difference of 45%.
6 How general is this relationship?
Section 5 indicates that an inverse-U relationship exists between accuracy and PS on varying the depth of residual networks. In this section, we attempt to generalize this relationship as a function of other implicit hyperparameters such as layer composition, width, depth and training hyperparameters of the considered architectures. In particular, we assess this relationship between accuracy and PS as a function of hyperparameters via 1D sweeps. We vary a single hyperparameter of a network (e-g- width) along a ID grid which changes the network's accuracy and as an effect, the corresponding PS. In Appendix H for all our sweeps, the validation accuracy does not decrease during training, indicating none of the networks overfit the ImageNet train set. For a given sweep, provide two sets of plots: 1) Scatter plots between PS/ PS/accuracy. 2) PS against the hyperparameter values. We choose 5 architectures: ResNets and ViTs with the best and worst PS, (ResNet-6, ResNet-200, ViT-B/8, VIT-L/4) plus a standard NesNet-50.
We sweep across the following hyperparamters: Number of training steps, network width, network depth, weight decay (Hanson Pratt 1988), dropout (Srivastava et al. 2014), and label smoothing (Szegedy et al. 2016b). We define Pmaz be the optimal PS and a(pmaz) to be the accuracy at which Pmax reached. We observe the inverse-U relationship across all these hyperparameters with the exception of label smoothing
7

---

Published in Transactions on Machine Learning Research (09/2022)
and dropout. For these hyperparameter settings, we indicate Pmoz and a(Pmaz) with dotted lines parallel to the x and axes, respectively.
Number of train epochs. All networks exhibit the inverse- U shaped behaviour (Fig. 5a and 5c]. a(pmax) is fairly consistent within a given family of architectures, ResNets have a(Pmax) low values between 15-25 % and ViTs have a(Pmax) at moderate values between 40-50 %.
70.0
Res
ResNet
69.0
VIT
VIT
65.5
69.5
68.5
66.5
69.0
69.0
$0.0
08.0
68.5
68.5
42.5
68.0
67.0
67.0
67.0
56.0
10
29
30
$6.0
50
150
250 360
Accurecy
Number Cocchi
Accuraty
Number of Eoochs
(a)
(c)
(d)
Figure 5: Figs. 5a and 5c show PS vs accuracy on varying train epochs in ResNets and ViIs. Figs.
5b and
5d depict PS as a function of train epochs.
PS peaks early during training across all networks (Fig. 5d and 5b4. ResNet-50 and ResNet-200 peak at the first few epochs of training (Pmaz = 69.7) while the VIT models peak at lower values (Pmer 68.4) and later around 60 epochs. After the peak, PS of better classifiers decrease more drastically. ResNets are trained with a learning rate schedule that causes a step-wise increase in accuracy as function of training steps. Interestingly, in Fig. 5b they exhibit step-wisc decrease in PS that matches this step-wise accuracy increase.
Width and Depth. In Fig. 6a we see that ResNet-6 achieves Pmaz = 69.6 at a(pmax) = 20% while ResNet-200 has Pmaz 67.6 a(Pmaz) 65%. As the model capacity is increased from ResNet-6 ResNet-200, the peak shifts from the top left to the bottom right. Compound scaling is model scaling technique (Tan & Le. 2019: Szegedy et al. 2016a) that improves model accuracy efficiently by scaling the width and depth of network together. We observe peculiar "inverse compound scaling" phenomena; depth and width of networks have to be scaled down simultaneously to improve on PS significantly.
Restiet
ResNet
VIT
VIT
70.0
70.0
19.0
$9.0
63.5
678.5
58.5
58.5
65.0-
68.5
65.0
07.5
67.5
56.3
66.5
67.0
30
67.0
0.5
1.0
2.5
2.0
255
16.0
10
20
30
40
50
60
$6.0
04 L19
255
512
Accurry
With plaer
Accumcy
With
(a)
(b)
(c)
(d)
VIT
VIT
09.0
00.5
68.5
68.0
68.0
67.5
67.0
50.3
665
66.0
14 16 18 Douth
Accur ocy
(c)
(f)
Figure 6: Figs.
6a
0c
and
6c
show PS vs accuracy on varying width in ResNets, width in ViTs and depth
in ViTs. Figs.
6b
6d and depict PS as a function width in ResNets, width in ViTs and depth in ViTs.
8

---

Published in Transactions on Machine Learning Research (09/2022)
a(Pmaz) varying the width and depth of ViTs are at 20% to 30% and 45-50% respectively. (Figs. 6e
Gel. A shallow ViT model of depth 2 gets close to 40% accuracy. Hence, there might just not be enough points between 20% to 40% in Fig, 6e This could explain the shift of a(pmar) to the right in Fig. 6e when
compared to Fig. 6c
Shallower and narrower architectures perform better as shown in Figs. 6d
and 6D
The optimal width of
V/T-B/8 and VIT-L/4 are and 12% of their default widths while their optimal depths are just 2 transformer blocks. ResNet-6 and ResNet-50 exhibit similar properties with the optimal width being 25% of their original widths. ResNet-200 is the outlier with small peak at its original width.
Central Crop. Modern networks employ random crops of high-resolution rectangular images during train-
ing. This artificially increases the effective quantity of training data available the network. Replacing random crops with central crops has been shown increase shape bias (Hermann al, 2019 and reduce the discrepancy of object scales between training and testing (Touvron et al. 2019
Reshet: Center Crop
VIT: Center Crop
68.5
68.5
B15
65.0
68.0
68.5
67.5
B15
98.0
67.0
230
67.0
10
20
10
60
50
60
70
05.2
10
20
35
40
50
50
70
Accurecy
Accurwyy
(a)
(b)
Figure 7: Figs. and ViTs.
7 and
7b
show PS vs accuracy when random crop is replaced with central crop in ResNets
Fig. 6 shows the accuracies and PS of the 5 architectures trained with center crops. Each architecture moves towards the top-left with lower accuracies and higher PS. ViT-L/4 is the exception as it moves towards the bottom-right. It encounters a significant. reduction in accuracy, lowering it below a(Pmaz), which could explain the decrease its PS. All center-cropped architectures lie along an inverse-U, with ResNet-6 at the optimum.
Weight Decay, Figs. Sc and 8c display the impact of weight decay PS. VIT-L/4 has minimal variation in accuracy as the weight decay factor is varied; so we omit ViT-L//4 from this study. ResNets and ViTs achieve their worst PS at the default weight decay around 10-4 and 0.3, respectively. The PS increases on either side of this optimum. This correlates with their changes in accuracy as a function of weight decay. ResNets and ViTs achieve their best accuracies at these weight decay values and decrease either direction.
ResNet
ResNet
VIT
65.5
Peroeptout Scate
08.5
58.
69.0
09.0
sa.
68.0
68.5
68.5
47.5
67.5
68.0
62.0
67.0
10
13
10
10
10.4
10:4
66.0
30
66.0
10
10
10F
Accurecy
Wuign
Accuway
wiglk
(a)
(b)
(c)
(d)
Figure 8:
Figs Sa and show PS vs accuracy on varying weight decay in ResNets and VITs. Figs
Sc
and
8d depict PS as function of weight decay.
Label Smoothing and Dropout. Across all our controlled settings, label smoothing and dropout are the only hyperparameters that decrease both PS and accuracy.
Varying label smoothing produces less drastic changes in accuracy as compared to other hyperparameters. Fig.
Dal very weak positive correlation exists between accuracy and PS for ResNet-50 and ResNeet-20.. Tn

---

Published in Transactions on Machine Learning Research (09/2022)
ResNet
ResNet
VIT
VIT
70.0
70.0
$9.0
$5.0
69.5
$0.5
68.5
67.5
68.0
67.0
67.0
67.3
56.3
26.3
67.0
10
20
30
60
0.1
0.2
0.3
04
0.5
0.7 08 0.
06.0
10
20
30
40
50
60
10
0.1
02.
0.3
05
06
0.5 09
(8)
(b)
(c)
(d)
ResNet
RecNot
VIT
VIT
70.0
70.0
$9.0
55.0
69.5
09.0
24
68.0
60.5
$7.0
$7.0
67.5
66.5
66.5
0.1 0.2
03
04 0.5 0.5
D.
05
66.0
10
20
30
40
50
60
70
01
02
0.3
04
05 D6 07 05 69
Droport
Accurccy
(c)
(g)
(h)
Figure 9: Figs Da 9c
Dg show PS vs accuracy on varying label smoothing ResNets, label smoothing and 9h are the same graphs with PS as
in ViTs, dropout in Res Nets and dropout in ViTs. Figs function of the corresponding hyperparameters.
9b
9d
9
Fig. 9b PS of ResNet-50 and ResNet 200 decrease with more label smoothing, while the Vision Transformers and ResNet-6 are almost invariant. Our results indicate that clean labels are necessary to obtain high PS. However, varying label smoothing does not change accuracy a great deal within each architecture class, the dynamic range is insufficient to observe the inverse-U U relationship.
In Figs. and 9h the PS consistently decreases a function of dropout. In Fig. 9e and Fig. 9g it also correlates with accuracies. Curiously, dropout is the only factor to negatively influence both accuracy and PS simultancously.
Conclusion. In Fig. I we plot the accuracy and PS from all our above experiments. While their exact relationship architecture and hyperparameter dependent, we uncover a global Pareto frontier between PS and accuracies, see Fig. Up to certain peak, better classifiers achieve PS and beyond this peak, better accuracy hurts PS.
7 Scaling down improves Perceptual Scores
Table 1: Perceptual Score improves by scaling down ImageNett models. Each value denotes the improvement obtained by scaling down a model across a given hyperparameter over the model with default hyperparam- eters.
Model
Default
Width
Depth
Weight Decay
Central Crop
Train Steps
Best
ResNet-6
69.1
+0.4
+0.3
0.0
+0.5
69.6
ResNet-50
68.2
+0.4
+0.7
+0.7
+1.5
69.7
ResNet-200
67.6
+0.2
+1.3
+1.2
+1.9
69.5
ViT B/8
67.6
+1.1
+1.0
+1.3
+0.9
+1.1
68.9
ViT L/4
67.9
+0.4
+0.4
-0.1
-1.1
+0.5
68.4
Our results in Sention prescribe a simple strategy to make an architecture's PS better: Scale down the model to reduce its accuracy till a(Pmaz).
Table summarizes the improvements in PS obtained by scaling down each model across every hyperparam- eter. With the exception of ViT-I,/4, across all architectures, early stopping yields the highest improvement
10

---

Published in Transactions on Machine Learning Research (09/2022)
in PS. In addition, early stopping is the most efficient strategy there is no need for expensive grid search.
70
70
Reference Widh Traning Epochs
. Reference
Width
Depth
get Decay Cencral Crop
Training Epocha
50
Accuracy
50
Weight Docay
Center Crop
40
40
20
20
67.5
68.0
68.5
69.0
69.5
67.5
68.0
68.5
69.0
69.5
Perceptual Score
Perceptual Score
(a) ResNett200
(b) VIT R/8
Figure 10: Each curve in a subplot showcases the validation accuracy vs Perceptual Score Pareto frontier for that hetrtturr/hyperpaammttte combination. Each point on a line denotes the minimum accuracy loss achievable for a particular PS gain. The dashed gray line the reference Pareto frontier obtained with out-of-the-box architectures.
How much does PS improvement cost terms of accuracy? Fig. sh the accuracy-PS Pareto frontier for ResNet-200 and ViT-B/8. Each point the Pareto frontier denotes the maximum possible achievable accuracy for given PS. The gray line is the reference Pareto-frontier obtained from training networks with their default settings. Except for Width + ViT-B/8 that lies below the reference Pareto frontier, all curves lie very close to the reference Pareto frontier. Early-stopping any of the architectures to improve their PS, as seen in Table I greatly reduces their accuracy.
8 Further Exploration: Reasons behind the inverse-U relationship
8.1 The inverse-U phenomenon persists with improved perceptual similarity functions
We first posit that the perceptual similarity function is suboptimal, and alternative would not yield inverse-U relationship.
The perceptual similarity function in (Zang et al. 2018 averages per-pixel differences across the spatial dimensions of the image. This assumes a direct correspondence between pixels, which may not hold for warped, translated rotated images. For a similarity function that compares global representations of images, the inverse-U relationship may no longer exist. We investigate two such functions in two different settings: 1) Out-of-the-box ResNets and Efficient.Nts.. 2) ResNet-200 function of train steps.
Style. We adopt the style-loss function from the Neural Style Transfer work of (Gatys et al. 2015). Let % and o E Wixxx be the spatially normalized outputs two images To and ø at the Ith layer of the network. We compute G/ e RCixC, the inter-channel cross-correlation matrix of % and similarly G for Ø Go and G1 also known as Gram matrices capture global information in io and z/ respectivcly. The style function given by Σ, EIG G
Mean Pool. Let vé and ví E Xxx be the unnormalized outputs at the /th layer of the network. We spatially average them to global representations and then normalize across channels to obtain Th and A E R The distance function is then given by Σ C 1 - 1
In Figs. Ia and l1b we present scatter plots between accuracy and PS with the style and mean pool similarity functions. Fig. [aa displays the accuracy and PS of ResNets and EfficientNets trained with their default hyperparameters. Each point in Fig. Ilb represents ResNet-200 model at different epoch during the course of training.
11

---

Published in Transactions on Machine Learning Research (09/2022)
70.0
70.0
Local Mean Pool Style
Local Style
Perceptual Score
69.5
Perceptual Score
69.5
69.0
69.0
68.5
68.5
68.0-
68.0
67.5
67.5
67.0
67.0
10
20
30
40
50
60
70
10
20
30
40
50
60
70
Accuracy
Accuracy
(a)
(b)
Figure 11: Relationship between accuracy and PS via two alternate perceptual functions: Mean Pool and Style. Fig. 11a displays the accuracy and PS of ResNets and EfficientNees trained with their default hyperparameters. Each point in Fig. 1lb represents ResNet-200 model at different epoch during the course of training.
Both functions yield better PS than the bascline ("Local") (Zhang al. 2018). In Fig. 11a ResNet-6 with its Mean Pool and Style variants outperform the baseline (local) 69.1 with scores of 69.7 and 69.5 respectively. In Fig. [1b for early-stopped ResNet-200 model, the mean pool and style functions improve upon the bascline score of 69.5 with 69.8 and 69.7 respectively.
We additionally observe that the optimal early-stopped ResNet-6 from Table 1 further improves its perfor- mance with its mean pool variant achieving a PS of 70.2. This matches the best reported PS in (Zhang et al. 2018), where the AlexNet model is trained from scratch on the BAPPS train set. Note that none of our networks have seen the BAPPS train set during ImageNet training.
However, although the improved perceptual functions attain better PS compared to the baseline, the inverse-U correlation is still prominent. Therefore, can conclude that while the per-pixel comparison function is suboptimal, it is not the main cause of the inverse correlation between accuracy and PS.
Learned linear layer on pretrained features. Lastly we investigate what happens the similarity lunction is learned on supervised data. Although the main goal of our paper to assess the inherent perceptual properties of ImageNet models, we may also train linear layer on top of pretrained ImageNet features to match supervised human judgements BAPPS. The PS gap between ResNet-6 and ResNet-200 narrows down from 1.5 to 0.8, but even after training, the R still outperforms the ResNet-200. See Appendix for more details.
8.2 Low PS models not necessarily less sensitive distortions
Here, we explore whether sensitivity to distortions the common latent factor influencing both PS and accuracy. Intuitively, better networks will be loss sensitive to the distortions the BAPPS datasot, since the class will not change under these distortions. This intuition is supported by results that show that accuracy under distribution shifts (including artificial corruptions) correlates strongly with "clean" ImageNet accuracy (Taori et al. 2020). Therefore, if decreased sensitívity is related to poorer PS, due to inability to distinguish different class- puss-preeeriing perturbations, then this could explain our observations.
From the BAPPS dataset, we retain only the examples, where the human raters unanimously agree that one of the target patches closer the reference patch than the other, i.c 1.0 or p 0.0. For each such triplet (zy, x, 21) where p 1.0 or p 0.0, we denote xy to be the farther patch and on be the nearer patch. Concretely, in Eq 2 when p 1.0, 3 Ho F.n 21 or p 0.0, 31, 3 Zo--
12

---

Published in Transactions on Machine Learning Research (09/2022)
www
0.26
Resket Efficient
Distortion Sensitivity
0.35
0.24
Distance
0.30
0.22
0.25
0.20
0.20
0.18
0.10
0.16
Farther Patch
0.05
0.14
0
20
40
60
80
67.0
67.5
68.0
68.5
69.0
Number of Train Epochs
Perceptual Score
(a)
(b)
Figure In Fig. 2a
distance assigned the nearer and farther patch for ResNet-200 model as a
function of train steps. Fig- 12b displays a scatter plot between the distortion sensitivity measure (See: 8.2)
and accuracy of networks trained with their default hyperparameters.
We mcasure distortion sensitivity using the distance margin 2 d() Ez,,, T.). We expect this margin to be larger for a distortion sensitive network. In Fig. 12a among out-of-the-box classification networks, there exists no positive correlation between distortion sensitivity and PS. As another experiment, in Fig. 12b we plot Ez,z d(x, (Farther Patch) and Ez,,, d(x, xn) (Nearer Patch) as a function of training epochs (ResNet-2200) From Fig. 5b we know that PS decrcases as function of epochs after it reaches a peak. However in Fig. 12b] the distance margin between the farther and nearer patch remains fairly constant function of epochs. Hence, low PS models not necessarily less sensitive to distortions.
8.3
Sub-optimal features are not cause of the inverse-U relationship
69.5 5
Early
69.0
Baseline
Perceptual Score
Perceptual Score
69.0
68.5
101
68.5
68.0
67.5
68.0
67.0
67.5
66.5
67.0
66.0
1
3
1
3
Reduction Stage
Reduction Stage
Figure 13: 'Per-layer PS' for ResNets with different depths in Fig. 13a and for early-stopped Res.Net-200 model with optimal PS in Fig. 13b
Remember, PS is averaged over many layers, see Eq 2 However, it might be the case that optimal features for PS are buried in specific layers for better classifiers (e.g. lower layers), while other layers (e.g. high layers) exhibit different behaviour more optimal for classification. Therefore, we look the best PS across layers.
We define the wiie PS* d(ab. af) to be H.W; Σr... 11 - Tihh 112. The PS in Eq 2 can be viewed as mean ensemble of wise PS". The "optimal layer-wise PS is then maxtec d(z) 21 Fig. 13a displays the layer-wist PS for each of the 2D representations across different ResNet depths. The optimal for all depths is and larger depths attain worse PS even at this optimal Z. We additionally that in Fig. 13b
13

---

Published in Transactions on Machine Learning Research (09/2022)
ResNet-200 under-performs the optimal layer-wise PS of its carly-stopped variant at 1 Therefore, we can conclude that sub-optimal features are not a cause of the inverse-U U relationship.
8.4 ImageNet class granularity cannot explain why ResNet-6 outperforms ResNet-200 PS
ImageNet is 1000 class classification problem that includes fine-grained classes. A classifier that mod- els such classes successfully could have a reduced PS, since it could compromise on learning general fea- tures. The low accuracy ResNet-6 implies that its capacity sufficient model only a subset of these classes, and its inability to model tougher classes might explain its high PS. We create random subsets having number of classes ranging from 50 to 900 and train ResNet-6 and ResNet-200 networks on each of these subsets. In Fig. 8.4 the PS gap between ResNet-200 and ResNet-6 reduces as the number of classes are decreased. But, ResNet-200 still under- performs ResNet-6. Therefore, class granular- ity cannot fully explain why less-accurate ResNet-6 significantly outperforms ResNet- 200 on PS. In Appendix EL] we show similar results with a class subset selection strategy guided by a pretrained ResNet-6.
70.0
69.5
200
Perceptual Score
69.0
68.5
68.0
67.5
67.0
200
400
600
800
1000
Number of classes
Figure 14: We create ImageNet subsets by randomly sampling x number of classes and then train models on each of these subsets. The plot shows PS of ResNet-6 and ResNet-200 models number of ImageNet classes (x) in these subsets.
8.5 High P5 features do not necessarily have high entropy
Wang et al. (2021a) show that ResNets are not suitable for style transfer due to the presence of skip connections. Skip connections result in features with low entropy which prevent capturing all style modes from a ground-truth style image. We explore the mean entropy of activations can explain the inverse-U phenomenon between PS and accuracy. As done in Wang al.] (2021a), we convert intermediate activations x E #x into probability distribution across W C values by applying softmax transformation, We then report the average normalized entropy of this distribution across four 2-D representations the BAPPS validation set.
Fig. 15a plots the normalized entropy of each of the four ResNet-200 reduction stages (marked 1 - 4) across training. As seen in Wang et al. (2021a). representations closer to the output at later reduction stages have a much lower entropy than representations closer to the input al earlier reduction stages. The entropy also decreases as a function of train steps. For the first few training epochs, where the entropy is between 0.9 and 1.0, there is a negative correlation between PS and mean activation entropy. Note that the maximum entropy is at initialization and not after few training epochs where ResNet-200 obtains its highest PS. After the first few epochs, there is a positive correlation where entropy and PS both decrease during training. Fig 15a suggests that there is optimal entropy as function of train steps, where the PS peaks.
However, Fig. 15d shows that there is almost no correlation between entropy and PS across ResNets with various depths. ResNet-6 achieves the highest. PS a mean entropy of 0.5 while ResNet-50 features have the highest entropy around 0.7 and have much lower PS of 68.0. Therefore, entropy does not fully explain the observed effect.
We remove all skip connections in the ResNets to increase the entropy of the intermediate features as done in Wang et al. (2021a). Note that the maximum depth that we are able to successfully train without skip connections is 50. In Fig. [6a] removing skip connections increase the entropy across all depths. Specifically, 50 has a huge increase in entropy, making the activations close to a uniform distribution. Even after removing the skip connections in Fig. [66 and Fig. 16c ResNet-6 and carly-stopped ResNets attain the highest PS respectively similar to the baseline ResNets. While increasing the entropy of intermediate features improve results of ResNets on style transfer, they don't 't improve PS.
14

---

Published in Transactions on Machine Learning Research (09/2022)
ResNel-200
Restiet
acthorton Entree
actvacoon Entrrpp
$9.0
0.6
68.5
0044
68.5
$8.0
68.0
67.5
0.5
0.7
10
161
200
67.00
0.0
(c)
(d)
Figure 15: Fig. 15a depicts the entropy of each of the four ResNet reduction stages as function of train epochs. The black line the mean activation entropy across the 4 reduction stages. Fig. [5c] plots the
entropy as a function of depths in ResNets. Figs. 15b and 15d are the corresponding scatter plots between entropy and PS.
70.0
Mean Actvvatton Entropy
We alp
Wil 1
0.8
Percoptual Score
0.6
68.5
68.5
0.4
68.0
68.0
0.2
67.5
67.5
0.0
67.0
67.0
30
10
20
60
100
Depth
Depth
Number of Epochs
(a)
(b)
(c)
Figure 16: Figs. 16a 16a shows the entropy of ResNet models without skip connections compared to bascline ResNets across all depths. Figs. 16b and 16e display the PS of ResNets without, skip connections across depths and train epochs; the pattern is the same for ResNets with skip connections.
8.6
Low PS models not necessarily more reliant on high frequency information for classification
100
100
Normalized Accuracy
200
Normalized Accuracy
Early-Stop
80
80
60
60
40
40
20
20
0
0
10
20
30
40
50
60
10
20
30
40
50
60
Filter Size
Filter Size
(a)
(b)
Figure 17: Normalized Accuracy function of low-pass filter sizes for ResNets trained with default hyperparameters. (Fig. [7a] and ResNet-200 Baseline (converged) vs early-stopped (Fig. 17b
Networks that rely more on high-frequency information for classification could be less robust to high- frequency distortions or removal of high frequencics from an image (Yin al. 2019), and as an effect have low PS. We analyze the relationship between spatial frequency sensitivity different networks and their PS. A low-pass square filter of side filters out the high frequencies in image outside square with
15

---

Published in Transactions on Machine Learning Research (09/2022)
edge length in its Fourier spectrum. We measure the "normalized accuracy", which is the accuracy low-pass filtered images divided by its accuracy on clean images as a function of r. A model more reliant on high frequency information will have a higher "normalized accuracy" slope at high values of r. ResNet-6 has a higher "normalized accuracy" slope at a high r = 40 to 50 as compared to ResNet-6 (Fig. 17bj. Despite being more reliant on higher spatial frequencies. ResNet-6 achieves a higher PS compared to ResNet-200. Similarly, ResNet 200 becomes more reliant on higher spatial frequencies if early stopped (Fig. 17a
while also increasing its PS (Fig. 5bp. These results indicate that models that have low PS are not necessarily more reliant high frequency information for classification.
9 Conclusion
In this paper, explore the question if better classifiers serve as better feature extractors for perceptual metrics. To answer this question, we conduct experiments across ResNets and ViTs across many different hyperparameters. Except for label smoothing and dropout, we see that PS exhibits an inverse-U relationship with accuracy across the hyperparameters we considered. We then probe a number of explanations for the inverse-U U relationship involving skip connections, Global Similarity Functions, Distortion Sensitivity, Layer- wise Perceptual Scores, Spatial Frequency, Sensitivity, and ImageNet Class Gramularity. While none of these explanations can offer explanation for the observed tradcoff between ImageNet accuracy and perceptual similarity, we hope paper opens the door for further research in this area.
Broader Impact Statement Our results are based on BAPPS, which consists of exclusively low-level distortions as opposed to high-level semantic differences. We believe low-level distortions such gaussian blur and color distortions less likely to be susceptible to bias across different human categorics compared to high-level semantic features such as facial features. It is an open and interesting question whether different categories of humans like race and gender perceive low-level distortions differently. Increasing the diversity of both distortions and human labels in future perceptual similarity datasets is another interesting direction that might help to mitigate human biases.
References
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of- distribution robustness throughout fine-tuning, arxiv preprint arXiv:2106.15891 2021.
Maria Attarian, Brett D Roads, and Michael C Mozer. Transforming neural network visual representations to predict human judgments of similarity. arXiv preprint (xXx:201000635) 2020.
Mohammad Babacizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint ar arXiv:2106. 13195, 2021.
Goutam Bhat, Martin Danclljan, Lue Van Gool, and Radu Timofte. Deep burst super-rcsolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. June 2021.
Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photographic global tonal adjustment with a database of input/ /output image pairs. In CVPR 2011, pp. 97-104. IEEE, 2011.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision, pp. 1511-1520, 2017.
Ekin D Cubuk, Barret Zoph, Samuel Schoenholz, and Quoc V Le. Intriguing properties of adversarial examples. Xiv preprint arXiv: 1at:1711.022 1.02846, 2017.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoangment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113 123, 2019.
Cecilia Pasquini, Valentina Conotter, and Giulia Boato. Raise: A raw images datasct for digital image forensics. In Proceedings of the 6th ACM multimedia systems conference, pp. 219 224, 2015.
Tue-
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D'Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 16458-16468, 2021.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29:658 666, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Rep- resentations, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 12873- 12883, 2021.
Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pp. 3233 3246. PMLR, 2020.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.064 2015.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Tmagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arviv preprint arxie:1811. 12231, 2018.
Stephen Hanson and Lorien Pratt. Comparing biases for minimal network construction with back- propagation. Advances in neural information processing systems, 1:177-185, 1988.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Katherine L Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXie preprint arXim:1911.09071 2019.
Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Tan Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trado-offs for modern convolu- tional object detectors. In Proceedings of the IEFE conference on compuler vision and pallern vecognition, pp. 7310 03117 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer, Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXic preprint arXiv:1602.07338 07360 2016.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European conference on computer vision, pp. 694 711. Springer, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110 8119-111,9111 2020.
Simon Kornblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2661-2671, 2019.
Alex Krizhevsky, Πyy Sutskever, and Geoffrey E, Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25:1097-1100 2012.
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblurring (orders-of- magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8878 888788887 2019.
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Hisan Yang. Learning blind video temporal consistency. In Proceedings of the European conference on computer vision (ECCV), pp. 170 185, 2018.
Alex X Lee, Richard Zhang, Frederik Ebert, Picter Abbeol, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint 1811:0804, 2018a.
Hsin- Ying Lee, Hung-Yo Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming- Hsugn Yang. Diverse image- to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018b.
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. aixio preprint wxXiv:100..01080 2017.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14458 14467, October 2021.
Andreas Lugmayr, Martin Danelljan, Lue Van Gool, and Radu Timofte. Srflow: Learning the super- resolution space with normalizing flow. In European Conference on Computer Vision, pp. 715-732. Springer, 2020.
Andreas Lugmayr, Martin Danelljan, and Radu Timofte. Ntire 2021 learning the super-resolution space challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 596 612, June 2021.
Rafal Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on grophics (TOG), 30(4):1-14, 2011.
Ricardo Martin-Bruadd Noha Radwan, Mchdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7210-7219, June 2021.
Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksckgonul, Byron Rogers, Matthias Bethge, and Mackenzie W. Mathis. Pretraining boosts out-of-domain robustness for pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1859-1868 January 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
18

---

Published in Transactions on Machine Learning Research (09/2022)
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: On the strong correlation between out-of- ditt----iiiiitttin and in-distribution generalization. In International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.
Reiichiro Nakano. A discussion of 'adversarial examples are not bugs, they are features': Adversarially robust neural style transfer. Distill, 2019. doi: 1..23915/distill.00011.. https://diitiillppbbb 2019///22///// advex-
gogeddiicussion/essposee.
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mchran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.0080 2019.
A Emin Orhan. Robustness properties of facebook's resnext wsl models. orXiv preprint arXiv:1907.07644 2019.
Nikolay Ponomarenko, Lina Jin, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chchdi, Marco Carli, Federica Battisti, et al. Image database tid2013: Peculiarities, results and perspectives. Signal processing: Image communication,, 30:57 77, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. aiv preprint arXiv:2103, 00020, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5440. PMLR, 2019.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yolam Nitzan, Yaniv Avar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: A stylegan encoder for image-to-image translation. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2287-2296, June 2021.
Gernot Riegler and Vladlen Koltun. Free view synthesis. In European Conference on Computer Vision, pp. 623 640. Springer, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
Martin Schrimpf, Jonas Kubilius, Ha Hong. Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, el al. Brain-score: Which artificial neural network for object recognition most brain-like? BioRrív, pp. 407007, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 140 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929 1958, 2014. URL
wttpp/irrrrggpppp/////s////////. http://////rriaaaaaaaaaaaaa html
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2818 2826, 2016b.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pp. 6105 6114. PMT.R, 2019.
19

---

Published in Transactions on Machine Learning Research (09/2022)
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mea- suring robustness to natural distribution shifts in image classification. arXiv preprint orXin:2007.0044. 2020.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrep- ancy. arXiv preprint arXiv:1906.00 06423, 2019.
Richard Tucker and Noah Snavely, Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 551-560, 2020.
Pei Wang, Yijun Li, and Nuno Vasconcelos. Rethinking and improving the robustness of image style transfer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 124-133, 2021a.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view Emage-basee rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4690-4699, June 2021b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Fero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600 612, 2004.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7467-7477, 2020.
Bohan Wu, Suraj Nair, Roberto Martin-Martin Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognation, pp. 2318-2328, 2021.
Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoe Le. Smooth adversarial training. arXiv preprint XrXi::2006. 14536, 2020a.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10687-10699 2020b.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 32:13276- 13286, 2019.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of repre- sentation learning with the visual task adaptation benchmark. arXiv preprint arXi::1910. 048667,, 2019.
Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2737 2746, 2020.
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEFE Transactions on Image Processing, 20(8):2378 2386, 2011. doi: 10.1109/TIP.2011 2109730.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Youshan Zhang and Brian D. Davison. Impact of imagenet model selection on domain adaptation. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, March 2020.
20

---

Published in Transactions on Machine Learning Research (09/2022)
Shengyu Zhao, Jonathan Cui, Yihun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. orXiv preprint orXiu:2103. 10428, 2021.
Barret Zoph, Golnaz Ghiasi, Tsung- Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. Rethinking pre-training and self-training. Xiviv preprint or Xiv:20006. 06882, 2020.
A Perceptual Scores: High Resolution Models
We train out-of-the-bos ImageNet models their typical resolutions (224 224) and above. We then resize the 64 64 BAPPS images to the resolutions which the models trained on and report their accuracies and P$ in Figure 18a The inverse-U relationship still cxists. But as expected, the perceptual scores across all models are considerably worse. Our best and worst PS are 67.6 and 64.2 as compared to 69.1 and 67.3 obtained with models trained directly on 64 x 64 images. Therefore, we advise to train models on smaller images (64 x 64) to have a high PS.
We also evaluate the PS of out-of-the- box ResNets trained on high-resolution images using 64 x 64 BAPPS images directly and observe the same phenomenon in 18b
B Perceptual Score: Sub-dataset Breakdown
BAPPS consists of two sub-datasels "Distortions" and "Real Algorithms" that differ in the generalive process of the target patches. In Figs. 21 and 22 show scatter plots for accuracy-PS and PS vs hyper-parameter values for the "Distortion" subset. We show similar results for the "Real Algorithms" subset in Figs. 23 and
24 The inverse-U relationship generalizes across each sub-dataset of BAPPS.
C
Perceptual Score: Learned
Previous work found that is possible improve the prediction of human binary choice on a dataset of bird images Attarian al. (2020) and human 2-AFC judgements Zhang et al. (2018) by finding appropriate linear transformations. Through our paper focused assessing the inherent properties of ImageNet pretrained models to capture perceptual similarity, yet following the linear evaluation protocol in [Zhang ct al. 2018), we train a lincar layer on top of pretrained ImageNet features to match supervised human judgements on BAPPS. The PS gap between ResNet-6 and ResNet-200 narrows down from 1.5 to 0.8, but even after training, there a negative corelation among high-performing state-of-the-art residual networks in Fig.
D Perceptual Score: Learning Rate
We demonstrate that the inverse-U U phenomenon also exists as function of the peak learning rate in ResNets
in
20
E
Class Granularity: More Results
We replace the "random sampling" strategy in our previous "Class Granularity" experiment with two vari- ants. We compute ResNet-6's accuracy on cach ImageNet class and rank the classes according to the per-class accuracy. In our "lop" and "bottom" experiments, each subset of l classes consists of the top k and bot- tom classes according to this ranking respectively. In Figs. 25b and 25a we vary from 50 to 900. As observed our previous experiment, as the number of classes are reduced, ResNet-200's PS improves but still underperforms ResNet-6. Interestingly, the particular sampling strategy used does seem to have a significant. effect.
21

---

Published in Transactions on Machine Learning Research (09/2022)
67.5
68.0
67.0
Perceptual Score Score
66.5
67.5
Perceptual
65.5
67.0
65.0
64.5
Reshet
Rendee reduced depith
66.5
10
20
30
40
20
40
60
80
Accuracy
Accuracy
(a)
(b)
Figure 18: We train all tut-oo-thhe-ox networks their typical higher resolutions (224 X 224) and above. In Fig. T8a we resize the 64 x 64 BAPPS images to match the resolutions at which the corresponding
networks are trained. In Fig. 18b we display the PS evaluated on 64 x 64 images directly on out-of-the-ban
ResNets. The inverse-U relationship exists but the perceptual scores across all architectures are considerably worse
Accuracy vs Calibrated Perceptual Score
70.2
Reste:
Perceptual Score
70.0
69.8
69.6
69.4
40
45
50
55
60
65
Accuracy
Figure 19: We train a linear layer on top of ImageNet features. ttete-of-the-art residual networks.
The negative correlation exists among
F Spatial Frequency Analysis: Further Exploration
display the "normalized accuracy slope" func- tion of r. The "normalized accuracy slope" at radius r is the difference between the "normalized accuracy" at radius r and the "normalized accuracy" at radius r- 1. It helps us understand: at which frequencies do different models have their highest accuracy gains? Models that have high PS such as ResNet-6 and early-stopped ResNet-200 have higher slopes at higher frequencies as compared to low PS models.
Normalized Frequency Slope. Figs. 26a and 26b
G Augmentations:
We investigate the effect of Gaussian Noise Augmentation and AutoAugment on PS. By construction, any augmentation strategy forces the network to produce the same outputs the last layer of the network, for different distortions of image. One can thus expect augmentation to make network less sensitive to various distortions of an image, and as result lower its PS.
Gaussian Noise. For a given noise factor 8, we first sample o ~ UO, s) per-image. We then sample per-pixel noise independently from the Gaussian distribution A/O0, σ), truncate it between 0.0 and 1.0 and add it to the original image. We show results for noise factors, 0.1, 0.2, 0.5 and 1.0. The "Distortions" subset of BAPPS consists of Gaussian-noise based distortions, and Gaussian noise augmentation would make
22

---

Published in Transactions on Machine Learning Research (09/2022)
Perceptual Score
69.0
68.5
68.0
67.5
67.0
66.5
10
20
30
40
50
60
Accuracy
Figure 20:
Inverse-U U as a function of peak learning rates.
ResNet: Number of Train Epochs
VIT: Number Train Epochs
VIT: Depth
78.5
perceptual Stare
77.5
78.0
Uk
78.5
77.0
Perceetual
76.5
78.0
76.0
77.5
76.5
77.0
75.0
76.0
50
Accuraty
Accuracy
(b)
(c)
VIT: Width
ResNet: Width
Center Crap
77.5
79.0
69.0
$0
200
78.5
68.5
A/8
Beroagtual
27.0
78.0
50
76.5
77.5
68.0
4-44
208
76.0
77.0
75.5
76.5
67.0
75.0
76.0
66.5
Accuracy
Accuracy
Accuracy
(d)
(c)
Weight Decay
Label Smoothing
Dropout
79.0
78.5
Parcaptual Score
78.0
77.5
77.5
78.0
77.0
76.5
77.5
76.5
76.0
76.0
77.0
75.0
70
40
50
Accuracy
Accuracy
Accuracy
(g)
(h)
The relationship between Perceptual Scores and accuracy when varying different hyperparameterss on the tradional algorithms subset of BAPPS. Each plot depicts the ImageNet accuracy (x axis) and Perceptual Score (y axis) obtained by a 1D sweep over that particular hyperparameter.
Figure 21: Distortions:
the network less-sensitive to such distortions. As an effect, in Fig. 27b an the noise factor increases, the PS on the "Distortions" subset of BAPPS decreases by a huge amount.
Augment.. We train models with statc-of-the-art augmentation technique AutoAugment Cubuk et al. (2019) that consists of mixture of different augmentations. As seen in Figs. 27d architecture, AutoAugment decreases both the accuracy and PS. We note that AutoAugment strategy
27 27c for every
23

---

Published in Transactions on Machine Learning Research (09/2022)
78.5
77.
78.5
77.0
77.5
77.0
76.0
96.5
77.0
76.
76.0
76.5
75.5
77.0
60
80
15.0
30
100
150
200
230
200
10
25
28
20
75.0
10
22
04
128
255
512
(N)
(b)
(c)
(d)
79.0
76.5
78.5
76.5
76.0
70.0
77.5
78.0-
77.5
37.0
26.5
27.0
77.5
7600
76.5
77.0 0.5
055
14
3.5
2.0
2.5
10 18
10 19tH
10
10
75.0
01
02
0.1
04
05
0.5
07
08
0.1 02 0.1
94
05 06 07 0.8 ne
Fiter
Dropout
Smoothing
(e)
(E)
(g)
(h)
Figure 22: Distortions: In each plot, we vary a single hyperparameter along a 1D grid and plot the Perceptual Scores on the distortions subset of BAPPS
ResNet: Number of Train Epochs
VIT: Number of Train Epochs
VIT: Depth
64.2
64.0
64.0
Percaptial
63.8
63.6
63.4
63.2
10
30
40
50
60
20
30
40
50
60
70
10
20
30
40
50
60
70
Accuracy
Accuraty
Accuracy
(a)
(b)
(c)
ViT: Width
ResNet: Width
Center Crop
65.0
64.5
Perceppbal Scare
64.5
50
64.0
1/44 4y80
64.0
63.5
4/4
63.5
63.0
50
63.0
Bis
bentre
63.0
10
20
30
40
50
60
70
10
20
30
40
50
60
70
62.5
Carte
Accuracy
Accuracy
Accuracy
(d)
(c)
(f)
Weight Decay
Label Smoothing
Dropout
65.5
64.5
64.5
64.5
64.0
64.0
63.5
63.0
63.5
62.5
63.0
2
62.0
62.5
62.5
61.5
62.0
10
30
40
50
60
70
40
50
60
70
62.0
10
20
30
40
50
60
70
Accuracy
Accuracy
Accuracy
(h)
Figure 23: Real Algorithms: The relationship between Perceptual Scores and accuracy when varying dif- ferent hyperparameters on the real algorithms subset of BAPPS. Each plot depicts the ImageNet accuracy (x axis) and Perceptual Score (y axis) obtained by a 1D sweep over that particular hyperparameter.
24

---

Published in Transactions on Machine Learning Research (09/2022)
65.0
$4.0
$4.0
64.3
Peresttual
Percsptual
63.5
63.5
03.0
$2.5
62.5
62.0
90
100
150
200
230
300
10
18
20
16
64
129
256
512
(
(b)
(c)
(d)
05.0
61.0
$4.5
64.5
Parceptual Score
64.0
64.0
65.5
64.0
83.5
61.0
83.0
62.5
67.5
52.9
$2.0
62.0
0.5
055
1.0
25
2.0
2.5
62.0
01
02
03
0.8
05
05
07
00
01
02 03 04
05 D6 07 0. 69
Smouthing
(c)
(0)
(h)
Figure 24: Real Algorithms: In each plot, we vary a single hyperparameter along a 1D grid and plot the Perceptual Scores on the real algorithms subset BAPPS.
69.5
69.5
Perceptual Score
69.0
Perceptual
68.5
68.5
68.0
68.0
67.5
67.5
0
200
400
600
800
1000
200
400
600
800
1000
Number trained
Number of trained
(b)
Figure 25: We dive deep into the impact of class granularity. We create class subsets according to ResNet- 6's accuracy, instead of random subset. In Fig. [5a and Fig. 25b show the accuracy and PS of the networks trained on the top and botton æ number of classes as given by ResNet-6's accuracy.
developed explicitly by maximizing the validation accuracy on high resolution (224 x 224) images, and thus the policy might not transfer to low resolution (64 64) images. This can explain the decrease in accuracy
H Accuracy vs Hyperparameters
We plot the accuracy as a function each hyperparameter where we observe the inverse-U U relationshop between validation accuracy and Perceptual Score Fig. 28. We note that the validation accuracy stcadily increases as function of the hyperparameter. The peak in Perceptual Scores occur in the underfitting regime where the ImageNet models have poor to moderate accuracies.
Efficient Net: Perceptual Score
In Fig 29
we obscrve the inverse-U tradeoff in EfficientNet-BO and EfficientNet-B5 on varying function of depth, width and number of epochs respectively. EfficientNees attain their peak PS at extremely small width, depth and number of train steps.
25

---

Published in Transactions on Machine Learning Research (09/2022)
8
Normalized Accuracy
Eaily Stop
Normalized Accuracy:
6
4-
3-
2
0
10
20
30
40
50
60
0
10
20
30
40
50
60
Filter Size
Filter
(a)
(b)
Figure 26:
We further explore the spatial frequency responses to networks via Normalized Frequency
Slopes: (Figs.
26a and 26b
69.0
68.0
645
68.6
76.4
68.2
75.8
64.0
68.0
75.2
67.0
74.6
63.
74.0
67.4
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Gaussian Noise
Gaussian Noise
Gaussian Noise
(a) All
(b) Traditional
(c) Real
69.2
Bocbre
78.6
20
Imdire
64.5
69.0
78.4
68.6
20
78.2
64.0
68.4
78.0
perceptual
63.5
68.2
68.0
77.8
2104
63.0
200
77.6
67.6
62.5
35
40
45
50 Accuracy
55
60
65
40
45
50
60
65
35
40
45
50 Accuracy
55
60
65
Accuracy
(d) All
(e) Traditional
(f) Real
Figure 27: We investigate the effect of Gaussian Noise Augmentation and AutoAugment strategies on PS
26

---

Published in Transactions on Machine Learning Research (09/2022)
Accuracy
70
50.0
60
horue233
60
Arcurayy
40
50
Accurscy
50
42.
30
40.0
20
35.0
10
20
32.5
20
60
80
50
100
150
200
25.
300
0
50
100
150
200
250
300
Number epochs
VIT: Number epochs
VIT: Number apochs
(a)
(b)
(c) Real
62-
60
20
10
25
138
2,0 15
200
10- 16-1
16-4
16-1
1617
16
Dapei
VIT
Fllarmabsste
Weigh Decaa
(d)
(e)
(f)
(g)
Figure 28: In each plot, we vary a single hyperparameter along a 1D grid and plot the validation accuracy.
EfficientNat
EfficientNet
EfficientNet
EfficientNet
69.5
65.0
69.0
69.0
66.0
68.0
68.0
68.0
66.0
67.5
57.5
67.5
67.0
withh Cueffic
(a)
(b)
(c)
(d)
EfficientNet
EfficientNit
69.5
69.0
Farsgkull Scate
69.0
68.0
58.0
67.5
67.0
(e)
(f)
Figure 29: Figs.
29a
29c
and
PS vs accuracy on varying width, depth and number of epochs depict PS function of width, depth and number of epochs in 29d
29e
show
in EfficientNess. Figs.
29b
and
291
NefisientNets.
J Distance Margins: Rank Correlation
The scoring function (Eq. 2 takes into account only if d, do, and not the actual margins (d, do) themselves. To assess if the inverse-U relationship is agnostic to this choice, takes the distance margins into account. Namcly, we measure the rank correlation between the distance margins between the distance margins (dl - do) and the ground truth probabilities p in BAPPS. A model having a high rank correlation, should assign large positive distance margins at high probabilities and large negative distance margins low probabilitics. In Fig. 30 we show that similar PS, rank correlation exhibits an inverse-U relationship for a) ResNets different depths b) ResNet-200 as function of train epochs. Therefore the conclusion remains unchanged.
scoring function that
27

---

Published in Transactions on Machine Learning Research (09/2022)
Restlet
ResNet
Reshet
ResNet
68
60
80-
64
Cerea
Bank
bork
10
20
40
50
50
50
75
100
125
150
175
200
10
20
59
60
10
20
40
(a)
(b)
(c)
(d)
Figure 30: We show that the rank correlation between the distance margins d1 dg in 2 and p BAPPS exhibits an inverse-U behaviour. Figs. 30a and 30c the rank correlation accuracy on varying ResNet depths and train epochs. Figs. 30b
and
30d depict the rank correlation function of ResNet depths and
train epochs.
K Inverse-U U on TID2013
BAPPS explicitly constructed using 64 64 images that human raters focus low-level local changes as opposed to high-level semantic differences. show results on the TID2013 dataset (Ponomarenko et al. 2015), which consists of higher resolution images 384 x 512. Each example in TID2013 consists of a reference image æ, a distorted image and a mean opinion score rating s. For every model, we evaluate the rank correlation between 8 and d(ar,)) y) (as given Eq. [ Firstly, we assess ResNets of various depths trained on ImageNet 224 224. Secondly, we estimate the rank correlation of ResNet-50 model trained Net 384 384 during the course of training. Fig. 31b and 31d show that shallow ResNets and that the inverse-U U
early-stopped ResNets achieve the highest rank correlation. In Figs. and 31c relationship exists on TID2013.
Roshiet TI02013
Rastict: TID2013
ResNiot: TID2013
RosNet TID2013
Coreeatio
81
75-
80
25
29
60
AccuFaCy
Degihh
Acouracy
Number Epechs
(a)
(b)
(c)
(d)
Figure 31: Figs. 3la and 31c show the rank correlation vs accuracy on varying the depth and train epochs depict the rank correlation vs ResNet depths and train epochs
on TID2013 (384 x 512).
Figs. 31b
and 31d
on TID2013 (384 x 512)
L Code Snippets
We present code snippets for the different distance functions used in our paper.
Listing 1: Code Snippets for different perceptual functions
def prnceptual(tensorr tensor2, eps=le-10): "Default perceptual distance function.
Args:
tensorl: shape=(B, H. W. C)
tensor2: (B H, W, C)
Returns:
dist: shape=(B.)
28

---

Published in Transactions on Machine Learning Research (09/2022)
tensorl np. linalg innalgnnrmm normm(ensool, [tensorl, ord=2, axis= keepdims=True)
tensor2 n = np. linalg norm temsor2, ord=2, axis=-l. 1, keepdims-Truu
tensorl = tensorl / (tensorl__ + eps)
lensor2 = lensor2 / (Lensor2
eps)
dist np.sm ppo.su((tenso)))
tensor2) )+*2 axis=-1)
dist up. app.mean (dist, axis=(1, 2))
return dist
def mean pool temsor2, eps-le-10) "Man Pool perceptual distance function.
Args:
lensorl: : shape-(B, H, W, C)
tensor2: shape=(B, H. W, C)
Returns:
dist: shape=(B.)
...
tensorl = up. mean (tensorl, (1, 2), keepdims=True)
tensor2 = up. mean (tensor2, (1, 2), keepdims-True)
return perceptual (tensorl, tensor2, eps=eps)
def compute_ gram (omm(tenn [tensor, eps):
"Returns (BxCrC) cross correlation. ...
h, w, c = tensor.. .hape
lensor = np- reshape (lensor, (-1, haw, c))
tensor tensor_norm norm up. linalg norm (tensor, ord=2, axis=l, keepdims-True) (tensor norm eps)
tensor = tensor
# Channel-wise cross correlation.
tensor t = up. transpose (tensor, (0, 2, 1))
return up. matmulltensor........... tensor_tt t, tensor)
def style (tensorl, tensor2, eps=le-10): "'Style perceptual distance function.
Args:
tensorl: shape=(B, H. W. C)
lensor2: shape=(B, H, W, C)
Returns:
dist: shape=(B.)
tensorl gram
gram tensorl, eps=eps)
tensor2 gram compute gram tensor2, eps=eps)
dist = tensorl gram tensor2_ gram return up. mean dist x*2, axis=(1 2))
M Default
We provide the default training hyper-parameters for the ResNets, L'fficient and Vision Transformers in Tables
and
29

---

Published in Transactions on Machine Learning Research (09/2022)
Hypem-Par
Value
Batch Size
1024
Base Learning Rate
0.1
Train Steps
112590
Momentum
0.9
Weight Decay
0.0001
Label Smoothing
0.0
LR Schedule
Step-wise Decay
Batch-Norm Momentum
0.9
Table 2: ResNet: Default Hyperparameters
Hyper-Parameter
Value
Batch Size
2048
Base Learning Rate
0.128
Train Steps
218949
Optimizer
RMSProp
Momentum
0.9
Weight Decay
1e-5
Label Smoothing
0.1
LR Schedule
Warmup + Exp Decay
Batch-Norm Momentum
0.9
Polyak Average
0.9999
Table 3: EfficientNet: Default Hyperparameters
Hyper-Parameter
Value
Batch Size
4096
Base Learning Rate
3e-3
LR Schedule
Warmup Cosine
LR Warmup Steps
10000
Train Steps
93834
Optimizer
Adam
Betal
0.9
Beta2
0.999
Weight Decay
0.3
Hidden Size
768
MLP Dim
3072
Number of Layers
12
Number of Heads
12
Dropout
0.1
Label Smoothing
0.1
Table 4: ViT-B/8: Default Hyperparameters
30

---

Published in Transactions on Machine Learning Research (09/2022)
Hlyper-Paamees
Value
Batch Size
4096
Base Learning Rate
lc-4
LR Schedule
Warmup Cosine
LR Warmup Steps
10000
Train Steps
93834
Optimizer
Adam
Betal
0.9
Beta2
0.999
Weight Decay
0.01
Hidden Size
1024
MLP Dim
4096
Number of Layers
24
Number of Heads
12
Dropout
0.1
Label Smoothing
0.1
Table 5: ViT-L/4: Default Hyperparameters
31


Reviews:
[{"review_id": "aXpT5vhpFk", "reviewer": "Reviewer_KGpd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The experiments are systematic, including modern architectures like ViT and sweeps over a large number of hyperparameters.\n- With the aim of improving the PS, a concrete suggestion is proposed: reduce the scale of the model.\n- Discussion of the hyperparameter sweeps helps to shed insight into the relationship between accuracy and PS.\n\nWeaknesses\n- The takeaway message is somewhat unclear. The experiments do a good job of establishing the relationship between accuracy and PS, but more could be done to motivate the problem of trying to improve PS and whether a higher PS is intrinsically useful.\n- The discussion and analysis of the results are relatively light. The findings seem to indicate that weaker models are better for PS, but there is little discussion of the deeper reasons for this. For example is the inverse-U phenomenon related to overfitting, lack of calibration, lack of biological plausibility, etc.?\n- Some of the figures are difficult to parse, especially Figure 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the relationship between the accuracy of an image classification network and its ability to predict perceptual similarity as judged by humans. Previous work had identified a positive correlation between the accuracy of a network and its ability to predict perceptual similarity, but this work finds that the opposite is true for more modern architectures. The relationship is explored over a variety of architectures and hyperparameters and the results indicate that there is an inverse-U relationship between accuracy and perceptual score (PS): as the accuracy improves the PS increases up until a point and then decreases again.\n\nContributions include:\n- Experiments that characterize the relationship between accuracy and perceptual score for modern architectures.\n- Hyperparameters sweeps that demonstrate the effect thtat changes to the architecture and training procedure have on the PS.\n- Proposal of an approach to improve PS, namely to reduce the scale of the model until the optimal (lower) accuracy is achieved.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Improve the motivation for perceptual similarity and how better perceptual similarity could have a positive impact.\n- Include more discussion about the reasons for the inverse-U relationship, ideally supported by theory and experiments.\n- Improve presentation of the figures and include missing labels in Figures 3 and 4.\n- Limitations should be discussed in greater detail. In particular, two points come to mind:\n  1. To what extent are these results predicated on the nature of the BAPPS dataset? If there were a hypothetical dataset that had collected perceptual similarity judgments on higher resolution images, e.g. 224 x 224, how would the results and conclusions of this paper change?\n  2. What are the limitations of the distance function (eq 1) and scoring function (eq 2)? The scoring function only cares if $d_0 > d_1$ or vice versa, but is there any merit to considering a scoring function where the network should predict $p$ directly, rather than simply which distance is larger?", "broader_impact_concerns": "Discussion could be added on how the results of this paper depend on the underlying perceptual similarity dataset and the suitability of a global perceptual similarity score. In particular, might it be the case that the conclusions of this paper depend on internal biases of the human subjects whose similarity judgments were collected to form the dataset? Is it appropriate to target a single global perceptual similarity score, or might it be the case that different individuals perceive images differently? If so, what might the implications be of using a neural network to model perceptual similarity in real-world applications?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OO0xTFCcfa", "reviewer": "Reviewer_9e1v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(a) The topic of the relationship between ImageNet accuracy and Perceptual Scores is worth exploring and could be helpful to the community.\n(b) The paper is overall well-organized and the writing quality is pretty good.\n(c) The paper is packed with experiments, the author provides a lot of ablation studies on hyperparameters (e.g. depth, width, number of training steps, weight decay, label smoothing), w.r.t perception scores.\n(d) The paper further explores the more advanced setting, such as different Global Similarity Functions, Distortion Sensitivity, Layer-wise Perceptual Scores, Spatial Frequency, Sensitivity, and ImageNet Class Granularity, that also brings value to the community.\n\nWeaknesses:\n(a) The paper provides a lot of ablation studies In Sec. 4/5/6, however, lacks some insight on WHY those observations should happen in a neural architecture perspective. for example, (i) In Figure 2, ResNet, EfficientNet, ViT all have \"skip\" passes, Could that be the reason why they turned out better in classification and less effective in perception scores? (ii) Shallow, Early-stopped ResNets has a good Perceptual Score compared to other settings, is it because Shallow ResNets are easier to train?.\n(b) Why there is no similar ablation on EfficientNets in Figure 3?\n(c) minor: legend in Figure 2, Efficient -> EfficientNet, Figure 6: Number of ImageNet classes trained on XXX?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the relationship between ImageNet accuracy and Perceptual Scores of modern deep models such as ResNets, EfficentNets, and Vision Transformers, and found they are inferior in Perceptual Scores. Through a series of ablation studies on hyperparameters, such as depth, width, number of training steps, weight decay, and label smoothing, the paper further discover some interesting observation such as shallow, early-stopped ResNets has a surprisingly good Perceptual Score of 70.2.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See weakness above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MD4qd1LT93", "reviewer": "Reviewer_7meX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strenghts:\n- The related work section is well researched (as far as I can judge) and well written. The authors include the relevant papers and position their work well in this context.\n- I really liked the paper until the results section which I found quite confusing. But I think the dissemination of the results should be fixable, and then it would be a really nice paper.\n- The paper shows a high level of rigor in the conducted experiments which is great.\n- I believe the research question is interesting to the computer vision research community.\n\n### Weaknesses:\n- I have written a very detailed review below which details all issues I have with this submission.\n- I think most of my points are well addressable, and are mostly issues with the presentation of the results/ some formulations/ typos. I strongly advise the authors to restructure their results section as it is very hard to parse currently, and impossible to follow in some occasions.\n- I would restructure the results such that the subsection title of the results says e.g. \"The correlation between perceptual similarity and accuracy follows an inverse V-shape for a wide range of models over the course of training\". Then show the Figs 3a and 3b below the paragraph. This will make it much easier for the reader to parse the results.\n- The BAPPS dataset should be explained better and example images of the task should be provided.\n- A summary and a discussion of the findings at the end of the paper would be nice. Currently, the authors present their main findings in Figure 1 and conduct ablations to find out the possible causes. But they never summarize which of the causes they think contributes most/ have any form of a more high-level discussion of their results. This is especially a problem, because the authors show SO many results, such that the reader simply gets lost. I have linked a paper in the “Requested Changes” field which I think may explain their findings and think the authors should look into that paper and see whether the same argument applies in this case. As a summary of this point, I currently understand that there is a pareto frontier between PS and accuracy, but I have no idea why.\n\n\n### General comments:\nEq. 2: It would be nice to give some intuition on eq. 2. I started inserting random numbers for d1, d2 and p to try to understand how it behaves. It would be helpful to draw a graph with synthetic data for some combinations of (d1,d2,p), such as e.g., for typical values of p and varying d1 and d2. This will just give the reader a better intuition how this PS score behaves. This is crucial as this quantity seems to be central for understanding the paper. The quantities from the “Dynamic Range” paragraph could be added into this graph as lines. Currently, the paragraph “Dynamic Range” is hard to grasp /visualize.\n\nPage 4: “We train our networks on ImageNet at a resolution of 64 × 64 and report their accuracies on the ImageNet validation set and PS on BAPPS.” The argumentation around using 64x64 images makes sense since this is also used in BAPPS. Could the authors comment on whether they think their results would generalize to the larger 224x224 resolution which is commonly used? \n\nPage 4: When introducing the BAPPS dataset, please show example images and provide more details what this dataset shows. Are there near-duplicates which the PS score is supposed to find? Are there style similarities or corrupted inputs which the PS score should track? Please show examples for the reference image x and the target images. Are there style-content conflicts? If the query image is a sketch of a dog and the target images are 1) an image of a real dog and 2) a sketch of a cat, which do we think would be more similar in terms of what human subjects say, and what does the PS metric say is more similar? Given that the used PS metric correlates strongly with the “style” PS metric (Figs 6a+6b), I think the similarity between the query image and the target image (2) would be picked.\n\nPage 5: Paragraph “representations”: Please specify which layers have been used to calculate the PS score in the literature here. I assume that the last sentence (“Our initial experiments on using features at the input of every layer instead of every block or reduction stage lead to worse PS across all architectures.”) addresses this point. Please make it more specific.\n\nPage 5: “All modern networks (red, green, and black) in Fig. 2 obtain a higher PS than FSIMc and randomly initialized networks.” Please add the FSIMc and the randomly initializeds networks PS numbers in brackets here.\n\nPage 6: “ResNet-6 happens to achieve this optimal accuracy with a PS of 69.1, outperforming AlexNet.” Please add the Alexnet number here.\n\nPage 6: “3) ResNet-200 and ResNet-3 achieve similar PS, while having a 45% difference in accuracy.” I believe one should say “a difference of 45 percentage points”.\n\n\nFig 3 is ordered quite chaotically. I would group the ResNets and ViTs in the first 5 graphs and not interchange them all the time. Legends are missing in some graphs. Please add legends in all graphs as I find the graphs currently hard to read, because the legends change from graph to graph.\n\nFig 4: legend for the resnets is missing.\n\n[IMPORTANT] Fig 3 + Fig 4: Please use the same y-axis in all plots, at least in those with the same architectures. Then, one will be able to also judge how strongly the different hyperparameters influence the PS scores. This is true for all Figures, e.g. 6a+6b.\n\nSection 8 is very confusing in terms of writing and presentation of results. First, Figs. 6a and 6b should be separate Figures and not included in the 9x9 figures plot. I think it would be much better if the authors could break this section into parts where they write one question they want to analyze, e.g. comparing the style / mean pool loss to the local loss and then showing the Figures 6a+6b directly underneath the text with the discussion. Similarly, the current Figs 6c+6d should be placed separately below the distortion sensitivity subsection.\n\nI would advise the authors to rename their subsection headings in section 8 (and possibly the other results sections as well) such that the result is evident from the title, e.g., 8.2. could say “Decreased distortion sensitivity of stronger models does not explain their lower PS score”. This will make it easier to understand the results and is a quite customary way of disseminating results.\n\n“In Figs. 6a and 6b, we present scatter plots between accuracy and PS with the style and mean pool similarity functions.” What do 6a and 6b show? In terms of notation, they look exactly the same to me. Looking at the caption, I see that “(Fig. 6a: Out-of-the-box Networks and Fig. 6b: ResNet-\n200 on varying Train Epochs)” What does “out-of-the-box networks” mean?\n\n\n“ResNet-6 with its Mean Pool and Style variants outperform the baseline (local) 69.1 with scores of 69.7 and 69.5 respectively. On early-stopped ResNet-200, the mean pool and style functions improve upon the baseline score of 69.5 with 69.8 and 69.7 respectively. Early-stopped ResNet-6 with its mean pool variant achieves 70.2.” ResNet6 is referenced twice with different numbers. Does the text relate to Fig.6a or 6b or some other plot?\n\n\nPage 10: “The PS gap between ResNet-6 and ResNet-200 narrows down from 1.5 to 0.8, but even after\ntraining, the ResNet-6 still outperforms the ResNet-200. See the appendix for more details.” Please add more detailed references to the Appendix, e.g. “see appendix A.12 for more details.”\n\n\nPage 10: “Intuitively, better networks will be less sensitive to the distortions introduced in the BAPPS\ndataset, since the class will not change under these distortions.” You should describe the BAPPS dataset in detail earlier in the paper, such that the reader knows that there are distortions in this dataset.\n\n\nPage 10: “Learned.” Very strange heading for a paragraph.\n\nPage 11: “Let xf and xn be the farther and nearer patch to a reference image x, respectively.” What are farther or nearer patches? Please provide example images, I currently have no intuition what you mean here.\n\nIt says Fig 6e twice. I believe it should be 6i.\n\nIf the authors want the reader to read through the appendix, they should refer to it in the paper. Otherwise, the reader has no incentives to open the appendix.\n\n### Minor:\n\nFig. 1 is not referenced in the text in the introduction. Fig. 1 is referenced later, but should be also mentioned in the intro since it occurs so early in the paper.\n\nPage 3: „Hermann et al. Hermann et al. (2019) show that among high-performing models, shape-bias is correlated with ImageNet accuracy.” Should be \\citep{} not \\citet{} here.\n\nPage 4: “Unlike accuracy that can vary from 0 to 100, PS have a narrow dynamic range.” PS has …\n\nPage 5: “Additionally, all networks perform worse than A lexNet.” Typo\n\nPage 6: “In particular, we assess this relationship between accuracy and PS as a function of via 1D\nhyperparameter sweeps.” Typo\n\nPage 6: “2) PS against the hyperparameter values ((Fig. 4).” Typo \n\n\nFig. 3 + Fig4: I would remove the top+right canvas axes as described in: https://stackoverflow.com/questions/925024/how-can-i-remove-the-top-and-right-axis-in-matplotlib\n\n\nPage 9: “we uncover a global Pareto frontier between PS and accuracies, see Fig.. 1.” Typo\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the perceptual similarity metric for different models. So far, the community has mostly used the AlexNet or VGG features to calculate the perceptual similarity metric. In this paper, the authors perform a wide range of experiments to study other possible feature extractors for this metric. They find an inverse V relationship between accuracy and perceptual similarity.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Questions to authors:\n- As far as I know, researchers still use pretrained VGG features for style transfer and ResNet models do not work as well as VGG. Is this a similar phenomenon as the one observed in this paper? In this paper (https://arxiv.org/abs/2104.05623), the authors study the reasons why ResNets perform worse than VGG for style transfer, and find that the issue lies in the skip connections which lead to low entropy features. Could it be that the same issue happens here? Entropy is decreased over the course of training after the network has found some optimum it starts converging into, and this may explain the decreased PS scores?\n- The authors have only tested models with skip connections. Do VGG and AlexNet have a similar accuracy/PS score curve compared to the ResNets/ViTs over training? Please test these models to study the impact of skip connections on the observed phenomenon.\n- Please go through my detailed review and incorporate the suggestions.\n", "broader_impact_concerns": "I do not have any concerns in regards to ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is reviewed by three expert reviewers in the field. Initially, the reviewers have concerns with the clarity of the experimental sections and unclear motivations. Reviewers KGpd and 7meX appreciate the detailed responses by authors and their efforts on revising the manuscript. The quality and clarity of the revised paper has been significantly improved with strong empirical results on the studied phenomenon. 

Both Reviewers KGpd and 7meX expressed that the authors have done a great job addressing all their major concerns and recommend to accept as is. While Reviewer 9e1v did not update the final recommendation, but after reading the authors' responses, the AE agrees that the authors revision sufficiently addressed the concerns. In particular, the added portion in Section 8.4 that explore the relationship between the entropy of activations and PS and the newly included ablation for EfficientNet-B0 and EfficientNet-B5 in Appendix I.

The AE agrees with the reviewers and recommends to accept as is. Congratulations! 
Decision: Accept (poster)