Robust and Generalizable Visual Representation Learning via Random Convolutions
Authors: Zhenlin Xu
Deyi Liu
Junlin Yang
Colin Raffel
Marc Niethammer
Venue: ICLR 2021 Poster (2021)
Keywords: domain generalization
robustness
representation learning
data augmentation

Abstract: While successful for various computer vision tasks, deep neural networks have shown to be vulnerable to texture style shifts and small perturbations to which humans are robust. In this work, we show that the robustness of neural networks can be greatly improved through the use of random convolutions as data augmentation. Random convolutions are approximately shape-preserving and may distort local textures. Intuitively, randomized convolutions create an infinite number of new domains with similar global shapes but random local texture. Therefore, we explore using outputs of multi-scale random convolutions as new images or mixing them with the original images during training. When applying a network trained with our approach to unseen domains, our method consistently improves the performance on domain generalization benchmarks and is scalable to ImageNet. In particular, in the challenging scenario of generalizing to the sketch domain in PACS and to ImageNet-Sketch, our method outperforms state-of-art methods by a large margin. More interestingly, our method can benefit downstream tasks by providing a more robust pretrained visual representation.

one of the Published as a conference paper at ICLR 2021
ROBUST AND GENERALIZABLE VISUAL REPRESENTA- TION LEARNING VIA RANDOM CONVOLUTIONS
Zhenlin Xu', Deyi Liu', Junlin Yang", Colin Raffel', and Mare Niethammer'
University of North Carolina Chapel Hill
2 Yale University
[izhenlinx, mn, craffel 8cs. wnc.cnncc..eeccooooo.oco..e..... unc unc.edu, unc.ve.unc.ddi "Junlin.yangyya le.eyallleed..
ABSTRACT
While successful for various computer vision tasks, deep neural networks have shown to be vulnerable to texture style shifts and small perturbations to which humans are robust. In this work, we show that the robustness of neural networks can be greatly improved through the use of random convolutions as data augmentation. Random convolutions are upproximately textures. Intuitively, randomized convolutions create an infinite number of new domains with similar global shapes but random local texture. Therefore, we explore using outputs of multi-scale random convolutions new images or mixing them with the original images during training. When applying a network trained with our approach to unseen domains, our method consistently improves the performance on domain generalization benchmarks and is scalable to ImageNet. In particular, in the challenging scenario of generalizing to the sketch domain in PACS and ImageNet-Sketch, our method outperforms state-of-art methods by a large margin. More interestingly, our method can benefit downstream tasks by providing robust pretrained visual representation.
prpp--preeeriigg and may distort local
INTRODUCTION
Generalizability and robustness to out-of out-of-distributin samples have been major pain points when applying deep neural networks (DNNs) in real world applications (Volpi et al. (0188) Though DNNs are typically trained on datasets with millions of training samples, they still lack robustness to domain shift, small perturbations, and adversarial examples (Luo et al. 2019). Recent research has shown that neural networks tend to use superficial features rather than global shape information for prediction even when trained on large-scale datasets such as ImageNet (Geirhos et al. 2019). These superficial features be local textures even patterns imperceptible humans but detectable to DNNs, as is the case for adversarial examples (Ilyas et al. 2019). In contrast, image semantics often depend more on object shapes rather than local textures. For image data, local texture differences main sources of domain shift, e.g., between synthetic virtual images and real data (Sun & Saenko 2014). Our goal is therefore learn visual representations that invariant to local texture and that generalize to unseen domains. While texture and color may be treated as different concepts, we follow the convention Geirhos et al. (2019) and include color when talking about texture.
We address the challenging setting of robust visual representation learning from single domain data. Limited work exists in this setting. Proposed methods include data augmentation (Volpi Qiao et al. (2200) Geirhos et 20,12019), domain randomization (Tobin et al.] 2017] Yue
et al.
2018
ct al. 2019). self-supervised learning (Carlucci et al., 2019), and penalizing the predictive power
of low-level network features (Wang et al.) (019a). Following the spirit of adding inductive bias
towards global shape information over local textures, we propose using random convolutions to improve the robustness to domain shifts and small perturbations. While recently Lee et al. (2020)
proposed a similar technique for improving the generalization of reinforcement learning agents in
Code available at httphhtt////// /ttttp:://ittti www.iihtubb... com www.wiiwww...ooo........m...o.. www.:tttoohhhhhooooooooooo kanndConv

---

Published as a conference paper at ICLR 2021
conv filters
3
N 36
Input
k =
k= 3
k = 5
k k=7
k == 11
== 15
Input
0.9
0.7
0.5
0.3
0.1
= 0
Figure I: Top: Illustration that RandConv randomize local texture but preserve shapes in the image. Middle: First column is the input image of size 2242; following columas are convolutions results using random filters of different sizes k. Bottom: Mixing results between image and of its random convolution results with different mixing coefficients a.
unseen environments, we focus on visual representation learning and examine our approach on visual domain generalization benchmarks. Our method also includes the multiscale design and mixing variant. In addition, considering that many computer vision tasks rely on training deep networks based on umageNet-pretrained weights (including some domain generalization benchmarks), we ask "Can more robust pretrained model make the finetuned model more robust on downstream tasks?" Different from (Kornblith et al. 2019 [Salman et al.) 2020) who studied the transferability of pretrained ImageNet representation to tasks while focusing in-domain generalization, explore generalization performance on unseen domains for new tasks.
We make the following contributions:
We develop RandConv, data augmentation technique using multi-scale random-convolutton to generate images with random texture while maintaining global shapes. We explore using the RandConv output training images mixing it with the original images. We show that consistency loss can further enforce invariance under texture changes.
We provide insights and justification on why RandConv augments images with different local texture but the same semantics with the
property of random convolutions.
We validate RandConv and its mixing variant in extensive experiments synthetic and real- world benchmarks as well as on the large-scale ImageNet dataset. Our methods outperform single domain generalization approaches by large margin digit recognition datasets and for the challenging case of generalizing to the Sketch domain in PACS and to ImageNer-Sketch.
We explore if the robustness/geneeraizzabl show that transferring a model pretrained with RandConv on ImageNet can further improve domain generalization performance on new downstream tasks the PACS dataset.
of pretrained representation can transfer. We
RELATED WORK
Domain Generalization (DG) aims at learning representations that perform well when transferred to unseen domains. Modern techniques range between feature fusion (Shen et al. 2019), meta- learning (Li et al. 2018a Balaji et al. 2018), and adversarial training (Shao al. 2019 Li et al.) 2018b). Note that most current DG work (Ghifary et al. 2016 Li et al. 201 8a b) requires a multi- source training setting to work well. However, in practice, might be difficult and expensive to collect data from multiple sources, such as collecting data from multiple medical centers (Raghupathi & Raghupathi 2014). Instead, consider the more strict single-domain generalization DG setting,
2

---

Published as a conference paper at ICLR 2021
where train the model source data from single domain and generalize it new unseen domains (Carlucci et al. 2019 Wang et al. 2019b).
Domain Randomization (DR) was first introduced as a DG technique by Tobin et al. (201 7 to handle the domain gap between simulated and real data. As the training data in (Tobin et al. 2017 is synthesized in virtual environment, it is possible to generate diverse training samples by randomly selecting background images, colors, lighting, and textures of foreground objects. When a simulation environment is not accessible, image stylization can be used to generate new domains (uue et al. 2019, Geirhos et al. 2019). However, this requires extra effort to collect data and to train an additional model; further, the number of randomized domains is limited by the number of predefined styles.
Data Augmentation has been widely used to improve the generalization of machine learning mod- els (Simard al. 2003). DR approaches can be considered a type of synthetic data augmentation. To improve performance on unseen domains, Volpi et al. (2018) generate adversarial examples to augment the training data: Qiao et al.] (2020) extend this approach via meta-learning. As with other adversarial training algorithms, significant extra computation is required to obtain adversarial
examples.
Learning Representations Biased towards Global Shape Geirhos et al. (2019) demonstrated that convolutional neural networks (CNNs) tend to use superficial local features even when trained on large datasets. To counteract this effect, they proposed to train stylized ImageNet, thereby forcing network to rely on object shape instead of textures. Wang et al. improved out-of-domain performance by penalizing the correlation between learned representation and superficial features such as the gray-level co-occurrence matrix (Wang et al.) 2019b). or by penalizing the predictive power of local, low-level layer features in neural network an adversarial classifier (Wang al. 2019a). Our approach shares the idea that learning representations invariant to local texture helps generalization to unseen domains. However, RandConv avoids searching over many hyper-parameters, collecting extra data, and training other networks. It also scales to large-scale datasets since it adds minimal computation overhead.
Random Mapping in Machine Learning Random projections have also been effective for di- mensionality reduction based on the cestance-preservinng property of the MaansonnLindenstraamt lemma (Johnson & Lindenstrauss) 1984). (Vinh et al., 2016) applied random projections on entire images as data augmentation to make neural networks robust to adversarial examples. [ee et al. (2020) recently used random convolutions to help reinforcement learning (RL) agents generalize to new environments. Neural networks with fixh random weights can encode mcaningful reppenseeereeeeeeeeeeeeee representations (Saxe al.) 2011] and are therefore useful for neural architecture search (Gaier & Hap 2019), genera- tive models (He et a 2016b). natural language processing (Wieting & Kiela, and RI. (Osband et al. 2018 Burda al. 2019). In contrast, RandConv uses non-fixed randomly-samples weights to generate images with different local texture.
RANDCONV: RANDOMIZE LOCAL TEXTURE AT DIFFERENT SCALES
We propose using a convolution layer with non-fixed random weights as the first layer of a DNN during training. This strategy generates images with random local texture but consistent shapes, and is beneficial for robust visual representation learning. Sec. 3.1 justifies the shape-preserving property of a random convolution layer. Sec. 3.2 describes RandConv, our data augmentation algorithm using a multi-scale randomized convolution layer and input mixing.
3.1 A RANDOM CONVOLUTION LAYER PRESERVES GLOBAL SHAPES
Convolution is the key building block for deep convolutional neural networks. Consider a convolution layer with filters θ E R x CCut with an input image I R xffxWxC.. where H and W are the height and width of the input and C,n, and Cout are the number of feature channels for the input and output, and h and wo are the height and width of the layer's filter. The output (with appropriate input padding) will be g = I θ with g E R xxxWxxx W C.ott.
In images, nearby pixels with similar color texture be grouped into primitive shapes that represent parts of objects or the background. A convolution layer linearly projects local image patches to features corresponding locations on the output map using shared parameters. While
3

---

Published as a conference paper at ICLR 2021
convolution with random filters can project local patches to arbitrary output features, the output of random linear projection approximately preserves relative similarity between input patches, proved in Appendix B In other words, since any two locations within the same shape have similar local textures the input image, they tend to be similar in the output feature map. Therefore, shapes that emerge in the output feature map similar to shapes in the input image provided that the filter size is sufficiently small compared to the size of a typical shape.
In other words, the size of a convolution filter determines the smallest shape can preserve. For example, Ixl random convolutions preserve shapes at the single-pixel level and thus work as a random color mapping; large filters perturb shapes smaller than the filter size that are considered local texture of shape at this larger scale. See Fig. [oo examples. More discussion and a formal proof are Appendix and
3.2 MULTI-SCALE IMAGE AUGMENTATION WITH A RANDOMIZED CONVOLUTION LAYER
Algorithm 1 Learning with Data Augmentaiion by Random Convolutions
I: Input: Model 4 task loss Ltask, training images [ and their labels (16 J pool of filter sizes
= =(I,..., n), fraction original data p. whether mix with original images, consistency loss weight
2: function RANDCONv(l, K, mix, p)
3: Sample ~ (0,
4: if po < p and mix False then return
When not in mix mode, use original image with probability p
else
7:
Sample scale le ~ IC Sample convolution weights θ E Rexxxxxx ~ N(0, stz)
8:
9:
I..
I O
Apply convolution I
10:
if mix is True then
11:
Sample a ~ U(, return al + a
12:
Mix with original images
13:
else
14:
return Ire 15: Learning Objective:
16: for = + N do
17:
for j 1
18:
8 = ((RandConv(I:))
Predict labels for three augmented variants the same image
19:
Leons = > 3 (L( where DJ = Σ}= 0 /
Consistency Loss
20:
f = frask y.) )1 + ALcons
b Learning with the task loss and the consistency loss
Sec. 3.1 discussed how outputs of randomized convolution layers approximately maintain shape information scale larger than their filter sizes. Here, we develop our RandConv data augmenta- tion technique using a randomized convolution layer with Cout = Cin to generate shape-consistent images with randomized texture (see Alg. T]. Our goal is not represent texture as in previous filter-bank based texture models (Heeger & Bergen
RandConv to parameterize or 1995] Portilla & Simoncelli 2000). Instead, only the three-channel outputs of RandConv as images with the same shape and different "style" (loosely referred to as "texture"). We also note that, a convolution layer different from convolution operation in image filtering. Standard image filtering applies the same 2D filter on three color channels separately. In contrast, our convolution layer applies three different 3D filters and each takes all color channels input and generates one channel of the output. Our proposed RandConv variants are as follows:
RCngg Augmenting Images with Random Texture A simple approach is use the randomized convolution layer outputs, * O as new images; where O are the randomly sampled weights and a training image. If the original training data is in the domain D a sampled weight O₂ generates images with consistent global shape but random texture forming the random domain D* Thus, by random weight sampling, we obtain an infinite number of random domains D1, D ......, D Input image intensities assumed to be standard normal distribution (O,, 1) (which is often true in practice thanks to data whitening). As the outputs of RandConv should follow the same distribution, sample the convolution weights from N (O0,0* x?) where 1/vCin x h x which is commonly applied for network initialization (He et al. 2015). We include the original images for training at a ratio p as hyperparameter.
4

---

Published as a conference paper at ICLR 2021
RCmi:: Mixing Variant As shown in Fig. outputs from RCing can vary significantly from the appearance of the original images. Although generalizing to domains with significantly different local texture distributions is useful, we may not want to sacrifice much performance on domains similar to the training domain. Inspired by the AugMix (Hendrycks et al. 2020b) strategy, we propose to blend the original image with the outputs of the RandConv layer via linear convex combinations al + (1 - (1 * ①), where is the mixing weight uniformly sampled from [0, 1].In RCmix. the RandConv outputs provide shape-consistent perturbations of the original images. Varying a, we continuously interpolate between the training domain and the randomly sampled domains of RCimg.
Multi-scale Texture Corruption As discussed in Sec. 3.I, image shape information at a scale smaller than filter's s size will be corrupted by RandConv. Therefore, we can use filters of varying sizes to preserve shapes at various scales. We choose to uniformly randomly sample filter size k from pool KC = 1, 3. ...n before sampling convolution weights θ E R4 XCout from Gaussian distribution ((0, F Fig. shows examples of multi-scale RandConv outputs.
Consistency Regularization To learn representations invariant to texture changes, we use loss encouraging consistent network predictions for the same RandConv-augmentt image for different random filter samples. Approaches for transform-invariant domain randomization (Yue et al. 2019). data augmentation (Hendrycks al., 2020b), and semi-supervised learning (Berthelot et al. 2019)
use similar strategies. We use Bellbacck-eeible (KL) divergence to measure consistency. However, enforcing prediction similarity of two augmented variants may be too strong. Instead, following (Hendrycks et al.) 2020b), we use RandConv to obtain 3 augmentation samples of image I: G,
RandConv' (I) for j 1,2, and obtain their predictions with a model 4: y = $(G)). ). We then compute the relaxed loss as A Ej=1 KL [])), where y = Σj= w /33 is the sample average.
4 EXPERIMENTS
Secs. 4.1 [0 4.3 evaluate our methods on the following datasets: multiple digit recognition datasets, PACS, and ImageNet-sketch. Sec. Sec.F4.4jur f.44uuss uses PACS to explore the out-of-domain generalization of a pretrained representation in transfer learning by checking if pretraining on ImageNet with our method improves the domain generalization performance in downstream tasks. All experiments are in the single-domain generalization setting where training and validation sets drawn from one domain. Additional experiments with ResNet18 8 as the backbone are given in the Appendix.
4.1 DIGIT RECOGNITION
The five digit recognition datasets (MNIST LeCunn ct al. 1998), ) MNIST-M (Ganin et al. 2016).
SVHN (Netzer et al.) 20 SYNTH (Ganin & Lempitsky 2014) and USPS (Denker ct al.
1989
have been widely used for domain adaptation and seneralization research (Peng et al.) 201 9a b Qiao et al.) 2020 Following the setups in (Volpi et al. 2018) and (Qiao et al., 2020), we train simple CNN with 10,000 000 MNIST samples and evaluate the accuracy on the test sets of the other four datasets. We also on MNIST-C (Mu & Gilmer 2019). robustness benchmark with 15 common
corruptions of MNIST and report the average accuracy over all corruptions.
71
75
73
73
Kagg
70
72
Azzuraty
71
89
88
87
68
0.3
1-3
1-5
1-7
1-9
1-11
20
(x) p Original data fraction
(b) Multi scale
(c) Weight consistency loss
Figure 2: Average accuracy and 5-run variance of MNIST model on MNIST-M, SVHN, SYNTH and USPS. Studies for: (a) original data fraction p for RCingi (b) multiscale design (I-n refers to using scales 1,3....n) for RCing,p=0.5 (orange) and RCmix (bluc); (c) consistency loss weight À for RC RCing1 7. (orange) and RCmia1-7 (blue).
5

---

Published as a conference paper at ICLR 2021
Selecting Hy Hyperparameters and Ablation Study. Fig. 2 aa shows the effect of the hyperparameter p on RCing with filter size 1. We see that adding only 10% RandConv data (p = 0.9) immediately improves the average performance (DG-Avg) on MNIST-M. SVHN, SYNTH and USPS performance from 53.53 to 69.19, outperforming all other approaches (see Tab. T for every dataset. We choose p 0.5, which obtains the best DG-Avg. Fig- 2b) shows results for multiscale ablation study. Increasing the pool of filter sizes up to improves DG-Avg performance. Therefore we use multi- scale 1-7 to study the consistency loss weight X, shown in Fig. 2 Adding the consistency loss improves both RandConv variants on DG-avg: RCmix1 -7 favors À = 10 while R..mg1-7.p=0.5 10 for all subsequent experiments.
performs similarly for À and X = 10. We choose
Results. Tab. compares the performance of 0.= and 3Cmmx1-7,A=11 with other state-of-the-an approaches. We show results of the adversarial training based methods GUD (Volpi et at. 2018). M-ADA (Qiao et al. (2020), and PAR (Wang et al. (019a). The baseline model is trained only on the standard classification loss. To show RandConv is more than a trivial color/contrast adjustment method, we also compare to ColorJitter data augmentation (which randomly changes image brightness, contrast, and saturation) and GreyScale (where images are transformed to grey- scale for training and testing). We also tested data augmentation with a fixed Laplacian of Gaussian filter (Band-Pase Passd-ass of size=3 and o = 1 and the data augmentation pipeline (Multi-Aug) that was used in recently proposed large scale study on domain generalization algorithms and datasets (Gulrajani] & Lopez-Paz 2020). RandConv and its mixing variant outperforms the best competing method (M- ADA) by 17% on DG-Avg and achieves the best 91.62% accuracy on MNIST-C. While the difference between the two variants of RandConv is marginal, RC 7, performs better on both DG-Avg and MNIST-C. When combined with Multi-Aug, RandConv achieves improved performance except on MNIST-C. Shows t-SNE image feature plots for unseen domains generated by the baseline approach and RC m0mxx1-7,x=10. The RandConv embeddings suggest better generalization to unseen domains.
Table I: Average accuracy and 5-run standard deviation (in parenthesis) of MNISTIOK model on MNIST-M, SVHN. SYNTH, USPS and their average (DG-avg); and average accuracy of 15 types of corruptions in MNIST-C. Both RandConv variants significantly outperform all other methods.
MNIST
MNIST-M
SVHN
USPS
SYNTH
DG-Avg
MNIST-C
Baseline
98.40m884)
58,87a.73
23.41
79.27(2.0)
42.43(5.46)
28.20(2.10 23.50k42)
GreyScale
98.82(m.02)
(8.41
36.06(148)
80.45(1000)
45.00ka.0)
89.15(044) 54.98m..))
ColorJitter
98.72...5))
62.72.0.66)
39.61(088)
79.18
46.40ko.4)
56.98/0.9)
89, 399.48(o.8)
BandPass
98.65
70.22(2.73)
48.34(2.5)
78.601a.2)
57.17(200)
63.58(.89)
87. (9.6))
MultiAug
98.80 (8.80(..s))
62.32:0.66)
39.07
79.31(1.2)
46.48(0.50)
56.79.0.4)
89. 19.111
PAR (our imp)
98.79(0.05)
61.160221)
36.08
79.95(118)
35.48(0.35)
55.67/0.3)
89.34
GUD
60.41
35.51
77.26
45.32
54.62
M-ADA
67.94
42,55
78.53
48.95
59.49
RCimgi-7 p=0.5, A=5
98. 98.866.05)
87.670.37)
(1.95(1.90)
82.08(146)
63.37(155)
72.02(1.55
90.94...51
RCmixi-7. 7,1-10
87.76.0.3) 98. (8.85
57.52(2.09
83, 96.36
68.88)))
91.62m.77 72.88
RCmix1-7,A== -,,Ax1-7,A=10 + MultiAng
98.82(0.05)
87.89
62.07(062)
84.39
63.90ma4))
74.56..46)
91.40093
4.2 PACS EXPERIMENTS
The PACS dataset Li et al. 2018b) considers 7-class classification on 4 domains: photo, art painting, cartoon, and sketch, with very different texture styles. Most recent domain generalization work studies the multi-source domain setting on PACS and uses domain labels of the training data. Although we follow the convention to train on domains and test on the fourth, we simply pool the data from the 3 training domains in (Wang al. 2019a), without using domain labels during the training.
Baseline and State-of-the-Art. Following (Li et al. 2017), we use Deep-All as the baseline, which finetunes an ImageNett AlexNet on 3 domains using only the classification loss and tests on the fourth domain. We test our RandConv variants RCimg1-T.p=0.3 and RCmix-77 with and without consistency loss, and datasets. We also implemented PAR (Wang et al. 2019a) using our baseline model. RCmix1-7
data augmentation as in the digit
"ee PyTorch documentation for implementation details; all parameters are set 0.5.

---

Published as a conference paper at ICLR 2021
MNIST-M
SVHN
USPS
SYNTH
Figure 3:
t-SNE feature embedding visualization for digit datasets for models trained on MNIST without (top) and with our RCmix1-7,A=110 approach (bottom). Different colors denote different classes.
Table 2: Mean and 5-run standard deviation (in parenthesis) results for domain generalization on PACS. Best results with our Deep-All baseline are in bold. The domain name in each column represents the target domain. Base column indicates different baselines and results under different baselines are not directly comparable. MLDG and CIDDF used domain labels for training.
Base
Method
Photo
Art
Cartoon
Sketch
Average
Deep-All
86.77
60.1 (1.33)
64.12
55.28
66.57(1.6)
GreyScale
83,93(447)
61.600.18
12
60.07.247
66.93
ColorJitter
(4.6)
59.01
61. 61.43
62.44<1.88)
66.88 66.88(0.33)
BandPass
87.08
59.46,277
(4.39(o.5))
55.39
66.58(0.33)
MultiAug
85.21 (0:47)
59.51
(1.0))
(1.677..76)
67.32.0.23)
Ours
PAR (our imp.)
(7.21
(7.17
63.63(0.58)
55.83
66.71
RCmmg1-7.p005
86,50
61. 10
64, (0.62)
68.50(183)
70.09(0.43)
RCmix1-7
86.60(o.7)
61. 50
(4.05(0.66
69.74(0.6))
70.53v.25)
RCmix1-7 MultiAug
86,231o74)
11.91
62.69
67.74(1.21
69.64(0.49
RCimg1-7.--05. A-10 RCaax1-7,A-10
81.15
59.56,079)
62.42,0.59
71.74k0.3)
68.72. (8.72)))
81.78(.11)
61.14
63.57
(1.97
(9.62.0.24)
Results below are not directly comparable due to different Deep-All implementations.
Deep-All (our run)
88.40
66.26
66.58
70.16
Wang et al
2019a
PAR (our run)
88.40
65.19
68.58
61.86
71.10
PAR (reported)
89.6
66.3
68.3
64.1
72.08
Deep-All
89.98
66.68
69.41
60.02
71.52
2019
Jigen
89.00
67.63
71.71
65.18
73.38
Deep-All
86.67
64.91
64.28
53.08
67.24
Li 2018a
MLDG (use domain
88.00
66.23
66.88
58.96
70.01
Deep-All
77.98
57.55
67.04
58.52
65.27
Li 2018c
CIDDG (use domaii labels)
78.65
62.70
69.73
64.45
68.88
combined with MultiAug is also tested. Further, we compare to the following ofate-of-theear approaches: Jigen (Carlucci et al., 2019) using self-supervision, MLDG (Li et al., 2018a] using meta-learning, and the conditional invariant deep domain generalization method CIDDG (Li et al. 2018c). Note that previous methods used different Deep-All baselines which make the final accuracy not directly comparable, and MLDG and CIDDG use domain labels for training.
Results. Tab, shows significant improvements on Sketch for both RandConv variants. Sketch is the most challenging domain with no color and much less texture compared to the other domains. The success on Sketch demonstrates that our methods can guide the DNN to learn global representations focusing shapes that robust to texture changes. Without using the consistency loss, RCmix1-7 achieves the best overall result improving over Deep-All by 44% but adding MultiAug does not further improve the performance. Adding the consistency loss with 10, RCmix1-7 and
7

---

Published as a conference paper at ICLR 2021
R0 performs better Sketch but degrades performance on the other 3 domains, so do GreyScale and ColorJitter. This observation will be discussed in Sec Sec 4.4
4.3 GENERALIZING AN IMAGENET MODEL TO IMAGENET-SKETCH Table 3: Accuracy of ImageNet-trained AlexNet ImageNet-Sketch (IN-S) data. Our methods outperform PAR by 5% and are on par with a was built on top of stronger baseline than our model, and both PAR and SIN fine-tuned the baseline model which helped the performance, while we train RandConv model from scratch.
(SIN) trained model. Note that PAR
Baseline Wang al. 20190
PAR Wang et 20190
Baseline
RCmix1-7, A-10
SIN
p=0.5AA=11
2019
Topl Top5
12.04 25.60
13.06 26.27
10.28
18.09
16.91 33.99
17.62
21.60
35.40
36.22
ImageNet-Sketch (Wang al. 2019a) is out-of-domain test set for models trained on ImageNet. We trained AlexNet from scratch with R. and RCmixI-7,A=10- We evaluate their performance on ImageNet-Sketch We use the AlexNet model trained without RandConv our baseline. Tab. B compares PAR and its baseline model and AlexNet trained with Stylized ImageNet (SIN) (Geirhos et al.] 2019) on ImageNet-Skettiis Although PAR uses a stronger bascline, RandConv achieves significant improvements over our baseline and outperforms PAR by large margin. Our methods achieve more than 7% accuracy improvement over the baseline and surpass PAR by 5%. SIN an image stylization approach that modify image texture in a hierarchical and realistic way. However, albeit its complexity, still performs on par with RandConv. Note that image stylization techniques require additional data and heavy precomputation. Further, the images for the style source also need to be chosen. In contrast, RandConv much easier to use: it can be applied to any dataset via a simple convolution layer. We also measure the shape-bite thape-bias metrie proposed by /eirhho et al.) (2019) for RandConv trained AlexNet. RCimg1-7.y --.pp11-----..-- 0-5,A=10 and RC improve the baseline from 25.36% 48.24% and 54.85% respectively.
4.4 REVISITING PACS WITH MORE ROBUST PRETRAINED REPRESENTATIONS
A common practice for many computer vision tasks (including the PACS benchmark) is transfer learning, i.e. finetuning backbone model pretrained on ImageNet. Recently, how the accuracy on ImageNet (Kornblith et al. 2019) and adversial robustness (Salman et al.] 2020 of the pretrained model affect transfer learning has been studied in the context of domain generalization. Instead, we study how out-of-doman generalizability transfers from pretraining to downstream tasks and shed light on how to better use pretrained models.
Impact of ImageNet Pretraining A model trained on ImageNet may be biased towards tex- tures (Geirhos et al. 2019). Finetuning ImageNet pretrained models on PACS may inherit this texture bias, thereby benefitting generalization but hurting performance on the Sketch domain. Therefore, as shown in Sec. 4.2 using RandConv to correct this texture bias improves results on Sketch, but degrades them Since pretraining has such a strong impact on transfer performance to new tasks, we ask: "Can the generalizability of a pretrained model transfer to downstream tasks? Le., does a pretrained model with better generalizability improve performance on unseen domains on new tasks?" To answer this, we revisit the PACS tasks based on Imaeeeett-reerrined weights where our two RandConv variants of Sec. 4.3 are used during ImageNet training. We study if this results in performance changes for the Deep- All baseline and for finetuning with RandConv.
the Photo domain (which is similar to ImageNet),
the Photo domain.
Better Performance via RandCony pretrained model We start by testing the Deep-All baselines using the two RandConv-trained ImageNet models of Sec. 4.3 as initialization. Tab. 4 shows significant improvements on Sketch. Results are comparable to finetuning with RandConv on normal pretrained model. Art is also consistently improved. Performance drops slightly on Photo as expected, since we reduced the texture bias in the pretrained model, which is helpful for the Photo domain. A similar performance improvement is observed when using the SIN- trained AlexNet as initialization. Using RandConv for both ImageNet training and PACS finetuning, accuracy on Sketch. As far as we know, this is the best performance using an AlexNet baseline. This approach even outperforms ligen (Carlucci et al. 2019 (71.35%) with stronger ResNet18 baseline
achieve 76. 1%
8

---

Published as a conference paper at ICLR 2021
Table 4: Generalization results on PACS with RandConv and SIN pretrained AlexNet. ImageNet column shows how the pretrained model is trained on ImageNet (baseline represents training the ImageNet model using only the classification loss); PACS column indicates the methods used for finetuning on PACS. Best and second best accuracy for each target domain are highlighted in bold and underlined.
PACS
ImageNet
Photo
Art
Cartoon
Sketch
Avg
Baseline
86.77
60.1
64. (2
55,28
66.57a2261
RCimg1 7.,=00.,A=10
84.48
62.61(123))
66. 100
69.24(0.0)
(0.61
Deep-All
RCmic1-TAA-1u
85.59%..4)
63.30(0.99)
(3.83
(8.29
70.25
SIN
85.33.0.6)
65.85
(5.39(0.22)
65.75(0.59)
70.58
Baseline
81.15
59.56.0.79
62.42.0.59)
71.74
68.72(0.5)
RCmmg--,
RCingt-7. D.00.- RC
84.3660.36
63.73/0.91)
68.07
(0.41
72.89
p-8.5,110
84.63.099)
63.41(1.22)
66,36,0433
74.59
72.25(0.4)
Baseline
81.78
61.14(0.51)
(3.57
71.97
69. 69.62(22))
RC A=10
D.
85.16))03)
(7..8)))38)
67, (7.68.0.60)
76.11(043)
73.03(0.46)
RCmi11-,22=10
170.56)
65.33
65.52(L.13)
73.21
72.56.0.50
model. Cartoon and Art are also improved. The best average domain generalization accuracy is 73.03% with more than 6% improvement over our initial Deep-All baseline.
This experiment confirms that generalizability may transfer: removing texture bias may not only make a pretrained model more generalizable, but it may help generalization downstream tasks. For similar target and pretraining domains like Photo and ImageNet, where learning texture bias may actually be beneficial, performance may degrade slightly.
CONCLUSION AND DISCUSSION
Randomized convolution (RandConv) is simple but powerful data augmentation technique for randomizing local image texture. RandConv helps focus visual representations on global shape information rather than local texture. We theoretically justified the approximate puapeepreeerving property of RandConv and developed RandConv techniques using multi-scale and mixing designs. We also make use of a consistency loss to encourage texture invariance. RandConv outperforms state-of-the-ar approaches on the digit recognition benchmark and on the sketch domain of PACS and on ImageNet-Sketee by large margin. By finetuning a model pretrained with RandConv on PACS, showed that the generalizability of pretrained model may transfer to and benefit new downstream task. This resulted in a new state-of-an art performance on PACS in the Sketch domain.
RandConv can help computer vision tasks when shape- shape-biased model is helpful e.g. for object detection. RandConv can also provide a shape-biased pretrained model to improve performance on downstream tasks when generalizing unseen domains. However, local texture features can be useful for many computer vision tasks, especially for fixed-domain fine-grained visual recognition. In such cases, visual representations that are invariant local texture may hurt in-domain performance. Therefore, important future work includes learning representations that disentangle shape and texture features and building models to use such representations in an explainable way.
Adversarial robustness of deep neural networks has received significant recent attention. Interestingly, Zhang& & Zhu (2019) find that adversarially-train trained models are more shape biased; (Shi et al. (2020) show that their method for increasing shape bias also helps adversarial robustness, especially when combined with adversarial training. Therefore, exploring how RandConv affects the adversarial robustness of models could be interesting future work. Moreover, recent biologically inspired models for improving adversarial robustness (Dapello et al. (020)) use Gabor filters with fixed random configurations followed by stochastic layer to add Gaussian noise to the network input, which may explain the importance of randomness in RandConv. Exploring connections between RandConv and biological mechanisms in the human visual system would be interesting future work.
Aeknowledgments We thank Zhiding Yu for discussions on initial ideas and the experimental setup. We also thank Nathan Cahill for advice proving the properties of random convolutions.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using retu-regulaiizatton. Advances Neural Information Processing Systems, pp. 998-1008, 2018
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5049-5059, 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network
distillation. In International Conference Learning Representations, 2019. URL https:
//openreview. net. fttp:/orummmmmo oi H
Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2229-2238, 2019.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems, 33, 2020.
John S Denker, WR Gardner, Hans Peter Graf, Donnic Henderson, Richard E Howard, W Hubbard, Lawrence D Jackel, Henry S Baird, and Isabelle Guyon. Neural network recognizer for hand- written zip code digits. In Advances in neural information processing systems, pp. 323-331, 1989.
Adam Gaier and David Ha. Weight agnostic neural networks. In Advances in Neural Information Processing Systems, pp. 5364-5378, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:/409. 1499.4999 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200 2030, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations. www. www.nnee/oormmmm.eeeeemmommm
2019. URL httpttttts///ppeeeeeteee
J09KX
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE transactions on pattern analysis and machine intelligence, 19(7):1414-1)) 2016
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kun He. Yan Wang, and John Hopcroft. A powerful generative model using random weights for the deep image representation. In Advances in Neural Information Processing Systems, pp. 631-639, 2016b.
David Heeger and James R Bergen. Pyramid-based texture analysis/synthesis. In Proceedings of the 22nd annual conference Computer graphics and interactive techniques, pp. 229-238. 1995.
10

---

Published as a conference paper at ICLR 2021
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-o--distributton generalization. arXiv preprint 262422006..6211, 2020a.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lak- shminarayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representutions, 2020b. URL https 7/openrevies net (tw.ormmmddllmmxxiiiii
Andrew Hyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
Simon Kornblihh, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement learning. In International Conference Learning Representations. http:://ppenreview net/forum, 2020.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang. and Alex Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 639, 2018c.
Yawei Luo, Liang Zheng. Tao Guan, Junqing Yu, and Yi Yang. Taking closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2507-2516 2019.
Norman Mu and Justin Gilmer. Mnist-c: robustness benchmark for computer vision. arXiv preprint arXiv: 023377 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8617-8629, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1416-1415. 2019a.
Xingchao Peng. Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled representations. In ICMI, 2019h.
11

---

Published as a conference paper at ICLR 2021
Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision, 40(1):49-70, 2000.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. arXiv preprint arXiv:2003.13211 2020.
Wullianallur Raghupathi and Viju Raghupathi. Big data analytics in healthcare: promise and potential. Health information science and systems, 2(1):3, 2014.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversari- ally robust imagenet models transfer better? arXiv preprint arXiv:2007.08489, 2020.
Andrew M Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. On random weights and unsupervised feature learning. In ICML, volume 2, pp. 2011.
Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-aaverrsrial discriminative deep domain generalization for face presentation attack detection. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 10023-10031, 2019.
William B Shen, Danfei Xu, Yuke Zhu, Leonidas J Guibas, Li Fei-Fei, and Silvio Savarese. Situational fusion of visual representation for visual navigation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 28812881220 1-2900, 2019.
Baifeng Shi, Dinghuai Zhang, Qi Dai, Zhanxing Zhu, Yadong Mu, and Jingdong Wang. Informative dropout for robust representation learning: A shape-bias perspective. In International Conference Machine Learning, 2020.
Patrice Y Simard, David Steinkraus, John C Platt, al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar, volume 3, 2003.
Baochen Sun and Kate Saenko. From virtual to reality: Fast adaptation of virtual object detectors to real domains. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Nguyen Xuan Vinh, Sarah Erfani, Sakrapee Paisitkriangkrai, James Bailey, Christopher Leckie, and Kotagiri Ramamohanarao. Training robust models using random projection. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 531-536. IEEE, 2016.
Riccardo Volpi. Hongseok Nankoong. Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems, pp. 3334-5344 2018
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019a.
Haohan Wang, Zexue He, and Eric Xing. Learning robust representations by projecting superficial statistics out. In International Conference on Learning Representations, 2019b. URL https
//openreview. n wttp://wwwww.eee//ooomeeeeeeeeee do
John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence
classification. In International Conference on Learning Representations, 2019. URL (hhttt
//openreview.co net. ftr.mmirummmmiie Bkkaaa//AA
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Lingiovanni-Vinceneele Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-reet generalization without accessing target domain data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2100-2110, 10, 2019.
Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural networks. In International Conference on Machine Learning, pp. 7502-7511, 2019.
12

---

Published as a conference paper at ICLR 2021
Figure 4: Left: An image with texture and shapes at different scales; Middle: The outpul of RandCoav with small filter size which largely preserves the shapes of the stones. Right: The output of RandConv with large filter size distorts shape the stones as well.
This supplementrry material provides additional details. Specifically, in Sec. A and B diseuss definitions of shapes and textures in images and justify why random convolution preserves global shapes and disrupts local texture formally by proving Theorem [T This theorem shows that random linear projections are approximately distance preserving. We also discuss our simulation-based bound based on 80% distance rescaling on real image data. Sec.Cproovdess more experimental details for the different datasets. Sec. D shows experimental results with stronger backbone architecture and on a new benchmark ImageNet-R (Hendrycks et al., 2020a). Sec. [E] provides more detailed results regarding hyperparameter selection and ablation studies. Lastly, Sec. shows example visualizations of RandConv outputs and for its mixing variant.
A SHAPES AND TEXTURE IN IMAGES
As discussed in the main text, we define shapes in images that are preserved by a random convolution layer as primitive shapes: spatial clusters of pixels with similar local texture. An object in image can be a single primitive shape alone but in most cases it is the composition shapes e.g.
multiple primitive includes wheels, body frames, windshields. Note that the definition of texture is not necessarily opposite to shapes, since the texture of a larger shape can includes smaller shapes. For example, in Fig 4 the left occluded triangle shape has texture composed by shapes of cobble stones while cobble stones have their own texture. Random convolution can preserve those large shapes that usually define the image semantics while distorting the small shapes local texture.
To formally define the prepe-preeeriig property, we assume (I1 (11,31), (I2, 82) and (æ3, ya) are three locations image and (#1, 31) has closer color and local texture with (I2, than (I3, y3). For example, (I1, y1) and (z22, V2) are within the same shape while (æ3, ys) is located at a neighboring shape. Then we have [|p(21, 3/1 p(22, U2)ll < |((1. 21, V1) p(rs, B where (( yt) is the image patch at location (I;, V:). A transformation f is shape-preserving if it maintains such relative distance relations for most location triplets, i.e.
[p v:)) f(p(z,, M5) l1/lll/,, v:) - p(z w;)1 ~
for any two spatial location (z;, and (Iy, 35): 2 0 is a constant.
B RANDOM CONVOLUTION IS SHAPE-PRESERVING AS RANDOM LINEAR PROJECTION IS DISTANCE PRESERVING
We express convolution layer local linear projection:
g(z, v) = Up(x.y).
(2)
where p(z,1) y) E Rd (d= - / g(æ, y) E IR Cout the output feature at location (z. y), and U E Cous is the matrix expressing the convolution layer filters @. Le.. for each sliding window centered at (x, y). convolution layer applics a linear transform f Rd Coet projecting the d dimensional local image patch p(æ, y) to its Covt dimensional feature g(r, y). When + is independently randomly sampled, e.g. from
Cin) is the vectorized image patch centerized at location (I, v).
13

---

Published as a conference paper at ICLR 2021
Gaussian distribution, the convolution layer preserves global shapes since that random linear projection is approximately Destance-preservin by bounding the range in Eq. Lin Theorem !I
Theorem 1. Suppose we have data points 2.... 2N Rd. Let f(z) = Uz be a random linear projection J -R_ I such that U E Rmxd and Ut, N(0, x²). Then we have:
{ris:= 1lf(z,) f(z,)l| ||
P
61) Se.
sup [JNJJI
1| 3/11
(3)
Tig: =- 11f(2;) [(2z)]]
P inf [SAJEIM
82) S.
12, 2z11
where 81 == o - MH-T
- (m) and 82 == o - (m). Here, (m) denotes the a-upper
quantile of the x² distribution with m degrees of freedom.
Thm. Ieell us that for any data pair (zi, 2;) in set of N points, the distance rescaling ratio ra after a random linear projection is bounded by 61 and 6n with probability 1 - e. A Smaller N and larger output dimension m give better bounds. E.g., when m = 3, N 1, 000, a 1 and c 0.1, 5, = 5.8 and 62 = 0.01. Thm. I gives theoretical bound for all the N(N 1)/2 pairs. However, in practice, preserving distances for a majority of N(N 1)/2 pairs sufficient. To empirically verify this, we test the range of central 80% of (ri,) on real image data. Using the same (m, N, W,a,e, c), 80% of the pairs lic in [0.56, 2.87]. which is significantly better than the strict bound: [0.01, 5.8]. A proof of the theorem and simulation details are given in the following.
Proof. Let U2 represent to the k-th row of U. It is easy check that V/ == (Us, z, 24.2211/122 2,1 ((0,0²). Therefore,
llf(z.) f(
1 (z.
2,)
U(z
21)
m
v
(m).
x2 12 2,
g²
1| 2z 11
2
Therefore, for 0 < 6 1, we have
P( 02 12 - 2, 112
-lf(z))- 12
2 2e
(m)) 5
2c
N(N-1)
N(N - 1)
From the above inequality, we have
P supixsA.JE[N] {
(1 112
x (m)
12: 2;
N(N-1)
P( SUPiAAii.ENN
[(2,))1(2)111
x2 N (ml
22,11
= =P U
1(2)))(())111
x2
N (m)}
-2|
ifjiJE[[N]
Σ
[(()))))())111
x(パー (m)
S
which is equivalent to
[f(z;) f(z,)))
sup [
> 0 /x22 (m)) S e.
[z: 2j
Similarly, we have
[lf(z.) - f(2;)]1
P inf (N
12 - 2
s
///
x- Ni-
(m) Se.
14

---

Published as a conference paper at ICLR 2021
Simulation Real Image Data To better understand the relative distance preservation property of random linear projections in practice. we use Algorithm 2 empirically obtain bound for real image data. We choose
and 0.1 in computing bounds. We use M = 1, 000 real images from the PACS dataset for this simulation. Note that the image patch size d does not affect the bound. We use patch size of 3 3 resulting in d 27. This simulation tell us that applying linear projections with a randomly sampled U on N local images patches in every image, have 1 chance that 80% of ri.j in the range [010% 890%].
3, N
000,
theoretical
Algorithm 2 Simulate the range of central 80% of r2,j on real image data
1: Input: M images (T number of data points N, projection output dimension m, standard deviation o of normal distribution, confidence level
2: for m 1 .M do
Sample images patches Lm at 1,000 locations and vectorize them as {2]" N.V. 4: Sample projection matrix U E Rm and U N (N,, o²)
5: for 1 do
for j = = N
Compute ri. = 11 1l
7:
where ((2 = Uz
8: 110% = 10% quantile of r for Im
190% 90% quantile of rij for T.
Get the central 80% of ray in each image
10: 810% = é quantile of all 910%
11: 890% = (1 - <) quantile of all 900%
D Get the confident bound for 410% and goo%
12: return 610%; 850%
C EXPERIMENTAL DETAILS
Digits Recognition The network for our digits recognition experiments is composed of two Conv5 x ReLU-MaxPool2x2 blocks with 64/128 output channels and three fully connected layer with 1024/1024/10 output channels. We train the network with batch size 32 for 10,000 iterations. During training, the model validated every 250 iterations and saved with the best validation score for testing. We apply the Adam optimizer with an initial learning rate of 0.0001.
PACS We use the official data splits for training/valie /alidatioon/tettinngg no extra data augmentation is applied. We use the official PyTorch implementation and the pretrained weights of AlexNet for our PACS experiments. AlextNet finetuned for 50,000 iterations with batch size 128. Samples are randomly selected from the training data mixed between the three domains. We use the validation data of source domains only at every 100 iterations. We use the SGD optimizer for training with an initial learning rate of 0.001, Nesterov momentum, and weight decay set to 0.0005. We let the learning rate decay by factor of 0.1 after finishing 80% the iterations.
ImageNet Following the PyTorch example training ImageNet models, set the batch size to 256 and train AlexNet from scratch for 90 epochs. We apply the SGD optimizer with an initial learning rate of 0.01, momentum 0.9, and weight decay 0.0001. We reduce the learning rate via factor of 0.1 every 30 epochs.
D MORE EXPERIMENTS WITH RESNET- 18
In this section, we demonstrate that RandConv also works on other stronger backbone architectures, e.g. for Residual Network He al. 6a). Specifically, we run the PACS and ImageNet exper- iments with R8 88 as the baseline and RandConv. As Table shows, RandConv improves the baseline using ResNet18 on Imagehet-sketch by 10.5% accuracy. When using RandConv pretrained ResNet-18 18 on PACS, the performance of finetuning with DeepAll and RandConv are both improved shown in Table 7 The best average domain generalization accuracy 84.09%, with a more than 8% improvement over our initial Deep-All baseline. A model pretrained with RC generally performs better than when pretrained with RCimg1-7,p=0.5..e We also provide the ResNet-18 performance of JiGen (Carlucci et al. 2019) PACS reference. Note
15

---

Published as a conference paper at ICLR 2021
that JiGen uses extra data augmentation and different data split than approach and it only improves over its own baseline by 1.5%. In addition, we test RandConv trained ResNet- on ImageNet-R (Hendrycks et al. 2020a), domain generalization benchmark that contains images of artistic renditions of 200 object classes from the original ImageNet dataset. As Table shows, RandConv also improve the generalization performance ImageNet-R and reduce the gap between the in-domain (1mageNet-200) and out-of-domain (ImageNet-R) performance.
Table 5: Accuracy of ImageNet-trained ResNet-18 on ImageNet-Sketod data.
Baseline
R 7-1005 A-10
RCmix1 A-10
Top1
20.23
28.79
30.70
Top5
37.26
49.02
51.80
Table 6: Top 1 Accuracy of ImageNet-trained ResNet- 18 on ImageNet-R data. ImageNet-200 the original ImageNet data with the same 200 classes as ImageNet-R.
Baseline
RGmg1-- A-05. A-10
RCmis1-7, A-10
ImageNet-200 (%)
88. 15
83.72
72.7
ImageNet-R (%)
33.06
37.38
35.75
Gap
55.09
46.34
36.95
Table Generalization results on PACS with RandConv pretrained model using ResNet-18. 18. Ima- geNet column shows how the pretrained model is trained on ImageNet (baseline represents training using only the classification loss); PACS column indicates the methods used for finetuning on PACS. Best and second best accuracy for each target domain are highlighted in bold and underlined. The performance of JiGen (Carlucci et al. 2019) and its baseline using ResNet-18 18 also given.
PACS
ImageNet
Photo
Art
Cartoon
Sketch
Avg
Baseline
95.45
74.96(.99
71.48(1.22)
62.09(112)
76.00to.77)
Deep-All
RCamg1-7. p.
94.65,o. 16)
73.85
74.78
73.51
79. 40)
RC RCmix1-7,A-10
94. 10
76.72(143)
73.41(1.29
77.60k..5)
80.46...4
Baseline
(2.37
76.50k..55)
71.33
79.65(1.2)
79. (9.96.)53))
RCmmg1-7
94.43(022)
79.80(1.0)
73.40ko.37)
81.5
82.28:0.38
A-10
RCmix1-7AA=10 94.57
81.32(1.00)
76.28
84.18
84.09
Baseline
93.57
77.73m99
71.24x..1)
75.53
79.52mm61
RCmixt-7 A=10
R0 RCmix1-7,A=10
95.23
80.56x082)
74. 74.18))33)
80.70(143)
82.67
95.01
81.09(1.4)
76.04mu2)
83.02/a.33
83.79hc.00)
Deep-All JiGen
95.73
77.85
74.86
67.74
79.05
Baseline
96.03
79.42
75.25
71.35
80.51
E HYPERPARAMETER SELECTIONS AND ABLATION STUDIES ON DIGITS RECOGNITION BENCHMARKS
We provide detailed experimental results for the digits recognition datasets. Table shows results for
different hyperameters for RCing1 Table 9 shows results for ablation study on the multi-scale design for RCmis and RCm...p=0.. Table 10 shows results for studying the consistency loss weight for RC RCmixi-7 and RC RCigg1-7,p=0.5 Tables 8 and [0 correspond to Fig. (a)(b)() in the main text respectively.
16

---

Published as a conference paper at ICLR 2021
Table 8: Ablation study of hyperparameter p for RCing1 digits recognition benchmarks. DG-Avg is the average performance on MNIST-M, SVHN, SYNTH and USPS. Best results are bold.
MNIST- 1OK
MNIST-M
SVHN
USPS
SYNTH
DG Avg
MNIST-C
Baseline
98.40.88)
58.87(3.3)
33.41(5.28
79.27
(2.43(3.46)
88.20 23.50
RGngg pm0.9
98.68.m.6)
(3.53
53.67(154)
80.38
59. 59.19ta.85
69.19
(9.79
Ring pw0.7
98.64..07)
84.17/0.61)
54.50
80.85
60.25
69. 69.94.0.50
89 60)
RCing 1. pwú.
98. 78..0))
85.17
55.97(0.54
80.3 (0.85)
61.07
70.63
88. (86666).)2)
RCmg11 p-03 pw0.3
98.71
85,45
54.62(1.5))
79.78:1.40
(a.41
70.09:0.0)
89 89.02(0.32)
RCmg1 p=0.1
98.660.06)
(..57
54.34(150)
79.21 c0.44)
60.18:0.6))
69.83
88.53
RCing1 peO
98.55%o.13)
86.27(0.2)
52.48(3.00)
79.01(1.11
59.53
69. 69.32(1.19)
88.01 (0.36)
Table 9: Ablation study of multi-scale RandConv on digits recognition benchmarks for RCmix and RCmmg.p=0.5- Best entries for each variant are bold.
MOIST-
MNIST-M
SVHN
USPS
SYNTH
DG Avg
MNIST-C
RCmixi
08.62.000
83.98...8)
53.26/2.5)
80.57(109)
59.25
69.26x..s
88. 38.59
RCmix1-3
98. 98.761002)
84.66x1.67)
55.89
80.95a.15
60.07a 05)
89 89.80ma.4) 70.39ka.55)
RC
98. 98.76)006)
84.32.043)
56.50/248)
81.85(1.8))
70.86xa.8)) 60.76..02)
90.06xaa0)
RCmix1-7
98. (8.82))00))
14.91
55.61(269)
82.09(1.0)
62.15
71.19(1.a)
90.30
R-mix1-9
98.8 10.12)
85.13/
18
82.07(128)
61.85(1.41)
70.8 (1.24)
90.83
RCingl- p-is
98.66 (8..06)0)))
(20.90)
55.59(029)
80.65a.71)
60.85
10.55(a.15)
89.00(a.45)
RC RCimgl-3. p-as
98.79)..07)
(5.3
55.60(1.9)
80.99(a.9)
61.261880)
70.80ku.0))
89. (0)20)
RCimg1-5. p-15 p-5s
98. 98.83
86.33xo.47
$4.99(2.4)
62.61 80.82(1.85)
71.19(1.5)
90.70
RCing1-7.pen po0. p505
98. 98.83()007)
86.08.0.27
54,93(127)
81.58..74)
62.78(0.6)
71.34:0.60)
91.18
RCing1-a. pe0.5
98.80 12)
85.63
52.82(200)
81.48(1.22)
62.55%0.4)
70.62
90.79ko.18)
Table 10: Ablation study of consistency loss weight À on digits recognition benchmarks for RCmix and RCimgi-7.p=0..5 DG-Avg is the average performance on MNIST-M, SVHN, SYNTH and USPS. Best results for each variant are bold.
X
10K
MNIST-M
SVHN
USPS
SYNTH
DG Avg
MNIST-C
20
98.90 (0:05)
87.18 (081) $7.68 (1.64) 83.55 (0.83)
63.08 (0:50)
72.87 (0.47)
91. (0.53)
10
98.85 (..00
87.76 (0.3)
57.52 (2.09)
83.36 (0.95)
62,88 (0.78)
72.88 (0.58)
91.62 (0.77)
R-
5
98.94 (0.09)
87.53 (0.1) 55.70 (2:22) 83.12 (1.08)
62.37 (0.98)
72.18
(1.00)
91.46 (0.50)
98.95 (0.05)
86.77 (0.79) 56.00 (2.39) 83.13 (0.71)
63.18 (0.7)) 72.27 (0.22)
91.15 (0.42)
0.1
98.84 (0:07)
85.41 (1.02) 56.51 (1.58)
81.84 (1.14)
61.86
(1.44)
71.41
(0.08)
90.72 (0.60)
0
98.82 (0:06)
84,91 (0.68)
55.61 (2.63)
82.09 (1.00)
62.15 (1.30)
71.19 (1.21)
90.30 (0.44)
20
98.79 (0.04)
87.53 (0.79)
53.92 (1.39)
81.83 (0.70)
62.16 (0.37)
71.36 (0.49)
91.20 (0.53)
10
98.86 (0.05)
87.67 (0:37)
54.95 (1.90)
82.08 (1.45)
63.37 (1.58)
72.02 (1.15)
90.94 (0.51)
5
98.90 (0:04)
87.77
55.00 (1.40)
82.10
63.58
72.11 (0.62)
90.83 (0.71)
98.86 (8:04)
86.74 (0.32) 53.26 (2.99)
81.51 (0.45)
62.00 (1.15) 70.88 (0.9))
91.11 (0.62)
0.1
98.85 (0.14)
86.85 (031)) 53.55 (363))
81,23 (1,02)
62.77
(0.00)
71.10
(131
91.13 (0.99
0
98.83 (0.07)
86.08 (0:27) 54.93 (122) 81.58
(0.74)
62.78
(0:86)
71.34
(0.61)
91.18 (0.38)
17

---

Published as a conference paper at ICLR 2021
F MORE EXAMPLES OF RANDCONY DATA AUGMENTATION
We provide additional examples of RandConv outputs for different convolution filter sizes in Fig. and for its mixing variants at scale that RandConv with different filter sizes retains shapes at different scales. The mixing strategy can continuously interpolate between the training domain and a randomly sampled domain.
with different mixing coefficients in Fig. | We observe
Input
0.9
0.7.
0.5
0.3
0.1
a
Figure 5: Examples of the RandConv mixing variant RCmir images of size 224 with different mixing coefficients a. When a = 1, the output just the original image input;when a 0, we use the output of random convolution layer as the augmented image.
18

---

Published as a conference paper at ICLR 2021
Original image
==
k=
==5
=
==
k
==
Figure 6: RandConv data augmentation examples on images of size 224 First column is the input image: following columns convolution results using random filters of different sizes k. We can see that the smaller filter sizes help maintain the finer shapes.
19


Reviews:
[{"review_id": "ePhHgHUU6Fv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper describes a method 'RandConv' which uses random convolutions to generate images with random textures but maintain global shape to improve training and generalization for classification tasks. The authors compare both random convolutions as images and mixing them with original training images. \n\nStrengths:\nThe authors demonstrate success with their method across various datasets as well as for pre-training experiments and evaluating on fine-tuning datasets. \n\nThe authors describe why their method should work as well as why their random convolutions preserve global shape. Far too often in ML papers we stop at 'this worked' and do not take the time to understand the why or how. \n\nThe authors compare their methods to other data augmentation methods and demonstrate improvements. \n\nThe t-SNE visualization of the difference in training is great evidence for this work and very clear from a reader standpoint. \n\nWeaknesses:\nThe authors neglect to compare across a significant amount of data augmentation methods, making it less clear how their method would compare. \n\nTheir method has worse performance in their PACS experiments for the Photo category and results vary significantly based on some algorithm choices in using their RandConv methods. For this to be more useful, a clearer understanding of the use cases is needed. \n\nMinor comment: typo on page 5 'w train a simple CNN' under 4.1 paragraph 1\n\nThe authors present a novel and interesting approach for data augmentation that can be directly coded into CNNs and demonstrate improvement for some tasks. I recommend accepting this work, although more work needs to be done to understand when and how to use their methods. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jnn6wx7ExqU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update\n-------\n\nI've updated by score in light of the discussion; as I said in the comments, from a purely experimental point of view there are good results, however the presentation of the paper confounds too many aspects. If the authors can address the terminology issues then it would make the work stronger.\n\n\nSummary\n-------\n\nThis paper proposes the use of a random convolution layer at the beginning of a network during training. The weights of this layer are not learned, and are changed randomly each time an input is passed through. In the experiments presented the layer has 3 input and output channels, so that the effect of the convolution is to jumble-up colour and modify local texture (at a scale determined by the size of the convolution kernel, which is itself uniformly sampled from a set of possible scales). During training the objective is modified with an additional loss term that intends to ensure that the distribution of network outputs from three independent forward passes of the network with the random convolution are consistent with each other. The normal task loss is just computed on the basis of the output from one of the forward passes.  The authors posit that this approach will lead to increased robustness and generalisation of the network as if forces a bias towards shape and away from texture (and colour), and demonstrate this with a wide range of experimental results.\n\n\nPositives\n---------\n\n- A wide range of metrics and datasets has been used to validate the performance of the proposed approach, and the evaluation appears thorough with respect to comparing against state-of-the-art approaches.\n- Aside from a slight lack of detail on the testing regime (see questions below), which is easily corrected, the paper itself reads very well.\n- The results do show that the proposed approach can have significant performance gains in a number of tasks.\n\n\nConcerns\n--------\n\n- The proposed method confounds texture and colour; these are different things and should be treated as such. One of the likely reasons for the reduction in performance on photos is for classes where colour is important to choosing the correct class. This probably also explains why mixing might be important/useful in some cases. I'm confident that if the authors had addressed this they would see further improvements in their results.\n- Whilst I agree that the random convolutions would obviously bias a network to learn a shape bias due to the destruction of local texture and colour information, using random convolutions to do this seems incredibly arbitrary given the myriad of possible ways of achieving this. For example, a trivial way to achieve the same thing would be to use a bandpass filter such as a Laplacian of Gaussian (coupled with random mixing of the colour channels if you really wanted to remove the colour information too). Such an approach could also trivially be made multi-scale if desired, and would be justifiable (e.g. Marr from a neuroscience perspective; Lindeberg from scale-space theory). \n- No adequate baseline has been provided; following on from the above point, I would have at least expected experimental comparison against a _simple_ baseline to demonstrate the advantages of the proposed technique (e.g. using a fixed bandpass filter instead of a random convolution perhaps).\n\n\nRationale for score\n-------------------\n\nAs it stands, although this paper clearly provides a method that gives good results, I find it difficult to give a high score because of the arbitrariness of the proposed method and the lack of justification for the approach (other than \"it seems to work\"). I am also strongly concerned by the way the paper confuses texture and colour.\n\nQuestions during rebuttal period \n--------------------------------\n\n- Just to confirm: what is the testing set-up used? Do you input the raw image data into the network without passing through the RandConv? \n- Have you measured the shape bias (Geirhos'19) of networks trained using RandConv? Do the networks show bias to shape over texture using this metric?", "extra_scores": {}, "extra_text": {}}, {"review_id": "7GQvGH93Mtw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper uses random convolution augmentation to learn representation that is robust for transferring across domains.\n\nQuality: The paper is well-written, easy to following. The method is presented clearly, and the enough experimental evaluation was conducted. Overall, the quality is good.\n\nClarity: The method is straightforward, and the authors describe it clearly. I think the idea is reasonable, but there is something to clarify in experiments:\n1. Compared with [1], all the methods have a drop in Table 2. Could you explain why? In fact, Table 2 is really confusing, the bolded numbers are not the best numbers.\n2. I notice that in [1], the performance of PAR can be further improved with strong data augmentation. In Table 2, the method is compared with Grey Scale/ Color Jitter. I wonder if this is fair comparison, since RandConv composes various random convolution schemes and mix. I wonder how the performance will change when the baseline uses stronger augmentation, for example, combining multiple data augmentation like self-supervised learning [2].\n3. Just wondering: will your method result in a network with higher robustness against adversarial attacks?\n\nOriginality: As far as I know, the method is new.\n\nSignificance: I think this paper proposes a simple yet effective method. The use of random convolutions is interesting. I assume this work can be a good contribution to the community if they can refine their experimental evaluation to better justify the effectiveness of their method. \n\nReference:\n[1] Learning Robust Global Representations by Penalizing Local Predictive Power https://arxiv.org/pdf/1905.13549.pdf\n[2] A Simple Framework for Contrastive Learning of Visual Representations https://arxiv.org/abs/2002.05709", "extra_scores": {}, "extra_text": {}}, {"review_id": "4BQoSUinQx_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple way to increase the robustness of the learned representations in a network perform a series of object recognition tasks by adding a random convolution layer as a pre-processing stage, thus “filtering the image” and preserving the global shape but altering the local `texture’ of the newly transformed image. Here, the hope is that  -- analogous to Geirhos et al. 2019 that induces a shape bias by transforming the image distribution into a new one with altered *global* textures that induce a shape bias and increases general robustness to o.o.d distortions --  the authors here go about doing something similar at the local level given the small size of the receptive field of the filter, thus preserving the shape and slightly altering “the texture”.\n\nPros:\n* While the innovation is simple and efficient, this data-augmentation scheme works, and I can see how other future works may use this as well as a data-augmentation technique for object recognition. I am not sure however if no one else has explored the effects of random convolutions for robustness. It sounds too good to be true, but then again -- there is always beauty in simplicity and it is possible that the authors have hit the nail on the head on finding a somewhat ‘contrived’ filtering process as a bonus rather than a limitation. Simple, yet counter-intuitive findings like these are relevant for ICLR.\n* Authors provide lots of experiments that to some degree prove the success of their augmentation strategy (although see Cons).\n\nCons:\n* Biological Inspiration: What is the biological mechanism linked to the success of using random convolutions. One could argue that this point is ‘irrelevant’ to the authors and the readers, but as there is a plethora of different data-augmentation techniques to choose from, why should computer vision and machine learning practitioners choose this one? (See Missing Reference for a suggestion)\n* Insufficient/Incomplete Baseline: The model is inspired loosely by Geirhos et al. 2019; but how does the model compete with Geirhos’ et al.’s Stylized ImageNet? I would have wanted to see a baseline between the authors proposed model and other texture-based augmentation strategies. This would elucidate the Global vs Local advantages of “texture”/style transfer on learned representations. I think this is where authors could capitalize more on.\n* The word `texture’ in the paper is a mis-nomer. Here what is really done is 1st order filtering via a convolution operation with a filter that does not happen to have a Gabor-like shape. “Texture” in other contexts going back to vision science and even computer vision and image processing (style transfer included), is usually computed by a set of cross-correlations between *outputs* of a filtered image (analogous to the Gramian Matrix of Gatys et al. 2015), or the principled Portilla-Simoncelli texture model from 1999.\n\nMissing references:\n* Excessive Invariance increases adversarial vulnerability by Jacobsen et al. ICLR 2019. The augmentation procedure proposed by the authors shows robustness to common distortions, but how about adversarial robustness? Is this relevant? Was this tried? I’d love to hear more about the authors thoughts on this to potentially raise my score.\n* Emergent Properties of Foveated Perceptual Systems (link: https://openreview.net/forum?id=2_Z6MECjPEa): An interesting concurrent submission to this year's ICLR has shown that the biological mechanism of visual crowding (that resembles texture computation for humans in the visual periphery) is linked to some of the operations introduced in the paper by the authors. It would be great if the authors potentially cite similar (and/or the before-mentioned) works to provide a link to a biological mechanism that may support why their data-augmentation procedure works and/or should be used; otherwise it seems contrived and could be seen as “yet another data-augmentation procedure that increases robustness but we don’t know why”.\n* Implementing a Primary Visual Cortex in the retina increases adversarial robustness by Dapello, Marques et al. 2020 (NeurIPS). This recently published paper in a way shows almost the opposite of what the authors are proposing here. Rather than using random convolutions, they actually mimic the gamut of spatial frequency tuning properties of Gabor filters in the first stages of convolution as done in human/monkey V1. The authors should discuss how their results fit with Dapello, Marques et al. 2020 and how they can reconcile their somewhat opposing views.\n\nFinal Assessment:\nI am on the fence of having this paper accepted at ICLR given the limitations expressed above, but I do like it’s simplicity that should not take away it’s merit -- thus my slight lean towards acceptance. I am willing to raise my score however if authors address some of the cons/limitations, and am also curious to see the opinion from other reviewers, it is possible that I may have missed a key reference regarding data-augmentation that may weaken my assessment.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers concurred that this is an interesting paper with contributions worthy of publication. The authors also provided many details in the rebuttal which makes the paper even more strong.
Decision: Accept (Poster)