Year: 2021

Abstract:
While successful for various computer vision tasks, deep neural networks have shown to be vulnerable to texture style shifts and small perturbations to which humans are robust. In this work, we show that the robustness of neural networks can be greatly improved through the use of random convolutions as data augmentation. Random convolutions are approximately shape-preserving and may distort local textures. Intuitively, randomized convolutions create an infinite number of new domains with similar global shapes but random local texture. Therefore, we explore using outputs of multi-scale random convolutions as new images or mixing them with the original images during training. When applying a network trained with our approach to unseen domains, our method consistently improves the performance on domain generalization benchmarks and is scalable to ImageNet. In particular, in the challenging scenario of generalizing to the sketch domain in PACS and to ImageNet-Sketch, our method outperforms state-of-art methods by a large margin. More interestingly, our method can benefit downstream tasks by providing a more robust pretrained visual representation. 1

heading: 1 INTRODUCTION
text: Generalizability and robustness to out-of-distribution samples have been major pain points when applying deep neural networks (DNNs) in real world applications (Volpi et al., 2018). Though DNNs are typically trained on datasets with millions of training samples, they still lack robustness to domain shift, small perturbations, and adversarial examples (Luo et al., 2019). Recent research has shown that neural networks tend to use superficial features rather than global shape information for prediction even when trained on large-scale datasets such as ImageNet (Geirhos et al., 2019). These superficial features can be local textures or even patterns imperceptible to humans but detectable to DNNs, as is the case for adversarial examples (Ilyas et al., 2019). In contrast, image semantics often depend more on object shapes rather than local textures. For image data, local texture differences are one of the main sources of domain shift, e.g., between synthetic virtual images and real data (Sun & Saenko, 2014). Our goal is therefore to learn visual representations that are invariant to local texture and that generalize to unseen domains. While texture and color may be treated as different concepts, we follow the convention in Geirhos et al. (2019) and include color when talking about texture.
We address the challenging setting of robust visual representation learning from single domain data. Limited work exists in this setting. Proposed methods include data augmentation (Volpi et al., 2018; Qiao et al., 2020; Geirhos et al., 2019), domain randomization (Tobin et al., 2017; Yue et al., 2019), self-supervised learning (Carlucci et al., 2019), and penalizing the predictive power of low-level network features (Wang et al., 2019a). Following the spirit of adding inductive bias towards global shape information over local textures, we propose using random convolutions to improve the robustness to domain shifts and small perturbations. While recently Lee et al. (2020) proposed a similar technique for improving the generalization of reinforcement learning agents in
1Code is available at https://github.com/wildphoton/RandConv.
unseen environments, we focus on visual representation learning and examine our approach on visual domain generalization benchmarks. Our method also includes the multiscale design and a mixing variant. In addition, considering that many computer vision tasks rely on training deep networks based on ImageNet-pretrained weights (including some domain generalization benchmarks), we ask “Can a more robust pretrained model make the finetuned model more robust on downstream tasks?” Different from (Kornblith et al., 2019; Salman et al., 2020) who studied the transferability of a pretrained ImageNet representation to new tasks while focusing on in-domain generalization, we explore generalization performance on unseen domains for new tasks.
We make the following contributions:
• We develop RandConv, a data augmentation technique using multi-scale random-convolutions to generate images with random texture while maintaining global shapes. We explore using the RandConv output as training images or mixing it with the original images. We show that a consistency loss can further enforce invariance under texture changes. • We provide insights and justification on why RandConv augments images with different local
texture but the same semantics with the shape-preserving property of random convolutions. • We validate RandConv and its mixing variant in extensive experiments on synthetic and real-
world benchmarks as well as on the large-scale ImageNet dataset. Our methods outperform single domain generalization approaches by a large margin on digit recognition datasets and for the challenging case of generalizing to the Sketch domain in PACS and to ImageNet-Sketch. • We explore if the robustness/generalizability of a pretrained representation can transfer. We
show that transferring a model pretrained with RandConv on ImageNet can further improve domain generalization performance on new downstream tasks on the PACS dataset.

heading: 2 RELATED WORK
text: Domain Generalization (DG) aims at learning representations that perform well when transferred to unseen domains. Modern techniques range between feature fusion (Shen et al., 2019), metalearning (Li et al., 2018a; Balaji et al., 2018), and adversarial training (Shao et al., 2019; Li et al., 2018b). Note that most current DG work (Ghifary et al., 2016; Li et al., 2018a;b) requires a multisource training setting to work well. However, in practice, it might be difficult and expensive to collect data from multiple sources, such as collecting data from multiple medical centers (Raghupathi & Raghupathi, 2014). Instead, we consider the more strict single-domain generalization DG setting,
where we train the model on source data from a single domain and generalize it to new unseen domains (Carlucci et al., 2019; Wang et al., 2019b).
Domain Randomization (DR) was first introduced as a DG technique by Tobin et al. (2017) to handle the domain gap between simulated and real data. As the training data in (Tobin et al., 2017) is synthesized in a virtual environment, it is possible to generate diverse training samples by randomly selecting background images, colors, lighting, and textures of foreground objects. When a simulation environment is not accessible, image stylization can be used to generate new domains (Yue et al., 2019; Geirhos et al., 2019). However, this requires extra effort to collect data and to train an additional model; further, the number of randomized domains is limited by the number of predefined styles.
Data Augmentation has been widely used to improve the generalization of machine learning models (Simard et al., 2003). DR approaches can be considered a type of synthetic data augmentation. To improve performance on unseen domains, Volpi et al. (2018) generate adversarial examples to augment the training data; Qiao et al. (2020) extend this approach via meta-learning. As with other adversarial training algorithms, significant extra computation is required to obtain adversarial examples.
Learning Representations Biased towards Global Shape Geirhos et al. (2019) demonstrated that convolutional neural networks (CNNs) tend to use superficial local features even when trained on large datasets. To counteract this effect, they proposed to train on stylized ImageNet, thereby forcing a network to rely on object shape instead of textures. Wang et al. improved out-of-domain performance by penalizing the correlation between a learned representation and superficial features such as the gray-level co-occurrence matrix (Wang et al., 2019b), or by penalizing the predictive power of local, low-level layer features in a neural network via an adversarial classifier (Wang et al., 2019a). Our approach shares the idea that learning representations invariant to local texture helps generalization to unseen domains. However, RandConv avoids searching over many hyper-parameters, collecting extra data, and training other networks. It also scales to large-scale datasets since it adds minimal computation overhead.
Random Mapping in Machine Learning Random projections have also been effective for dimensionality reduction based on the distance-preserving property of the Johnson–Lindenstrauss lemma (Johnson & Lindenstrauss, 1984). (Vinh et al., 2016) applied random projections on entire images as data augmentation to make neural networks robust to adversarial examples. Lee et al. (2020) recently used random convolutions to help reinforcement learning (RL) agents generalize to new environments. Neural networks with fixed random weights can encode meaningful representations (Saxe et al., 2011) and are therefore useful for neural architecture search (Gaier & Ha, 2019), generative models (He et al., 2016b), natural language processing (Wieting & Kiela, 2019), and RL (Osband et al., 2018; Burda et al., 2019). In contrast, RandConv uses non-fixed randomly-sampled weights to generate images with different local texture.

heading: 3 RANDCONV: RANDOMIZE LOCAL TEXTURE AT DIFFERENT SCALES
text: We propose using a convolution layer with non-fixed random weights as the first layer of a DNN during training. This strategy generates images with random local texture but consistent shapes, and is beneficial for robust visual representation learning. Sec. 3.1 justifies the shape-preserving property of a random convolution layer. Sec. 3.2 describes RandConv, our data augmentation algorithm using a multi-scale randomized convolution layer and input mixing.

heading: 3.1 A RANDOM CONVOLUTION LAYER PRESERVES GLOBAL SHAPES
text: Convolution is the key building block for deep convolutional neural networks. Consider a convolution layer with filters Θ ∈ Rh×w×Cin×Cout with an input image I ∈ RH×W×Cin , where H and W are the height and width of the input and Cin and Cout are the number of feature channels for the input and output, and h and w are the height and width of the layer’s filter. The output (with appropriate input padding) will be g = I ∗Θ with g ∈ RH×W×Cout . In images, nearby pixels with similar color or texture can be grouped into primitive shapes that represent parts of objects or the background. A convolution layer linearly projects local image patches to features at corresponding locations on the output map using shared parameters. While a
convolution with random filters can project local patches to arbitrary output features, the output of a random linear projection approximately preserves relative similarity between input patches, proved in Appendix B. In other words, since any two locations within the same shape have similar local textures in the input image, they tend to be similar in the output feature map. Therefore, shapes that emerge in the output feature map are similar to shapes in the input image provided that the filter size is sufficiently small compared to the size of a typical shape.
In other words, the size of a convolution filter determines the smallest shape it can preserve. For example, 1x1 random convolutions preserve shapes at the single-pixel level and thus work as a random color mapping; large filters perturb shapes smaller than the filter size that are considered local texture of a shape at this larger scale. See Fig. 1 for examples. More discussion and a formal proof are in Appendix A and B.

heading: 3.2 MULTI-SCALE IMAGE AUGMENTATION WITH A RANDOMIZED CONVOLUTION LAYER
text: Algorithm 1 Learning with Data Augmentation by Random Convolutions
1: Input: Model Φ, task loss Ltask, training images {Ii}Ni=1 and their labels {yi}Ni=1, pool of filter sizes K = {1, ..., n}, fraction of original data p, whether to mix with original images, consistency loss weight λ 2: function RANDCONV(I, K, mix, p) 3: Sample p0 ∼ U(0, 1) 4: if p0 < p and mix is False then 5: return I . When not in mix mode, use the original image with probability p 6: else 7: Sample scale k ∼ K 8: Sample convolution weights Θ ∈ Rk×k×3×3 ∼ N(0, 1
3k2 )
9: Irc = I ∗Θ . Apply convolution on I 10: if mix is True then 11: Sample α ∼ U(0, 1) 12: return αI + (1− α)Irc . Mix with original images 13: else 14: return Irc 15: Learning Objective: 16: for i = 1→ N do 17: for j = 1→ 3 do 18: ŷji = Φ(RandConv(Ii)) . Predict labels for three augmented variants of the same image 19: Lcons = λ ∑3 j=1 KL(ŷ j i ||ȳi) where ȳi = ∑3 j=1 ŷ j i /3 . Consistency Loss 20: L = Ltask(ŷ1i , yi) + λLcons . Learning with the task loss and the consistency loss
Sec. 3.1 discussed how outputs of randomized convolution layers approximately maintain shape information at a scale larger than their filter sizes. Here, we develop our RandConv data augmentation technique using a randomized convolution layer with Cout = Cin to generate shape-consistent images with randomized texture (see Alg. 1). Our goal is not to use RandConv to parameterize or represent texture as in previous filter-bank based texture models (Heeger & Bergen, 1995; Portilla & Simoncelli, 2000). Instead, we only use the three-channel outputs of RandConv as new images with the same shape and different “style” (loosely referred to as "texture"). We also note that, a convolution layer is different from a convolution operation in image filtering. Standard image filtering applies the same 2D filter on three color channels separately. In contrast, our convolution layer applies three different 3D filters and each takes all color channels as input and generates one channel of the output. Our proposed RandConv variants are as follows:
RCimg: Augmenting Images with Random Texture A simple approach is to use the randomized convolution layer outputs, I ∗Θ, as new images; where Θ are the randomly sampled weights and I is a training image. If the original training data is in the domain D0, a sampled weight Θk generates images with consistent global shape but random texture forming the random domain Dk. Thus, by random weight sampling, we obtain an infinite number of random domains D1, D1, . . . , D∞. Input image intensities are assumed to be a standard normal distribution N(0, 1) (which is often true in practice thanks to data whitening). As the outputs of RandConv should follow the same distribution, we sample the convolution weights from N(0, σ2) where σ = 1/ √ Cin × h× w, which is commonly applied for network initialization (He et al., 2015). We include the original images for training at a ratio p as a hyperparameter.
RCmix: Mixing Variant As shown in Fig. 1, outputs from RCimg can vary significantly from the appearance of the original images. Although generalizing to domains with significantly different local texture distributions is useful, we may not want to sacrifice much performance on domains similar to the training domain. Inspired by the AugMix (Hendrycks et al., 2020b) strategy, we propose to blend the original image with the outputs of the RandConv layer via linear convex combinations αI + (1 − α)(I ∗Θ), where α is the mixing weight uniformly sampled from [0, 1].In RCmix, the RandConv outputs provide shape-consistent perturbations of the original images. Varying α, we continuously interpolate between the training domain and the randomly sampled domains of RCimg.
Multi-scale Texture Corruption As discussed in Sec. 3.1„ image shape information at a scale smaller than a filter’s size will be corrupted by RandConv. Therefore, we can use filters of varying sizes to preserve shapes at various scales. We choose to uniformly randomly sample a filter size k from a pool K = 1, 3, ...n before sampling convolution weights Θ ∈ Rk×k×Cin×Cout from a Gaussian distribution N(0, 1k2Cin ). Fig. 1 shows examples of multi-scale RandConv outputs.
Consistency Regularization To learn representations invariant to texture changes, we use a loss encouraging consistent network predictions for the same RandConv-augmented image for different random filter samples. Approaches for transform-invariant domain randomization (Yue et al., 2019), data augmentation (Hendrycks et al., 2020b), and semi-supervised learning (Berthelot et al., 2019) use similar strategies. We use Kullback-Leibler (KL) divergence to measure consistency. However, enforcing prediction similarity of two augmented variants may be too strong. Instead, following (Hendrycks et al., 2020b), we use RandConv to obtain 3 augmentation samples of image I: Gj = RandConvj(I) for j = 1, 2, 3 and obtain their predictions with a model Φ: yj = Φ(Gj). We then compute the relaxed loss as λ ∑3 j=1 KL(y j ||ȳ), where ȳ = ∑3 j=1 y j/3 is the sample average.

heading: 4 EXPERIMENTS
text: Secs. 4.1 to 4.3 evaluate our methods on the following datasets: multiple digit recognition datasets, PACS, and ImageNet-sketch. Sec. 4.4 uses PACS to explore the out-of-domain generalization of a pretrained representation in transfer learning by checking if pretraining on ImageNet with our method improves the domain generalization performance in downstream tasks. All experiments are in the single-domain generalization setting where training and validation sets are drawn from one domain. Additional experiments with ResNet18 as the backbone are given in the Appendix.

heading: 4.1 DIGIT RECOGNITION
text: The five digit recognition datasets (MNIST (LeCun et al., 1998), MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), SYNTH (Ganin & Lempitsky, 2014) and USPS (Denker et al., 1989)) have been widely used for domain adaptation and generalization research (Peng et al., 2019a;b; Qiao et al., 2020). Following the setups in (Volpi et al., 2018) and (Qiao et al., 2020), we train a simple CNN with 10,000 MNIST samples and evaluate the accuracy on the test sets of the other four datasets. We also test on MNIST-C (Mu & Gilmer, 2019), a robustness benchmark with 15 common corruptions of MNIST and report the average accuracy over all corruptions.
Selecting Hyperparameters and Ablation Study. Fig. 2(a) shows the effect of the hyperparameter p on RCimg with filter size 1. We see that adding only 10% RandConv data (p = 0.9) immediately improves the average performance (DG-Avg) on MNIST-M, SVHN, SYNTH and USPS performance from 53.53 to 69.19, outperforming all other approaches (see Tab. 1) for every dataset. We choose p = 0.5, which obtains the best DG-Avg. Fig. 2(b) shows results for a multiscale ablation study. Increasing the pool of filter sizes up to 7 improves DG-Avg performance. Therefore we use multiscale 1-7 to study the consistency loss weight λ, shown in Fig. 2(c). Adding the consistency loss improves both RandConv variants on DG-avg: RCmix1−7 favors λ = 10 while RCimg1−7,p=0.5 performs similarly for λ = 5 and λ = 10. We choose λ = 10 for all subsequent experiments.
Results. Tab. 1 compares the performance of RCimg1−7,p=0.5,λ=10 and RCmix1−7,λ=10 with other state-of-the-art approaches. We show results of the adversarial training based methods GUD (Volpi et al., 2018), M-ADA (Qiao et al., 2020), and PAR (Wang et al., 2019a). The baseline model is trained only on the standard classification loss. To show RandConv is more than a trivial color/contrast adjustment method, we also compare to ColorJitter2 data augmentation (which randomly changes image brightness, contrast, and saturation) and GreyScale (where images are transformed to greyscale for training and testing). We also tested data augmentation with a fixed Laplacian of Gaussian filter (Band-Pass) of size=3 and σ = 1 and the data augmentation pipeline (Multi-Aug) that was used in a recently proposed large scale study on domain generalization algorithms and datasets (Gulrajani & Lopez-Paz, 2020). RandConv and its mixing variant outperforms the best competing method (MADA) by 17% on DG-Avg and achieves the best 91.62% accuracy on MNIST-C. While the difference between the two variants of RandConv is marginal, RCmix1−7,λ=10 performs better on both DG-Avg and MNIST-C. When combined with Multi-Aug, RandConv achieves improved performance except on MNIST-C. Fig 3 shows t-SNE image feature plots for unseen domains generated by the baseline approach and RCmix1−7,λ=10. The RandConv embeddings suggest better generalization to unseen domains.

heading: 4.2 PACS EXPERIMENTS
text: The PACS dataset (Li et al., 2018b) considers 7-class classification on 4 domains: photo, art painting, cartoon, and sketch, with very different texture styles. Most recent domain generalization work studies the multi-source domain setting on PACS and uses domain labels of the training data. Although we follow the convention to train on 3 domains and to test on the fourth, we simply pool the data from the 3 training domains as in (Wang et al., 2019a), without using domain labels during the training.
Baseline and State-of-the-Art. Following (Li et al., 2017), we use Deep-All as the baseline, which finetunes an ImageNet-pretrained AlexNet on 3 domains using only the classification loss and tests on the fourth domain. We test our RandConv variants RCimg1-7,p=0.5 and RCmix1-7 with and without consistency loss, and ColorJitter/GreyScale/BandPass/MultiAug data augmentation as in the digit datasets. We also implemented PAR (Wang et al., 2019a) using our baseline model. RCmix1-7
2See PyTorch documentation for implementation details; all parameters are set to 0.5.
combined with MultiAug is also tested. Further, we compare to the following state-of-the-art approaches: Jigen (Carlucci et al., 2019) using self-supervision, MLDG (Li et al., 2018a) using meta-learning, and the conditional invariant deep domain generalization method CIDDG (Li et al., 2018c). Note that previous methods used different Deep-All baselines which make the final accuracy not directly comparable, and MLDG and CIDDG use domain labels for training.
Results. Tab. 2 shows significant improvements on Sketch for both RandConv variants. Sketch is the most challenging domain with no color and much less texture compared to the other 3 domains. The success on Sketch demonstrates that our methods can guide the DNN to learn global representations focusing on shapes that are robust to texture changes. Without using the consistency loss, RCmix1-7 achieves the best overall result improving over Deep-All by∼4% but adding MultiAug does not further improve the performance. Adding the consistency loss with λ = 10, RCmix1-7 and
RCimg1-7,p=0.5 performs better on Sketch but degrades performance on the other 3 domains, so do GreyScale and ColorJitter. This observation will be discussed in Sec 4.4.

heading: 4.3 GENERALIZING AN IMAGENET MODEL TO IMAGENET-SKETCH
text: ImageNet-Sketch (Wang et al., 2019a) is an out-of-domain test set for models trained on ImageNet. We trained AlexNet from scratch with RCimg1-7,p=0.5,λ=10 and RCmix1-7,λ=10. We evaluate their performance on ImageNet-Sketch. We use the AlexNet model trained without RandConv as our baseline. Tab. 3 compares PAR and its baseline model and AlexNet trained with Stylized ImageNet (SIN) (Geirhos et al., 2019) on ImageNet-Sketch. Although PAR uses a stronger baseline, RandConv achieves significant improvements over our baseline and outperforms PAR by a large margin. Our methods achieve more than a 7% accuracy improvement over the baseline and surpass PAR by 5%. SIN as an image stylization approach that can modify image texture in a hierarchical and realistic way. However, albeit its complexity, it still performs on par with RandConv. Note that image stylization techniques require additional data and heavy precomputation. Further, the images for the style source also need to be chosen. In contrast, RandConv is much easier to use: it can be applied to any dataset via a simple convolution layer. We also measure the shape-bias metric proposed by Geirhos et al. (2019) for RandConv trained AlexNet. RCimg1-7,p=0.5,λ=10 and RCmix1-7,λ=10 improve the baseline from 25.36% to 48.24% and 54.85% respectively.

heading: 4.4 REVISITING PACS WITH MORE ROBUST PRETRAINED REPRESENTATIONS
text: A common practice for many computer vision tasks (including the PACS benchmark) is transfer learning, i.e. finetuning a backbone model pretrained on ImageNet. Recently, how the accuracy on ImageNet (Kornblith et al., 2019) and adversial robustness (Salman et al., 2020) of the pretrained model affect transfer learning has been studied in the context of domain generalization. Instead, we study how out-of-domain generalizability transfers from pretraining to downstream tasks and shed light on how to better use pretrained models.
Impact of ImageNet Pretraining A model trained on ImageNet may be biased towards textures (Geirhos et al., 2019). Finetuning ImageNet pretrained models on PACS may inherit this texture bias, thereby benefitting generalization on the Photo domain (which is similar to ImageNet), but hurting performance on the Sketch domain. Therefore, as shown in Sec. 4.2, using RandConv to correct this texture bias improves results on Sketch, but degrades them on the Photo domain. Since pretraining has such a strong impact on transfer performance to new tasks, we ask: "Can the generalizability of a pretrained model transfer to downstream tasks? I.e., does a pretrained model with better generalizability improve performance on unseen domains on new tasks?" To answer this, we revisit the PACS tasks based on ImageNet-pretrained weights where our two RandConv variants of Sec. 4.3 are used during ImageNet training. We study if this results in performance changes for the Deep-All baseline and for finetuning with RandConv.
Better Performance via RandConv pretrained model We start by testing the Deep-All baselines using the two RandConv-trained ImageNet models of Sec. 4.3 as initialization. Tab. 4 shows significant improvements on Sketch. Results are comparable to finetuning with RandConv on a normal pretrained model. Art is also consistently improved. Performance drops slightly on Photo as expected, since we reduced the texture bias in the pretrained model, which is helpful for the Photo domain. A similar performance improvement is observed when using the SIN-trained AlexNet as initialization. Using RandConv for both ImageNet training and PACS finetuning, we achieve 76.11% accuracy on Sketch. As far as we know, this is the best performance using an AlexNet baseline. This approach even outperforms Jigen (Carlucci et al., 2019) (71.35%) with a stronger ResNet18 baseline
model. Cartoon and Art are also improved. The best average domain generalization accuracy is 73.03%, with a more than 6% improvement over our initial Deep-All baseline.
This experiment confirms that generalizability may transfer: removing texture bias may not only make a pretrained model more generalizable, but it may help generalization on downstream tasks. For similar target and pretraining domains like Photo and ImageNet, where learning texture bias may actually be beneficial, performance may degrade slightly.

heading: 5 CONCLUSION AND DISCUSSION
text: Randomized convolution (RandConv) is a simple but powerful data augmentation technique for randomizing local image texture. RandConv helps focus visual representations on global shape information rather than local texture. We theoretically justified the approximate shape-preserving property of RandConv and developed RandConv techniques using multi-scale and mixing designs. We also make use of a consistency loss to encourage texture invariance. RandConv outperforms state-of-the-art approaches on the digit recognition benchmark and on the sketch domain of PACS and on ImageNet-Sketch by a large margin. By finetuning a model pretrained with RandConv on PACS, we showed that the generalizability of a pretrained model may transfer to and benefit a new downstream task. This resulted in a new state-of-art performance on PACS in the Sketch domain.
RandConv can help computer vision tasks when a shape-biased model is helpful e.g. for object detection. RandConv can also provide a shape-biased pretrained model to improve performance on downstream tasks when generalizing to unseen domains. However, local texture features can be useful for many computer vision tasks, especially for fixed-domain fine-grained visual recognition. In such cases, visual representations that are invariant to local texture may hurt in-domain performance. Therefore, important future work includes learning representations that disentangle shape and texture features and building models to use such representations in an explainable way.
Adversarial robustness of deep neural networks has received significant recent attention. Interestingly, Zhang & Zhu (2019) find that adversarially-trained models are more shape biased; Shi et al. (2020) show that their method for increasing shape bias also helps adversarial robustness, especially when combined with adversarial training. Therefore, exploring how RandConv affects the adversarial robustness of models could be interesting future work. Moreover, recent biologically inspired models for improving adversarial robustness (Dapello et al., 2020) use Gabor filters with fixed random configurations followed by a stochastic layer to add Gaussian noise to the network input, which may explain the importance of randomness in RandConv. Exploring connections between RandConv and biological mechanisms in the human visual system would be interesting future work.
Acknowledgments We thank Zhiding Yu for discussions on initial ideas and the experimental setup. We also thank Nathan Cahill for advice on proving the properties of random convolutions.

heading: A SHAPES AND TEXTURE IN IMAGES
text: As discussed in the main text, we define shapes in images that are preserved by a random convolution layer as primitive shapes: spatial clusters of pixels with similar local texture. An object in a image can be a single primitive shape alone but in most cases it is the composition of multiple primitive shapes e.g. a car includes wheels, body frames, windshields. Note that the definition of texture is not necessarily opposite to shapes, since the texture of a larger shape can includes smaller shapes. For example, in Fig.4, the left occluded triangle shape has texture composed by shapes of cobble stones while cobble stones have their own texture. Random convolution can preserve those large shapes that usually define the image semantics while distorting the small shapes as local texture.
To formally define the shape-preserving property, we assume (x1, y1), (x2, y2) and (x3, y3) are three locations on a image and (x1, y1) has closer color and local texture with (x2, y2) than (x3, y3). For example, (x1, y1) and (x2, y2) are within the same shape while (x3, y3) is located at a neighboring shape. Then we have ‖p(x1, y1) − p(x2, y2)‖ < ‖p(x1, y1) − p(x3, y3)‖, where p(xi, yi) is the image patch at location (xi, yi). A transformation f is shape-preserving if it maintains such relative distance relations for most location triplets, i.e.
‖f(p(xi, yi))− f(p(xj , yj))‖/‖p(xi, yi)− p(xj , yj)‖ ≈ r (1)
for any two spatial location (xi, yi) and (xj , yj); r ≥ 0 is a constant.

heading: B RANDOM CONVOLUTION IS SHAPE-PRESERVING AS RANDOM LINEAR
text: PROJECTION IS DISTANCE PRESERVING
We can express a convolution layer as a local linear projection:
g(x, y) = Up(x, y) , (2)
where p(x, y) ∈ Rd (d = h× w × Cin) is the vectorized image patch centerized at location (x, y), g(x, y) ∈ RCout is the output feature at location (x, y), and U ∈ RCout×d is the matrix expressing the convolution layer filters Θ. I.e., for each sliding window centered at (x, y), a convolution layer applies a linear transform f : Rd → RCout projecting the d dimensional local image patch p(x, y) to its Cout dimensional feature g(x, y). When Θ is independently randomly sampled, e.g. from
a Gaussian distribution, the convolution layer preserves global shapes since that a random linear projection is approximately distance-preserving by bounding the range of r in Eq. 1 in Theorem 1.
Theorem 1. Suppose we have N data points z1, · · · , zN ∈ Rd. Let f(z) = Uz be a random linear projection f : Rd → Rm such that U ∈ Rm×d and Ui,j ∼ N(0, σ2). Then we have:
P (
sup i 6=j;i,j∈[N ]
{ ri,j :=
‖f(zi)− f(zj)‖ ‖zi − zj‖
} > δ1 ) ≤ ,
P (
inf i 6=j;i,j∈[N ]
{ ri,j :=
‖f(zi)− f(zj)‖ ‖zi − zj‖
} < δ2 ) ≤ ,
(3)
where δ1 := σ √ χ2 2 N(N−1) (m) and δ2 := σ √ χ2 1− 2 N(N−1) (m). Here, χ2α(m) denotes the α-upper quantile of the χ2 distribution with m degrees of freedom.
Thm. 1 tells us that for any data pair (zi, zj) in a set of N points, the distance rescaling ratio ri,j after a random linear projection is bounded by δ1 and δ2 with probability 1− . A Smaller N and a larger output dimension m give better bounds. E.g., when m = 3, N = 1, 000, σ = 1 and = 0.1, δ1 = 5.8 and δ2 = 0.01. Thm. 1 gives a theoretical bound for all the N(N − 1)/2 pairs. However, in practice, preserving distances for a majority of N(N − 1)/2 pairs is sufficient. To empirically verify this, we test the range of central 80% of {ri,j} on real image data. Using the same (m,N, σ, ), 80% of the pairs lie in [0.56, 2.87], which is significantly better than the strict bound: [0.01, 5.8]. A proof of the theorem and simulation details are given in the following.
Proof. Let Uk represent to the k-th row of U. It is easy to check that vk := 〈Uk, zi−zj〉/‖zi−zj‖ ∼ N(0, σ2). Therefore,
‖f(zi)− f(zj)‖2
σ2‖zi − zj‖2 =
1 σ2 (zi − zj)>U>U(zi − zj) ‖zi − zj‖2 = m∑ k=1 v2k σ2 ∼ χ2(m).
Therefore, for 0 < < 1, we have P (‖f(zi)− f(zj)‖2
σ2‖zi − zj‖2 > χ2 2 N(N−1) (m) ) ≤ 2 N(N − 1) .
From the above inequality, we have P (
supi 6=j;i,j∈[N ] { ‖f(zi)−f(zj)‖2 ‖zi−zj‖2 } > σ2χ2 2 N(N−1) (m) ) = P ( supi 6=j;i,j∈[N ] { ‖f(zi)−f(zj)‖2 σ2‖zi−zj‖2 } > χ2 2 N(N−1) (m)
) = P
( ⋃ i 6=j;i,j∈[N ] { ‖f(zi)−f(zj)‖2 σ2‖zi−zj‖2 > χ 2 2 N(N−1) (m) }) ≤
∑ i 6=j;i,j∈[N ] P ( ‖f(zi)−f(zj)‖2 σ2‖zi−zj‖2 > χ 2 2 N(N−1) (m) ) ≤ ,
which is equivalent to P (
sup i 6=j;i,j∈[N ] {‖f(zi)− f(zj)‖ ‖zi − zj‖ } > σ √ χ2 2 N(N−1) (m) ) ≤ .
Similarly, we have P (
inf i 6=j;i,j∈[N ] {‖f(zi)− f(zj)‖ ‖zi − zj‖ } < σ √ χ2 1− 2 N(N−1) (m) ) ≤ .
Simulation on Real Image Data To better understand the relative distance preservation property of random linear projections in practice, we use Algorithm 2 to empirically obtain a bound for real image data. We choose m = 3, N = 1, 000, σ = 1 and = 0.1 as in computing our theoretical bounds. We use M = 1, 000 real images from the PACS dataset for this simulation. Note that the image patch size or d does not affect the bound. We use a patch size of 3 × 3 resulting in d = 27. This simulation tell us that applying linear projections with a randomly sampled U on N local images patches in every image, we have a 1− chance that 80% of ri,j is in the range [δ10%, δ90%].
Algorithm 2 Simulate the range of central 80% of ri,j on real image data
1: Input: M images {Ii}Mi=1, number of data points N , projection output dimension m, standard deviation σ of normal distribution, confidence level . 2: for m = 1→M do 3: Sample images patches in Im at 1,000 locations and vectorize them as {zml }Nl=1 4: Sample a projection matrix U ∈ Rm×d and Ui,j ∼ N(0, σ2) 5: for i = 1→ N do 6: for j = i+ 1→ N do 7: Compute rmi,j = ‖f(zmi )−f(z m j )‖
‖zmi −z m j ‖
, where f(z) = Uz
8: qm10% = 10% quantile of r m i,j for Im 9: qm90% = 90% quantile of r m i,j for Im . Get the central 80% of ri,j in each image 10: δ10% = quantile of all qm10% 11: δ90% = (1− ) quantile of all qm90% . Get the confident bound for qm10% and qm90% 12: return δ10%, δ90%

heading: C EXPERIMENTAL DETAILS
text: Digits Recognition The network for our digits recognition experiments is composed of two Conv5×5ReLU-MaxPool2×2 blocks with 64/128 output channels and three fully connected layer with 1024/1024/10 output channels. We train the network with batch size 32 for 10,000 iterations. During training, the model is validated every 250 iterations and saved with the best validation score for testing. We apply the Adam optimizer with an initial learning rate of 0.0001.
PACS We use the official data splits for training/validation/testing; no extra data augmentation is applied. We use the official PyTorch implementation and the pretrained weights of AlexNet for our PACS experiments. AlextNet is finetuned for 50,000 iterations with a batch size 128. Samples are randomly selected from the training data mixed between the three domains. We use the validation data of source domains only at every 100 iterations. We use the SGD optimizer for training with an initial learning rate of 0.001, Nesterov momentum, and weight decay set to 0.0005. We let the learning rate decay by a factor of 0.1 after finishing 80% of the iterations.
ImageNet Following the PyTorch example 3 on training ImageNet models, we set the batch size to 256 and train AlexNet from scratch for 90 epochs. We apply the SGD optimizer with an initial learning rate of 0.01, momentum 0.9, and weight decay 0.0001. We reduce the learning rate via a factor of 0.1 every 30 epochs.

heading: D MORE EXPERIMENTS WITH RESNET-18
text: In this section, we demonstrate that RandConv also works on other stronger backbone architectures, e.g. for a Residual Network He et al. (2016a). Specifically, we run the PACS and ImageNet experiments with ResNet-18 as the baseline and RandConv. As Table 5 shows, RandConv improves the baseline using ResNet18 on ImageNet-sketch by 10.5% accuracy. When using a RandConv pretrained ResNet-18 on PACS, the performance of finetuning with DeepAll and RandConv are both improved shown in Table 7. The best average domain generalization accuracy is 84.09%, with a more than 8% improvement over our initial Deep-All baseline. A model pretrained with RCmix1-7,λ=10 generally performs better than when pretrained with RCimg1-7,p=0.5,λ=10. We also provide the ResNet-18 performance of JiGen (Carlucci et al., 2019) on PACS as reference. Note
3https://github.com/pytorch/examples/tree/master/imagenet
that JiGen uses extra data augmentation and a different data split than our approach and it only improves over its own baseline by 1.5%. In addition, we test RandConv trained ResNet-18 on ImageNet-R (Hendrycks et al., 2020a), a domain generalization benchmark that contains images of artistic renditions of 200 object classes from the original ImageNet dataset. As Table 6 shows, RandConv also improve the generalization performance on ImageNet-R and reduce the gap between the in-domain (ImageNet-200) and out-of-domain (ImageNet-R) performance.

heading: E HYPERPARAMETER SELECTIONS AND ABLATION STUDIES ON DIGITS RECOGNITION BENCHMARKS
text: We provide detailed experimental results for the digits recognition datasets. Table 8 shows results for different hyperameters p for RCimg1. Table 9 shows results for an ablation study on the multi-scale design for RCmix and RCimg,p=0.5. Table 10 shows results for studying the consistency loss weight λ for RCmix1-7 and RCimg1-7,p=0.5. Tables 8, 9, and 10 correspond to Fig. 2 (a)(b)(c) in the main text respectively.
F MORE EXAMPLES OF RANDCONV DATA AUGMENTATION
We provide additional examples of RandConv outputs for different convolution filter sizes in Fig. 6 and for its mixing variants at scale k = 7 with different mixing coefficients in Fig. 5. We observe that RandConv with different filter sizes retains shapes at different scales. The mixing strategy can continuously interpolate between the training domain and a randomly sampled domain.

Review summaries:
This paper proposes a simple way to increase the robustness of the learned representations in a network perform a series of object recognition tasks by adding a random convolution layer as a pre-processing stage, thus “filtering the image” and preserving the global shape but altering the local `texture’ of the newly transformed image. Here, the hope is that  -- analogous to Geirhos et al. 2019 that induces a shape bias by transforming the image distribution into a new one with altered *global* textures that induce a shape bias and increases general robustness to o.o.d distortions --  the authors here go about doing something similar at the local level given the small size of the receptive field of the filter, thus preserving the shape and slightly altering “the texture”.