Robust and Generalizable Visual Representation Learning via Random Convolutions

1 INTRODUCTION . Generalizability and robustness to out-of-distribution samples have been major pain points when applying deep neural networks ( DNNs ) in real world applications ( Volpi et al. , 2018 ) . Though DNNs are typically trained on datasets with millions of training samples , they still lack robustness to domain shift , small perturbations , and adversarial examples ( Luo et al. , 2019 ) . Recent research has shown that neural networks tend to use superficial features rather than global shape information for prediction even when trained on large-scale datasets such as ImageNet ( Geirhos et al. , 2019 ) . These superficial features can be local textures or even patterns imperceptible to humans but detectable to DNNs , as is the case for adversarial examples ( Ilyas et al. , 2019 ) . In contrast , image semantics often depend more on object shapes rather than local textures . For image data , local texture differences are one of the main sources of domain shift , e.g. , between synthetic virtual images and real data ( Sun & Saenko , 2014 ) . Our goal is therefore to learn visual representations that are invariant to local texture and that generalize to unseen domains . While texture and color may be treated as different concepts , we follow the convention in Geirhos et al . ( 2019 ) and include color when talking about texture . We address the challenging setting of robust visual representation learning from single domain data . Limited work exists in this setting . Proposed methods include data augmentation ( Volpi et al. , 2018 ; Qiao et al. , 2020 ; Geirhos et al. , 2019 ) , domain randomization ( Tobin et al. , 2017 ; Yue et al. , 2019 ) , self-supervised learning ( Carlucci et al. , 2019 ) , and penalizing the predictive power of low-level network features ( Wang et al. , 2019a ) . Following the spirit of adding inductive bias towards global shape information over local textures , we propose using random convolutions to improve the robustness to domain shifts and small perturbations . While recently Lee et al . ( 2020 ) proposed a similar technique for improving the generalization of reinforcement learning agents in 1Code is available at https : //github.com/wildphoton/RandConv . unseen environments , we focus on visual representation learning and examine our approach on visual domain generalization benchmarks . Our method also includes the multiscale design and a mixing variant . In addition , considering that many computer vision tasks rely on training deep networks based on ImageNet-pretrained weights ( including some domain generalization benchmarks ) , we ask “ Can a more robust pretrained model make the finetuned model more robust on downstream tasks ? ” Different from ( Kornblith et al. , 2019 ; Salman et al. , 2020 ) who studied the transferability of a pretrained ImageNet representation to new tasks while focusing on in-domain generalization , we explore generalization performance on unseen domains for new tasks . We make the following contributions : • We develop RandConv , a data augmentation technique using multi-scale random-convolutions to generate images with random texture while maintaining global shapes . We explore using the RandConv output as training images or mixing it with the original images . We show that a consistency loss can further enforce invariance under texture changes . • We provide insights and justification on why RandConv augments images with different local texture but the same semantics with the shape-preserving property of random convolutions . • We validate RandConv and its mixing variant in extensive experiments on synthetic and real- world benchmarks as well as on the large-scale ImageNet dataset . Our methods outperform single domain generalization approaches by a large margin on digit recognition datasets and for the challenging case of generalizing to the Sketch domain in PACS and to ImageNet-Sketch . • We explore if the robustness/generalizability of a pretrained representation can transfer . We show that transferring a model pretrained with RandConv on ImageNet can further improve domain generalization performance on new downstream tasks on the PACS dataset . 2 RELATED WORK . Domain Generalization ( DG ) aims at learning representations that perform well when transferred to unseen domains . Modern techniques range between feature fusion ( Shen et al. , 2019 ) , metalearning ( Li et al. , 2018a ; Balaji et al. , 2018 ) , and adversarial training ( Shao et al. , 2019 ; Li et al. , 2018b ) . Note that most current DG work ( Ghifary et al. , 2016 ; Li et al. , 2018a ; b ) requires a multisource training setting to work well . However , in practice , it might be difficult and expensive to collect data from multiple sources , such as collecting data from multiple medical centers ( Raghupathi & Raghupathi , 2014 ) . Instead , we consider the more strict single-domain generalization DG setting , where we train the model on source data from a single domain and generalize it to new unseen domains ( Carlucci et al. , 2019 ; Wang et al. , 2019b ) . Domain Randomization ( DR ) was first introduced as a DG technique by Tobin et al . ( 2017 ) to handle the domain gap between simulated and real data . As the training data in ( Tobin et al. , 2017 ) is synthesized in a virtual environment , it is possible to generate diverse training samples by randomly selecting background images , colors , lighting , and textures of foreground objects . When a simulation environment is not accessible , image stylization can be used to generate new domains ( Yue et al. , 2019 ; Geirhos et al. , 2019 ) . However , this requires extra effort to collect data and to train an additional model ; further , the number of randomized domains is limited by the number of predefined styles . Data Augmentation has been widely used to improve the generalization of machine learning models ( Simard et al. , 2003 ) . DR approaches can be considered a type of synthetic data augmentation . To improve performance on unseen domains , Volpi et al . ( 2018 ) generate adversarial examples to augment the training data ; Qiao et al . ( 2020 ) extend this approach via meta-learning . As with other adversarial training algorithms , significant extra computation is required to obtain adversarial examples . Learning Representations Biased towards Global Shape Geirhos et al . ( 2019 ) demonstrated that convolutional neural networks ( CNNs ) tend to use superficial local features even when trained on large datasets . To counteract this effect , they proposed to train on stylized ImageNet , thereby forcing a network to rely on object shape instead of textures . Wang et al . improved out-of-domain performance by penalizing the correlation between a learned representation and superficial features such as the gray-level co-occurrence matrix ( Wang et al. , 2019b ) , or by penalizing the predictive power of local , low-level layer features in a neural network via an adversarial classifier ( Wang et al. , 2019a ) . Our approach shares the idea that learning representations invariant to local texture helps generalization to unseen domains . However , RandConv avoids searching over many hyper-parameters , collecting extra data , and training other networks . It also scales to large-scale datasets since it adds minimal computation overhead . Random Mapping in Machine Learning Random projections have also been effective for dimensionality reduction based on the distance-preserving property of the Johnson–Lindenstrauss lemma ( Johnson & Lindenstrauss , 1984 ) . ( Vinh et al. , 2016 ) applied random projections on entire images as data augmentation to make neural networks robust to adversarial examples . Lee et al . ( 2020 ) recently used random convolutions to help reinforcement learning ( RL ) agents generalize to new environments . Neural networks with fixed random weights can encode meaningful representations ( Saxe et al. , 2011 ) and are therefore useful for neural architecture search ( Gaier & Ha , 2019 ) , generative models ( He et al. , 2016b ) , natural language processing ( Wieting & Kiela , 2019 ) , and RL ( Osband et al. , 2018 ; Burda et al. , 2019 ) . In contrast , RandConv uses non-fixed randomly-sampled weights to generate images with different local texture . 3 RANDCONV : RANDOMIZE LOCAL TEXTURE AT DIFFERENT SCALES . We propose using a convolution layer with non-fixed random weights as the first layer of a DNN during training . This strategy generates images with random local texture but consistent shapes , and is beneficial for robust visual representation learning . Sec . 3.1 justifies the shape-preserving property of a random convolution layer . Sec . 3.2 describes RandConv , our data augmentation algorithm using a multi-scale randomized convolution layer and input mixing . 3.1 A RANDOM CONVOLUTION LAYER PRESERVES GLOBAL SHAPES . Convolution is the key building block for deep convolutional neural networks . Consider a convolution layer with filters Θ ∈ Rh×w×Cin×Cout with an input image I ∈ RH×W×Cin , where H and W are the height and width of the input and Cin and Cout are the number of feature channels for the input and output , and h and w are the height and width of the layer ’ s filter . The output ( with appropriate input padding ) will be g = I ∗Θ with g ∈ RH×W×Cout . In images , nearby pixels with similar color or texture can be grouped into primitive shapes that represent parts of objects or the background . A convolution layer linearly projects local image patches to features at corresponding locations on the output map using shared parameters . While a convolution with random filters can project local patches to arbitrary output features , the output of a random linear projection approximately preserves relative similarity between input patches , proved in Appendix B . In other words , since any two locations within the same shape have similar local textures in the input image , they tend to be similar in the output feature map . Therefore , shapes that emerge in the output feature map are similar to shapes in the input image provided that the filter size is sufficiently small compared to the size of a typical shape . In other words , the size of a convolution filter determines the smallest shape it can preserve . For example , 1x1 random convolutions preserve shapes at the single-pixel level and thus work as a random color mapping ; large filters perturb shapes smaller than the filter size that are considered local texture of a shape at this larger scale . See Fig . 1 for examples . More discussion and a formal proof are in Appendix A and B . 3.2 MULTI-SCALE IMAGE AUGMENTATION WITH A RANDOMIZED CONVOLUTION LAYER . Algorithm 1 Learning with Data Augmentation by Random Convolutions 1 : Input : Model Φ , task loss Ltask , training images { Ii } Ni=1 and their labels { yi } Ni=1 , pool of filter sizes K = { 1 , ... , n } , fraction of original data p , whether to mix with original images , consistency loss weight λ 2 : function RANDCONV ( I , K , mix , p ) 3 : Sample p0 ∼ U ( 0 , 1 ) 4 : if p0 < p and mix is False then 5 : return I . When not in mix mode , use the original image with probability p 6 : else 7 : Sample scale k ∼ K 8 : Sample convolution weights Θ ∈ Rk×k×3×3 ∼ N ( 0 , 1 3k2 ) 9 : Irc = I ∗Θ . Apply convolution on I 10 : if mix is True then 11 : Sample α ∼ U ( 0 , 1 ) 12 : return αI + ( 1− α ) Irc . Mix with original images 13 : else 14 : return Irc 15 : Learning Objective : 16 : for i = 1→ N do 17 : for j = 1→ 3 do 18 : ŷji = Φ ( RandConv ( Ii ) ) . Predict labels for three augmented variants of the same image 19 : Lcons = λ ∑3 j=1 KL ( ŷ j i ||ȳi ) where ȳi = ∑3 j=1 ŷ j i /3 . Consistency Loss 20 : L = Ltask ( ŷ1i , yi ) + λLcons . Learning with the task loss and the consistency loss Sec . 3.1 discussed how outputs of randomized convolution layers approximately maintain shape information at a scale larger than their filter sizes . Here , we develop our RandConv data augmentation technique using a randomized convolution layer with Cout = Cin to generate shape-consistent images with randomized texture ( see Alg . 1 ) . Our goal is not to use RandConv to parameterize or represent texture as in previous filter-bank based texture models ( Heeger & Bergen , 1995 ; Portilla & Simoncelli , 2000 ) . Instead , we only use the three-channel outputs of RandConv as new images with the same shape and different “ style ” ( loosely referred to as `` texture '' ) . We also note that , a convolution layer is different from a convolution operation in image filtering . Standard image filtering applies the same 2D filter on three color channels separately . In contrast , our convolution layer applies three different 3D filters and each takes all color channels as input and generates one channel of the output . Our proposed RandConv variants are as follows : RCimg : Augmenting Images with Random Texture A simple approach is to use the randomized convolution layer outputs , I ∗Θ , as new images ; where Θ are the randomly sampled weights and I is a training image . If the original training data is in the domain D0 , a sampled weight Θk generates images with consistent global shape but random texture forming the random domain Dk . Thus , by random weight sampling , we obtain an infinite number of random domains D1 , D1 , . . . , D∞ . Input image intensities are assumed to be a standard normal distribution N ( 0 , 1 ) ( which is often true in practice thanks to data whitening ) . As the outputs of RandConv should follow the same distribution , we sample the convolution weights from N ( 0 , σ2 ) where σ = 1/ √ Cin × h× w , which is commonly applied for network initialization ( He et al. , 2015 ) . We include the original images for training at a ratio p as a hyperparameter . RCmix : Mixing Variant As shown in Fig . 1 , outputs from RCimg can vary significantly from the appearance of the original images . Although generalizing to domains with significantly different local texture distributions is useful , we may not want to sacrifice much performance on domains similar to the training domain . Inspired by the AugMix ( Hendrycks et al. , 2020b ) strategy , we propose to blend the original image with the outputs of the RandConv layer via linear convex combinations αI + ( 1 − α ) ( I ∗Θ ) , where α is the mixing weight uniformly sampled from [ 0 , 1 ] .In RCmix , the RandConv outputs provide shape-consistent perturbations of the original images . Varying α , we continuously interpolate between the training domain and the randomly sampled domains of RCimg . Multi-scale Texture Corruption As discussed in Sec . 3.1 „ image shape information at a scale smaller than a filter ’ s size will be corrupted by RandConv . Therefore , we can use filters of varying sizes to preserve shapes at various scales . We choose to uniformly randomly sample a filter size k from a pool K = 1 , 3 , ... n before sampling convolution weights Θ ∈ Rk×k×Cin×Cout from a Gaussian distribution N ( 0 , 1k2Cin ) . Fig . 1 shows examples of multi-scale RandConv outputs . Consistency Regularization To learn representations invariant to texture changes , we use a loss encouraging consistent network predictions for the same RandConv-augmented image for different random filter samples . Approaches for transform-invariant domain randomization ( Yue et al. , 2019 ) , data augmentation ( Hendrycks et al. , 2020b ) , and semi-supervised learning ( Berthelot et al. , 2019 ) use similar strategies . We use Kullback-Leibler ( KL ) divergence to measure consistency . However , enforcing prediction similarity of two augmented variants may be too strong . Instead , following ( Hendrycks et al. , 2020b ) , we use RandConv to obtain 3 augmentation samples of image I : Gj = RandConvj ( I ) for j = 1 , 2 , 3 and obtain their predictions with a model Φ : yj = Φ ( Gj ) . We then compute the relaxed loss as λ ∑3 j=1 KL ( y j ||ȳ ) , where ȳ = ∑3 j=1 y j/3 is the sample average .

Summary: This paper proposes a simple way to increase the robustness of the learned representations in a network perform a series of object recognition tasks by adding a random convolution layer as a pre-processing stage, thus “filtering the image” and preserving the global shape but altering the local `texture’ of the newly transformed image. Here, the hope is that  -- analogous to Geirhos et al. 2019 that induces a shape bias by transforming the image distribution into a new one with altered *global* textures that induce a shape bias and increases general robustness to o.o.d distortions --  the authors here go about doing something similar at the local level given the small size of the receptive field of the filter, thus preserving the shape and slightly altering “the texture”.