Learning Robust Representations by Projecting Superficial Statistics Out

1 INTRODUCTION . Imagine training an image classifier to recognize facial expressions . In the training data , while all images labeled “ smile ” may actually depict smiling people , the “ smile ” label might also be correlated with other aspects of the image . For example , people might tend to smile more often while outdoors , and to frown more in airports . In the future , we might encounter photographs with previously unseen backgrounds , and thus we prefer models that rely as little as possible on the superficial signal . The problem of learning classifiers robust to distribution shift , commonly called Domain Adaptation ( DA ) , has a rich history . Under restrictive assumptions , such as covariate shift ( Shimodaira , 2000 ; Gretton et al. , 2009 ) , and label shift ( also known as target shift or prior probability shift ) ( Storkey , 2009 ; Schölkopf et al. , 2012 ; Zhang et al. , 2013 ; Lipton et al. , 2018 ) , principled methods exist for estimating the shifts and retraining under the importance-weighted ERM framework . Other papers bound worst-case performance under bounded shifts as measured by divergence measures on the train v.s . test distributions ( Ben-David et al. , 2010a ; Mansour et al. , 2009 ; Hu et al. , 2016 ) . While many impossibility results for DA have been proven ( Ben-David et al. , 2010b ) , humans nevertheless exhibit a remarkable ability to function out-of-sample , even when confronting dramatic distribution shift . Few would doubt that given photographs of smiling and frowning astronauts on the Martian plains , we could ( mostly ) agree upon the correct labels . While we lack a mathematical description of how precisely humans are able to generalize so easily out-of-sample , we can often point to certain classes of perturbations that should not effect the semantics of an image . For example for many tasks , we know that the background should not influence the predictions made about an image . Similarly , other superficial statistics of the data , such as textures or subtle coloring changes should not matter . The essential assumption of this paper is that by making our model depend less on known superficial aspects , we can push the model to rely more on the difference that makes a difference . This paper focuses on visual applications , and we focus on high-frequency textural information as the relevant notion of superficial statistics that we do not want our model to depend upon . The contribution of this paper can be summarized as follows . • We propose a new differentiable neural network building block ( neural gray-level cooccurrence matrix ) that captures textural information only from images without modeling the lower-frequency semantic information that we care about ( Section 3.1 ) . • We propose an architecture-agnostic , parameter-free method that is designed to discard this superficial information , ( Section 3.2 ) . • We introduce two synthetic datasets for DA/DG studies that are more challenging than regular DA/DG scenario in the sense that the domain-specific information is correlated with semantic information . Figure 1 is a toy example ( Section 4 ) . 2 RELATED WORK IN DOMAIN ADAPTATION AND DOMAIN GENERALIZATION . Domain generalization ( DG ) ( Muandet et al. , 2013 ) is a variation on DA , where samples from the target domain are not available during training . In reality , data-sets may contain data cobbled together from many sources but where those sources are not labeled . For example , a common assumption used to be that there is one and only one distribution for each dataset collected , but Wang et al . ( 2016 ) noticed that in video sentiment analysis , the data sources varied considerably even within the same dataset due to heterogeneous data sources and collection practices . Domain adaptation ( Bridle & Cox , 1991 ; Ben-David et al. , 2010a ) , and ( more broadly ) transfer learning have been studied for decades , with antecedents in the classic econometrics work on sample selection bias Heckman ( 1977 ) and choice models Manski & Lerman ( 1977 ) . For a general primer , we refer the reader to these extensive reviews ( Weiss et al. , 2016 ; Csurka , 2017 ) . Domain generalization ( Muandet et al. , 2013 ) is relatively new , but has also been studied extensively : covering a wide spectrum of techniques from kernel methods ( Muandet et al. , 2013 ; Niu et al. , 2015 ; Erfani et al. , 2016 ; Li et al. , 2017c ) to more recent deep learning end-to-end methods , where the methods mostly fall into two categories : reducing the inter-domain differences of representations through adversarial ( or similar ) techniques ( Ghifary et al. , 2015 ; Wang et al. , 2016 ; Motiian et al. , 2017 ; Li et al. , 2018 ; Carlucci et al. , 2018 ) , or building an ensemble of one-for-each-domain deep models and then fusing representations together ( Ding & Fu , 2018 ; Mancini et al. , 2018 ) . Metalearning techniques are also explored ( Li et al. , 2017b ) . Related studies are also conducted under the name “ zero shot domain adaptation ” e.g . ( Kumagai & Iwata , 2018 ) . 3 METHOD . In this section , we introduce our main technical contributions . We will first introduce the our new differentiable neural building block , NGLCM that is designed to capture textural but not semantic information from images , and then introduce our technique for excluding the textural information . 3.1 NEURAL GRAY-LEVEL CO-OCCURRENCE MATRIX FOR SUPERFICIAL INFORMATION . Our goal is to design a neural building block that 1 ) has enough capacity to extract the textural information from an image , 2 ) is not capable of extracting semantic information . We consulted some classic computer vision techniques for inspiration and extensive experimental evidence ( Appendix A1 ) , suggested that gray-level co-occurrence matrix ( GLCM ) ( Haralick et al. , 1973 ; Lam , 1996 ) may suit our goal . The idea of GLCM is to count the number of pixel pairs under a certain direction ( common direction choices are 0◦ , 45◦ , 90◦ , and 135◦ ) . For example , for an image A ∈ N m×m , where N denotes the set of all possible pixel values . The GLCM of A under the direction to 0◦ ( horizontally right ) will be a |N |× |N |matrix ( denoted byG ) defined as following : Gk , l = m−1∑ i=0 m∑ j=0 I ( Ai , j = k ) I ( Ai+1 , j = l ) ( 1 ) where |N | stands for the cardinality of N , I ( · ) is an identity function , i , j are indices of A , and k , l are pixel values of A as well as indices of G. We design a new neural network building block that resembles GLCM but whose parameters are differentiable , having ( sub ) gradient everywhere , and thus are tunable through backpropagation . We first flatten A into a row vector a ∈ N 1×m2 . The first observation we made is that the counting of pixel pairs ( pk , pl ) in Equation 1 is equivalent to counting the pairs ( pk , ∆p ) , where ∆p = pk−pl . Therefore , we first generate a vector d by multiplying a with a matrix D , where D is designed according to the direction of GLCM . For example , D in the 0◦ case will be a m2 ×m2 matrix D such that Di , i = 1 , Di , i+1 = −1 , and 0 elsewhere . To count the elements in a and dwith a differentiable operation , we introduce two sets of parameters φa ∈ R|N |×1 and φb ∈ R|N |×1 as the tunable parameter for this building block , so that : G = s ( a ; φa ) s T ( d ; φb ) ( 2 ) where s ( ) is a thresholding function defined as : s ( a ; φa ) = min ( max ( a φa , 0 ) , 1 ) where denotes the minus operation with the broadcasting mechanism , yielding both s ( a ; φa ) and s ( d ; φb ) as |N | ×m2 matrices . As a result , G is a |N | × |N | matrix . The design rationale is that , with an extra constrain that requires φ to have only unique values in the set of { n− |n ∈ N } , where is a small number , G in Equation 2 will be equivalent to the GLCM extracted with old counting techniques , subject to permutation and scale . Also , all the operations used in the construction of G have ( sub ) gradient and therefore all the parameters are tunable with backpropagation . In practice , we drop the extra constraint on φ for simplicity in computation . Our preliminary experiments suggested that for our purposes it is sufficient to first map standard images with 256 pixel levels to images with 16 pixel levels , which can reduce to the number of parameters of NGLCM ( |N | = 16 ) . 3.2 HEX . We first introduce notation to represent the neural network . We use 〈X , y〉 to denote a dataset of inputs X and corresponding labels y . We use h ( · ; θ ) and f ( · ; ξ ) to the bottom and top components of a neural network . A conventional neural network architecture will use f ( h ( X ; θ ) ; ξ ) to generate a corresponding result Fi and then calculate the argmax to yield the prediction label . Besides conventional f ( h ( X ; θ ) ; ξ ) , we introduce another architecture g ( X ; φ ) = σm ( ( s ( a ; φa ) s T ( d ; φb ) ) Wm + bm ) where φ = { φa , φb , Wm , bm } , s ( a ; φa ) sT ( d ; φb ) is introduced in previous section , { Wm , bm , σm } ( weights , biases , and activation function ) form a standard MLP . With the introduction of g ( · ; φ ) , the final classification layer turns into f [ h ( X ; θ ) , g ( X ; φ ) ] ; ξ ) ( where we use [ · , · ] to denote concatenation ) . Now , with the representation learned through raw data by h ( · ; θ ) and textural representation learned by g ( · ; φ ) , the next question is to force f ( · ; ξ ) to predict with transformed representation from h ( · ; θ ) that in some sense independent of the superficial representation captured by g ( · ; φ ) . To illustrate following ideas , we first introduce three different outputs from the final layer : FA = f ( [ h ( X ; θ ) , g ( X ; φ ) ] ; ξ ) FG = f ( [ 0 , g ( X ; φ ) ] ; ξ ) FP = f ( [ h ( X ; θ ) ,0 ] ; ξ ) ( 3 ) where FA , FG , and FP stands for the results from both representations ( concatenated ) , only the textural information ( prepended with the 0 vector ) , and only the raw data ( concatenated wit hthe 0 vecotr ) , respectively . 0 stands for a padding matrix with all the zeros , whose shape can be inferred by context . Several heuristics have been proposed to force a network to “ forget ” some part of a representation , such as adversarial training ( Ganin et al. , 2016 ) or information-theoretic regularization ( Moyer et al. , 2018 ) . In similar spirit , our first proposed solution is to adopt the reverse gradient idea ( Ganin et al. , 2016 ) to train FP to be predictive for the semantic labels y while forcing the FP to be invariant to FG . Later , we refer to this method as ADV . When we use a multilayer perceptron ( MLP ) to try to predict g ( X ; φ ) from h ( X ; θ ) and update the primary model to fool the MLP via reverse gradient , we refer to the model as ADVE . Additionally , we introduce a simple alternative . Our idea lies in the fact that , in an affine space , to find a transformation of representation A that is least explainable by some other representation B , a straightforward method will be projecting A with a projection matrix constructed by B ( sometimes referred as residual maker matrix. ) . To utilize this linear property , we choose to work on the space of F generated by f ( · ; ξ ) right before the final argmax function . Projecting FA with FL = ( I − FG ( FTGFG ) −1FTG ) FA ( 4 ) will yield FL for parameter tuning . All the parameters ξ , φ , θ can be trained simultaneously ( more relevant discussions in Section 5 ) . In testing time , FP is used . Due to limited space , we leave the following topics to the Appendix : 1 ) rationales of this approach ( A2.1 ) 2 ) what to do in cases when FTGFG is not invertible ( A2.2 ) . This method is referred as HEX . Two alternative forms of our algorithm are also worth mentioning : 1 ) During training , one can tune an extra hyperparameter ( λ ) through l ( arg maxFL , y ) + λl ( arg maxFG , y ) to ensure that the NGLCM component is learning superficial representations that are related to the present task where l ( · , · ) is a generic loss function . 2 ) During testing , one can use FL , although this requires evaluating the NGLCM component at prediction time and thus is slightly slower . We experimented with these three forms with our synthetic datasets and did not observe significant differences in performance and thus we adopt the fastest method as the main one . Empirically , we also notice that it is helpful to make sure the textural representation g ( X ; φ ) and raw data representation h ( X ; θ ) are of the same scale for HEX to work , so we column-wise normalize these two representations in every minibatch .

Summary: The paper proposes an unsupervised approach to identify image features that are not meaningful for image classification tasks. The goal is to address the domain adaptation (DA)/domain generalization (DG) issue. The paper introduces a new learning task where the domain identity is unavailable during training, called unguided domain generalization (UDG). The proposed approach is based on an old method of using gray level co-occurence matrix, updated to allow for differentiable training. This new approach is used in two different ways to reduce the effect of background texture in a classification task. The paper introduces a new dataset, and shows extensive and carefully designed experiments using the new data as well as existing domain generalization datasets.