Learning Robust Representations by Projecting Superficial Statistics Out

1 INTRODUCTION . Imagine training an image classifier to recognize facial expressions . In the training data , while all images labeled “ smile ” may actually depict smiling people , the “ smile ” label might also be correlated with other aspects of the image . For example , people might tend to smile more often while outdoors , and to frown more in airports . In the future , we might encounter photographs with previously unseen backgrounds , and thus we prefer models that rely as little as possible on the superficial signal . The problem of learning classifiers robust to distribution shift , commonly called Domain Adaptation ( DA ) , has a rich history . Under restrictive assumptions , such as covariate shift ( Shimodaira , 2000 ; Gretton et al. , 2009 ) , and label shift ( also known as target shift or prior probability shift ) ( Storkey , 2009 ; Schölkopf et al. , 2012 ; Zhang et al. , 2013 ; Lipton et al. , 2018 ) , principled methods exist for estimating the shifts and retraining under the importance-weighted ERM framework . Other papers bound worst-case performance under bounded shifts as measured by divergence measures on the train v.s . test distributions ( Ben-David et al. , 2010a ; Mansour et al. , 2009 ; Hu et al. , 2016 ) . While many impossibility results for DA have been proven ( Ben-David et al. , 2010b ) , humans nevertheless exhibit a remarkable ability to function out-of-sample , even when confronting dramatic distribution shift . Few would doubt that given photographs of smiling and frowning astronauts on the Martian plains , we could ( mostly ) agree upon the correct labels . While we lack a mathematical description of how precisely humans are able to generalize so easily out-of-sample , we can often point to certain classes of perturbations that should not effect the semantics of an image . For example for many tasks , we know that the background should not influence the predictions made about an image . Similarly , other superficial statistics of the data , such as textures or subtle coloring changes should not matter . The essential assumption of this paper is that by making our model depend less on known superficial aspects , we can push the model to rely more on the difference that makes a difference . This paper focuses on visual applications , and we focus on high-frequency textural information as the relevant notion of superficial statistics that we do not want our model to depend upon . The contribution of this paper can be summarized as follows . • We propose a new differentiable neural network building block ( neural gray-level cooccurrence matrix ) that captures textural information only from images without modeling the lower-frequency semantic information that we care about ( Section 3.1 ) . • We propose an architecture-agnostic , parameter-free method that is designed to discard this superficial information , ( Section 3.2 ) . • We introduce two synthetic datasets for DA/DG studies that are more challenging than regular DA/DG scenario in the sense that the domain-specific information is correlated with semantic information . Figure 1 is a toy example ( Section 4 ) . 2 RELATED WORK IN DOMAIN ADAPTATION AND DOMAIN GENERALIZATION . Domain generalization ( DG ) ( Muandet et al. , 2013 ) is a variation on DA , where samples from the target domain are not available during training . In reality , data-sets may contain data cobbled together from many sources but where those sources are not labeled . For example , a common assumption used to be that there is one and only one distribution for each dataset collected , but Wang et al . ( 2016 ) noticed that in video sentiment analysis , the data sources varied considerably even within the same dataset due to heterogeneous data sources and collection practices . Domain adaptation ( Bridle & Cox , 1991 ; Ben-David et al. , 2010a ) , and ( more broadly ) transfer learning have been studied for decades , with antecedents in the classic econometrics work on sample selection bias Heckman ( 1977 ) and choice models Manski & Lerman ( 1977 ) . For a general primer , we refer the reader to these extensive reviews ( Weiss et al. , 2016 ; Csurka , 2017 ) . Domain generalization ( Muandet et al. , 2013 ) is relatively new , but has also been studied extensively : covering a wide spectrum of techniques from kernel methods ( Muandet et al. , 2013 ; Niu et al. , 2015 ; Erfani et al. , 2016 ; Li et al. , 2017c ) to more recent deep learning end-to-end methods , where the methods mostly fall into two categories : reducing the inter-domain differences of representations through adversarial ( or similar ) techniques ( Ghifary et al. , 2015 ; Wang et al. , 2016 ; Motiian et al. , 2017 ; Li et al. , 2018 ; Carlucci et al. , 2018 ) , or building an ensemble of one-for-each-domain deep models and then fusing representations together ( Ding & Fu , 2018 ; Mancini et al. , 2018 ) . Metalearning techniques are also explored ( Li et al. , 2017b ) . Related studies are also conducted under the name “ zero shot domain adaptation ” e.g . ( Kumagai & Iwata , 2018 ) . 3 METHOD . In this section , we introduce our main technical contributions . We will first introduce the our new differentiable neural building block , NGLCM that is designed to capture textural but not semantic information from images , and then introduce our technique for excluding the textural information . 3.1 NEURAL GRAY-LEVEL CO-OCCURRENCE MATRIX FOR SUPERFICIAL INFORMATION . Our goal is to design a neural building block that 1 ) has enough capacity to extract the textural information from an image , 2 ) is not capable of extracting semantic information . We consulted some classic computer vision techniques for inspiration and extensive experimental evidence ( Appendix A1 ) , suggested that gray-level co-occurrence matrix ( GLCM ) ( Haralick et al. , 1973 ; Lam , 1996 ) may suit our goal . The idea of GLCM is to count the number of pixel pairs under a certain direction ( common direction choices are 0◦ , 45◦ , 90◦ , and 135◦ ) . For example , for an image A ∈ N m×m , where N denotes the set of all possible pixel values . The GLCM of A under the direction to 0◦ ( horizontally right ) will be a |N |× |N |matrix ( denoted byG ) defined as following : Gk , l = m−1∑ i=0 m∑ j=0 I ( Ai , j = k ) I ( Ai+1 , j = l ) ( 1 ) where |N | stands for the cardinality of N , I ( · ) is an identity function , i , j are indices of A , and k , l are pixel values of A as well as indices of G. We design a new neural network building block that resembles GLCM but whose parameters are differentiable , having ( sub ) gradient everywhere , and thus are tunable through backpropagation . We first flatten A into a row vector a ∈ N 1×m2 . The first observation we made is that the counting of pixel pairs ( pk , pl ) in Equation 1 is equivalent to counting the pairs ( pk , ∆p ) , where ∆p = pk−pl . Therefore , we first generate a vector d by multiplying a with a matrix D , where D is designed according to the direction of GLCM . For example , D in the 0◦ case will be a m2 ×m2 matrix D such that Di , i = 1 , Di , i+1 = −1 , and 0 elsewhere . To count the elements in a and dwith a differentiable operation , we introduce two sets of parameters φa ∈ R|N |×1 and φb ∈ R|N |×1 as the tunable parameter for this building block , so that : G = s ( a ; φa ) s T ( d ; φb ) ( 2 ) where s ( ) is a thresholding function defined as : s ( a ; φa ) = min ( max ( a φa , 0 ) , 1 ) where denotes the minus operation with the broadcasting mechanism , yielding both s ( a ; φa ) and s ( d ; φb ) as |N | ×m2 matrices . As a result , G is a |N | × |N | matrix . The design rationale is that , with an extra constrain that requires φ to have only unique values in the set of { n− |n ∈ N } , where is a small number , G in Equation 2 will be equivalent to the GLCM extracted with old counting techniques , subject to permutation and scale . Also , all the operations used in the construction of G have ( sub ) gradient and therefore all the parameters are tunable with backpropagation . In practice , we drop the extra constraint on φ for simplicity in computation . Our preliminary experiments suggested that for our purposes it is sufficient to first map standard images with 256 pixel levels to images with 16 pixel levels , which can reduce to the number of parameters of NGLCM ( |N | = 16 ) . 3.2 HEX . We first introduce notation to represent the neural network . We use 〈X , y〉 to denote a dataset of inputs X and corresponding labels y . We use h ( · ; θ ) and f ( · ; ξ ) to the bottom and top components of a neural network . A conventional neural network architecture will use f ( h ( X ; θ ) ; ξ ) to generate a corresponding result Fi and then calculate the argmax to yield the prediction label . Besides conventional f ( h ( X ; θ ) ; ξ ) , we introduce another architecture g ( X ; φ ) = σm ( ( s ( a ; φa ) s T ( d ; φb ) ) Wm + bm ) where φ = { φa , φb , Wm , bm } , s ( a ; φa ) sT ( d ; φb ) is introduced in previous section , { Wm , bm , σm } ( weights , biases , and activation function ) form a standard MLP . With the introduction of g ( · ; φ ) , the final classification layer turns into f [ h ( X ; θ ) , g ( X ; φ ) ] ; ξ ) ( where we use [ · , · ] to denote concatenation ) . Now , with the representation learned through raw data by h ( · ; θ ) and textural representation learned by g ( · ; φ ) , the next question is to force f ( · ; ξ ) to predict with transformed representation from h ( · ; θ ) that in some sense independent of the superficial representation captured by g ( · ; φ ) . To illustrate following ideas , we first introduce three different outputs from the final layer : FA = f ( [ h ( X ; θ ) , g ( X ; φ ) ] ; ξ ) FG = f ( [ 0 , g ( X ; φ ) ] ; ξ ) FP = f ( [ h ( X ; θ ) ,0 ] ; ξ ) ( 3 ) where FA , FG , and FP stands for the results from both representations ( concatenated ) , only the textural information ( prepended with the 0 vector ) , and only the raw data ( concatenated wit hthe 0 vecotr ) , respectively . 0 stands for a padding matrix with all the zeros , whose shape can be inferred by context . Several heuristics have been proposed to force a network to “ forget ” some part of a representation , such as adversarial training ( Ganin et al. , 2016 ) or information-theoretic regularization ( Moyer et al. , 2018 ) . In similar spirit , our first proposed solution is to adopt the reverse gradient idea ( Ganin et al. , 2016 ) to train FP to be predictive for the semantic labels y while forcing the FP to be invariant to FG . Later , we refer to this method as ADV . When we use a multilayer perceptron ( MLP ) to try to predict g ( X ; φ ) from h ( X ; θ ) and update the primary model to fool the MLP via reverse gradient , we refer to the model as ADVE . Additionally , we introduce a simple alternative . Our idea lies in the fact that , in an affine space , to find a transformation of representation A that is least explainable by some other representation B , a straightforward method will be projecting A with a projection matrix constructed by B ( sometimes referred as residual maker matrix. ) . To utilize this linear property , we choose to work on the space of F generated by f ( · ; ξ ) right before the final argmax function . Projecting FA with FL = ( I − FG ( FTGFG ) −1FTG ) FA ( 4 ) will yield FL for parameter tuning . All the parameters ξ , φ , θ can be trained simultaneously ( more relevant discussions in Section 5 ) . In testing time , FP is used . Due to limited space , we leave the following topics to the Appendix : 1 ) rationales of this approach ( A2.1 ) 2 ) what to do in cases when FTGFG is not invertible ( A2.2 ) . This method is referred as HEX . Two alternative forms of our algorithm are also worth mentioning : 1 ) During training , one can tune an extra hyperparameter ( λ ) through l ( arg maxFL , y ) + λl ( arg maxFG , y ) to ensure that the NGLCM component is learning superficial representations that are related to the present task where l ( · , · ) is a generic loss function . 2 ) During testing , one can use FL , although this requires evaluating the NGLCM component at prediction time and thus is slightly slower . We experimented with these three forms with our synthetic datasets and did not observe significant differences in performance and thus we adopt the fastest method as the main one . Empirically , we also notice that it is helpful to make sure the textural representation g ( X ; φ ) and raw data representation h ( X ; θ ) are of the same scale for HEX to work , so we column-wise normalize these two representations in every minibatch .

Summary: The paper proposed a novel differentiable neural GLCM network which captures the high reference textural information and discard the lower-frequency semantic information so as to solve the domain generalisation challenge. The author also proposed an approach “HEX” to discard the superficial representations. Two synthetic datasets are created for demonstrating the methods advantages on scenarios where the domain-specific information is correlated with the semantic information. The proposal is well structured and written. The quality of the paper is excellent in terms of novelty and originality. The proposed methods are evaluated thoroughly through experiments with different types of dataset and has shown to achieve good performance.