Learning Robust Representations via Multi-View Information Bottleneck

1 INTRODUCTION . The goal of deep representation learning ( LeCun et al. , 2015 ) is to transform a raw observational input , x , into a , typically lower-dimensional , representation , z , that contains the information relevant for a given task or set of tasks . Significant progress has been made in deep learning via supervised representation learning , where the labels , y , for the downstream task are known while p ( y|x ) is learned directly ( Sutskever et al. , 2012 ; Hinton et al. , 2012 ) . Due to the cost of acquiring large labeled datasets , a recently renewed focus on unsupervised representation learning seeks to generate representations , z , that are useful for a wide variety of different tasks where little to no labeled data is available ( Devlin et al. , 2018 ; Radford et al. , 2019 ) . Our work is based on the information bottleneck principle ( Tishby et al. , 2000 ) where a representation becomes less affected by nuisances by discarding all information from the input that is not useful for a given task , resulting in increased robustness . In the supervised setting , one can directly apply the information bottleneck principle by minimizing the mutual information between the data x and its representation z , I ( x ; z ) , while simultaneously maximizing the mutual information between z and the label y ( Alemi et al. , 2017 ) . In the unsupervised setting , discarding only superfluous information is more challenging , as without labels the model can not directly identify which information is relevant . Recent literature ( Devon Hjelm et al. , 2019 ; van den Oord et al. , 2018 ) has focused on the InfoMax objective maximizing I ( x , z ) instead of minimizing it , to guarantee that all the predictive information is retained by the representation , but doing nothing to discard the irrelevant information . In this paper , we extend the information bottleneck method to the unsupervised multi-view setting . To do this , we rely on a basic assumption of the multi-view literature – that each view provides the same task-relevant information ( Zhao et al. , 2017 ) . Hence , one can improve generalization by discarding all the information not shared by both views from the representation . We do this by maximizing the mutual information between the representations of the two views ( Multi-View InfoMax objective ) while at the same time eliminating the information not shared between them , since it is guaranteed to be superfluous . The resulting representations are more robust for the given task as they have eliminated view specific nuisances . Our contributions are three-fold : ( 1 ) We extend the information bottleneck principle to the unsupervised multi-view setting and provide a rigorous theoretical analysis of its application . ( 2 ) We define a new model 1 that empirically leads to state-of-the-art results in the low-label setting on two standard multi-view datasets , Sketchy and MIR-Flickr . ( 3 ) By exploiting data augmentation techniques , we empirically show that the representations learned by our model in single-view settings are more robust than existing unsupervised representation learning methods , connecting our theory to the choice of augmentation strategy . 2 PRELIMINARIES AND FRAMEWORK . The challenge of representation learning can be formulated as finding a distribution p ( z|x ) that maps data observations x ∈ X into a representation z ∈ Z , capturing some desired characteristics . Whenever the end goal involves predicting a label y , we consider only z that are discriminative enough to identify y . This requirement can be quantified by considering the amount of label information that remains accessible after encoding the data , and is known as sufficiency of z for y ( Achille & Soatto , 2018 ) : Definition 1 . Sufficiency : A representation z of x is sufficient for y if and only if I ( x ; y|z ) = 0 . Any model that has access to a sufficient representation z must be able to predict y at least as accurately as if it has access to the original data x instead . In fact , z is sufficient for y if and only if the amount of information regarding the task is unchanged by the encoding procedure ( see Proposition B.1 in the Appendix ) : I ( x ; y|z ) = 0 ⇐⇒ I ( x ; y ) = I ( y ; z ) . ( 1 ) Among sufficient representations , the ones that result in better generalization for unlabeled data instances are particularly appealing . When x has higher information content than y , some of the information in x must be irrelevant for the prediction task . This can be better understood by subdividing I ( x ; z ) into two components by using the chain rule of mutual information ( see Appendix A ) : I ( x ; z ) = I ( x ; z|y ) ︸ ︷︷ ︸ superfluous information + I ( y ; z ) ︸ ︷︷ ︸ predictive information . ( 2 ) Conditional mutual information I ( x ; z|y ) represents the information in z that is not predictive of y , i.e . superfluous information . While I ( y ; z ) determines how much label information is accessible from the representation . Note that this last term is independent of the representation as long as z is sufficient for y ( see Equation 1 ) . As a consequence , a sufficient representation contains minimal data information whenever I ( x ; z|y ) is minimized . Minimizing the amount of superfluous information can be done directly only in supervised settings . In fact , reducing I ( x ; z ) without violating the sufficiency constraint necessarily requires making some additional assumptions on the predictive task ( see Theorem B.1 in the Appendix ) . In the next section we describe the basis of our technique , a strategy to safely reduce the information content of a representation even when the label y is not observed , by exploiting redundant information in the form of an additional view on the data . 3 MULTI-VIEW INFORMATION BOTTLENECK . As a motivating example , consider v1 and v2 to be two images of the same object from different view-points and let y be its label . Assuming that the object is clearly distinguishable from both v1 and let v2 , any representation z containing all information accessible from both views would also contain the necessary label information . Furthermore , if z captures only the details that are visible from both pictures , it would eliminate the view-specific details and reduce the sensitivity of the representation to view-changes . The theory to support this intuition is described in the following where v1 and v2 are jointly observed and referred to as data-views . 1Code available at https : //github.com/mfederici/Multi-View-Information-Bottleneck 3.1 SUFFICIENCY AND ROBUSTNESS IN THE MULTI-VIEW SETTING . In this section we extend our analysis of sufficiency and minimality to the multi-view setting . Intuitively , we can guarantee that z is sufficient for predicting y even without knowing y by ensuring that z maintains all information which is shared by v1 and v2 . This intuition relies on a basic assumption of the multi-view environment – that the two views provide the same predictive information . To formalize this we define redundancy . Definition 2 . Redundancy : v1 is redundant with respect to v2 for y if and only if I ( y ; v1|v2 ) = 0 Intuitively , a view v1 is redundant for a task whenever it is irrelevant for the prediction of y if v2 is already observed . Whenever v1 and v2 are mutually redundant ( v1 is redundant with respect to v2 for y , and vice-versa ) , we can show the following : Corollary 1 . Let v1 and v2 be two mutually redundant views for a target y and let z1 be a representation of v1 . If z1 is sufficient for v2 ( I ( v1 ; v2|z1 ) = 0 ) then z1 is as predictive for y as the joint observation of the two views ( I ( v1v2 ; y ) = I ( y ; z1 ) ) . In other words , whenever it is possible to assume mutual redundancy , any representation which contains all the information shared by both views ( the redundant information ) is as predictive as their joint observation . By factorizing the mutual information between v1 and z1 analogously to Equation 2 , we can identify two components : I ( v1 ; z1 ) = I ( v1 ; z1|v2 ) ︸ ︷︷ ︸ superfluous information + I ( v2 ; z1 ) ︸ ︷︷ ︸ predictive information for v2 . Since I ( v2 ; z1 ) has to be maximal if we want the representation to be sufficient for the label , we conclude that I ( v1 ; z1 ) can be reduced by minimizing I ( v1 ; z1|v2 ) . This term intuitively represents the information z1 contains which is unique to v1 and is not predictable by observing v2 . Since we assumed mutual redundancy between the two views , this information must be irrelevant for the predictive task and , therefore , it can be safely discarded . The proofs and formal assertions for the above statements and Corollary 1 can be found in Appendix B . The less the two views have in common , the more I ( v1 ; z1 ) can be reduced without violating sufficiency for the label , and consequently , the more robust the resulting representation . At the extreme , v1 and v2 share only label information , in which case we can show that z1 is minimal for y and our method is identical to the supervised information bottleneck method without needing to access the labels . Conversely , if v1 and v2 are identical , then our method degenerates to the InfoMax principle since no information can be safely discarded ( see Appendix E ) . 3.2 THE MULTI-VIEW INFORMATION BOTTLENECK LOSS FUNCTION . Given v1 and v2 that satisfy the mutual redundancy condition for a label y , we would like to define an objective function for the representation z1 of v1 that discards as much information as possible without losing any label information . In Section 3.1 we showed that we can obtain sufficiency for y by ensuring that the representation z1 of v1 is sufficient for v2 , and that decreasing I ( z1 ; v1|v2 ) will increase the robustness of the representation by discarding irrelevant information . So we can combine these two requirements using a relaxed Lagrangian objective to obtain the minimal sufficient representation z1 for v2 : L1 ( θ ; λ1 ) = Iθ ( z1 ; v1|v2 ) − λ1 Iθ ( v2 ; z1 ) , ( 3 ) where θ denotes the dependency on the parameters of the encoder pθ ( z1|v1 ) , and λ1 represents the Lagrangian multiplier introduced by the constrained optimization . Symmetrically , we define a loss L2 to optimize the parameters ψ of a conditional distribution pψ ( z2|v2 ) that defines a minimal sufficient representation z2 of the second view v2 for v1 : L2 ( ψ ; λ2 ) = Iψ ( z2 ; v2|v1 ) − λ2 Iψ ( v1 ; z2 ) , ( 4 ) By defining z1 and z2 on the same domain Z and re-parametrizing the Lagrangian multipliers , the average of the two loss functions L1 and L2 can be upper bounded as follows : 1 LMIB ( θ , ψ ; β ) = −Iθψ ( z1 ; z2 ) + β DSKL ( pθ ( z1|v1 ) ||pψ ( z2|v2 ) ) , ( 5 ) where DSKL represents the symmetrized KL divergence obtained by averaging the expected value of DKL ( pθ ( z1|v1 ) ||pψ ( z2|v2 ) ) and DKL ( pψ ( z2|v2 ) ||pθ ( z1|v1 ) ) for joint observations of the two views , while the coefficient β defines the trade-off between sufficiency and robustness of the representation , which is a hyper-parameter in this work . The resulting Multi-View Infomation Bottleneck ( MIB ) model ( Equation 5 ) is visualized in Figure 1 , while the batch-based computation of the loss function is summarized in Algorithm 1 . The symmetrized KL divergence DSKL ( pθ ( z1|v1 ) ||pψ ( z2|v2 ) ) can be computed directly whenever pθ ( z1|v1 ) and pψ ( z2|v2 ) have a known density , while the mutual information between the two representations Iθψ ( z1 ; z2 ) can be maximized by using any sample-based differentiable mutual information lower bound . We tried the Jensen-Shannon IJS ( Devon Hjelm et al. , 2019 ; Poole et al. , 2019 ) and the InfoNCE INCE ( van den Oord et al. , 2018 ) estimators . These both require introducing an auxiliary parameteric model Cξ ( z1 , z2 ) which is jointly optimized during the training procedure using re-parametrized samples from pθ ( z1|v1 ) and pψ ( z2|v2 ) . The full derivation for the MIB loss function can be found in Appendix F .

Summary: In this paper, the authors extend the Information Bottleneck method (to build robust representations by removing information unrelated to the target labels) to the unsupervised setting. Since label information is not available in this setting, the authors leverage multi-view information (e.g., using two images of the same object) , which requires assuming that both views contain all necessary information for the subsequent label prediction task. The representation should then focus on capturing the information shared by both views and discarding the rest. A loss function for learning such representations is proposed. The effectiveness of the proposed technique is confirmed on two datasets. It is also shown to work when doing data augmentation with a single view.