The Connection between Out-of-Distribution Generalization and Privacy of ML Models

1 INTRODUCTION . Generalization of machine learning ( ML ) models to out-of-distribution data ( domains ) is important for the success of their deployment in practice . To address this challenge , several domain generalization ( DG ) learning techniques are proposed that achieve improved out-of-distribution ( OOD ) accuracy ( Dou et al. , 2019 ; Piratla et al. , 2020 ; Asadi et al. , 2019 ; Ilse et al. , 2020 ) . Among them , state-of-the-art solutions rely on the idea of learning stable feature representations whose effect on the output remains invariant across domains ( Arjovsky et al. , 2019 ; Mahajan et al. , 2021 ) . Since these stable features correspond to the causal mechanism by which real world data is generated , stable features-based learning methods reduce the OOD generalization gap ( Peters et al. , 2016 ) . Concurrently , there is a growing literature on ensuring privacy guarantees for ML models since privacy is an important requirement for deploying ML models , especially in sensitive domains . Privacy attacks such as membership inference can leak information about the data used to train models ( Shokri et al. , 2017 ) . Existing defenses such as differentially-private training prevent the leakage but hurt the model accuracy significantly ( Truex et al. , 2018 ; Zanella-Béguelin et al. , 2020 ; Carlini et al. , 2019 ) . Theoretically , failure to generalize ( i.e. , overfitting ) has been shown to be a sufficient condition for membership inference attacks when evaluated on the same distribution as training data ( Yeom et al. , 2018 ) . When evaluated on a different distribution , as is often the case in real-world domain generalization tasks , the membership attack risk is expected to worsen for standard algorithms ( e.g. , empirical risk minimization ) since generalization is harder . From the mitigation perspective under multiple data distributions , Tople et al . ( 2020 ) has theoretically shown that causal models that learn stable features are robust to membership inference attacks as compared to standard neural networks training . However , their result holds in the limited setting where all true stable features are known apriori ( causal sufficiency ) , which is unrealistic for most applications . With the empirical advances in DG algorithms that learn stable features , we ask whether better OOD generalization can provide a viable route to membership inference robustness . Specifically , we ask two questions : Would state-of-the-art domain generalization techniques that provide better out-ofdistribution generalization lead to membership privacy benefits ? And is learning stable features necessary for membership privacy benefits ? To answer these questions , we conduct an extensive empirical study using recent DG algorithms trained over multiple datasets : simulated datasets , semisimulated DG benchmarks , and a real-world Xrays dataset . Based on our experiments , we do not find a direct relationship between better OOD generalization and privacy . Our work provides the first empirical evidence that a higher generalization gap is not necessary to perform MI attacks in out-of-distribution deployment scenarios . Prior work has only confirmed the sufficiency argument from Yeom et al . ( 2018 ) for practical ML algorithms under indistribution deployment setting ( Shokri et al. , 2017 ; Salem et al. , 2019 ) . Specifically , we find that an algorithm with lower OOD generalization gap may have worse membership privacy compared to another with higher generalization gap . Instead , our results indicate that the ability of a model to learn stable features is a consistent indicator of its privacy guarantees in practice . This finding is true irrespective of whether the algorithm in theory relies on capturing stable features or not , thus distinguishing our findings from those of Tople et al . ( 2020 ) . When an ideal stable feature learner can be constructed , as in our synthetic datasets , the resultant model has the best privacy across all our experiments ( and also best OOD generalization ) , Moreover , with the same provable differential privacy guarantee , a stable feature learning algorithm obtains better accuracy than ERM , showing the privacy benefits of learning stable features . For the DG literature , our results point to a viable metric for measuring stability of learnt features , a fundamentally difficult task . Rather than OOD generalization , we find that MI robustness is better correlated with amount of stable features learnt and can be used to evaluate quality of DG algorithms . To summarize , our contributions are : • Through extensive experiments , we show that better out-of-distribution generalization does not always imply better privacy . We explain the result through a simple counter-example . • We provide the first empirical evidence that models that learn stable features are robust to membership inference attacks irrespective of the learning objective , thereby extending the theoretical results from Tople et al . ( 2020 ) . Further , for the same added noise for differential privacy , an algorithm with stable features provides better utility . • Current DG methods aimed to learn stable features do not do so even when they exhibit good OOD generalization . Therefore , we propose MI attack metric to evaluate quality of the learnt features , since it measures stable features better than OOD accuracy . 2 BACKGROUND & PROBLEM STATEMENT . We investigate connections between two streams of work : machine learning algorithms for generalization to unseen distributions and membership privacy risks of machine learning models . Out-of-Distribution/Domain Generalization . As standard training algorithms often fail at generalizing to data from a different distribution than the training data , there is increased attention on domain generalization algorithms that perform well on unseen distributions ( Arjovsky et al. , 2019 ; Ahuja et al. , 2020 ; Peters et al. , 2016 ; Hendrycks & Dietterich , 2019 ; Hendrycks et al. , 2020 ; Piratla et al. , 2020 ; Gulrajani & Lopez-Paz , 2020 ; Mahajan et al. , 2021 ) . In a typical domain generalization ( DG ) task , a learning algorithm is given access to data from multiple domains at the training time . The goal is to train a model that has high accuracy on data from both the training domains as well as new unseen domains that the model may encounter once it is deployed in the wild . Formally , different domains correspond to different feature distributions P ( X ) ( covariate shift ) and/or different conditional distributionsP ( Y|X ) ( concept drift ) . To emulate realistic scenarios , the unseen domains are constrained in a reasonable way , for example , domains might be different locations or regions , different views or lighting conditions for photos , etc . Hence , prior work assumes that all domains share some stable features XC that cause the output label Y , for which the ideal function P ( Y|XC ) remains invariant across all the domains ( Arjovsky et al. , 2019 ) . Stable Features or Stable Representation . Given ( x , y ) |dk|i=1 data over k training domains { d1 , d2 .. dk } , a learnt representation Φ ( x ) is called stable if the prediction mechanism conditioned on them , Pd ( y|Φ ( x ) remains invariant for all the domains d ∈ { d1 , d2 , .. dk } . Additionally , we aim to learn stable representations such that classifier learnt upon them is optimal for generalization to unseen domains . Therefore , it is not surprising that state-of-the-art DG algorithms are designed to learn these stable or causal XC features . One class of methods learn causal representation by aiming that any pair of inputs that share the same causal features have the same representation . They use matching-based regularizer such as MatchDG ( Mahajan et al. , 2021 ) , or Perfect-Match that assumes knowledge of ground-truth matched pairs . Other methods like Invariant Risk Minimization ( IRM ) build a representation that is simultaneously optimal across all training domains ( Arjovsky et al. , 2019 ; Ahuja et al. , 2020 ) , a property that is satisfied by the causal features . Another recent approach ( CSD ) aims to separate out input features into two parts such that one of them has common feature weights across domains , and uses only those common ( stable ) features for prediction ( Piratla et al. , 2020 ) . We use all these DG algorithms for our experiments ( Perfect-Match , MatchDG , IRM , CSD ) , in addition to non-stable learning algorithms , Random-Match ( that matches same-class inputs from different domains ( Motiian et al. , 2017 ) ) and ERM . Details of these algorithms are in Supp . B.1 . Membership Privacy . Ensuring membership privacy of ML models means hiding the knowledge of whether a particular record ( or user providing the record ) participated in the training of the model . Leakage of such membership information poses a serious privacy concern in sensitive applications such as healthcare . Remarkably , with only black-box access to the ML model ( and no access to the training data or the model parameters ) , attacks have been created that can guess the membership of an input with high accuracy ( Shokri et al. , 2017 ) . Overfitting to training dataset has been shown as one of the main reasons for membership leakage , both theoretically and empirically ( Yeom et al. , 2018 ; Shokri et al. , 2017 ) . Under bad generalization properties , the confidence in the prediction values and the resultant accuracy will be lower on a non-training point than a training point , and this difference has been exploited to design different variants of the membership inference attack ( Salem et al. , 2019 ; Song & Mittal , 2020 ; Nasr et al. , 2019 ) . For mitigating these attacks , training using differential privacy is the most widely used approach although it incurs substantial degradation in model utility ( Dwork et al. , 2014 ; Abadi et al. , 2016 ) . Research Questions . While prior work has studied the relationship of generalization and membership privacy of ML models for in-distribution setting , it is not well-established how this relationship transfers when we consider out-of-distribution evaluation data . This forms our first research question RQ1 . Does better out-of-distribution generalization always lead to better membership privacy ? As we shall see , the answer to the above question is negative . Given recent theoretical work ( Tople et al. , 2020 ) that shows models learning stable features are more differentially private and robust to membership privacy under certain conditions ( such as causal sufficiency ) , we ask a more nuanced question : RQ2 . Do methods that learn more stable features achieve better membership privacy ? Evaluation Metrics . To answer these questions , we compare DG models on three metrics , a ) Utility : Out-of-Distribution ( OOD ) accuracy on unseen domains ; b ) Privacy : attack accuracy using the loss-based membership inference attack ( Yeom et al. , 2018 ) ; and c ) Stable Features : The amount of stable features learnt . The first is the standard metric used to evaluate DG algorithms ( Gulrajani & Lopez-Paz , 2020 ) and the second is the simplest membership privacy attack metric that distinguishes members ( train domains ) from non-members ( test domains ) with only access to the loss values . The third , measuring stability of learnt features , is a fundamentally difficult question for DG models since stable features may correspond to unobserved high-level latent features . Therefore , we only measure it on the synthetic and semi-synthetic datasets where reliable metrics can be constructed . Further details regarding the evaluation metrics are provided in Supp . A .

Summary: The paper evaluates the connection between out of distribution (OOD) accuracy of a learning model and its privacy protection. They use Membership inference (MI) success rate as a metric to evaluate how much privacy some model provides.   The setting they evaluate is when a model is train data and test data are drawn from different distributions. The main idea is that models trained on data from some distribution should be robust to distribution shift. The key insight from the literature is that models that generalize to out-of-distribution data are able to learn stable features of the data (features that are invariant to modest distribution shifts). The paper empirically evaluates the connection between the generalization ability of a model and its privacy (using the MI metric).