Measuring causal influence with back-to-back regression: the linear case

1 INTRODUCTION . Natural sciences are tasked to find , from a set of hypothetical factors , the minimal subset that suffices to reliably predict novel observations . This endeavor is impeded by two major challenges . First , causal and non-causal factors may be numerous and collinear . This issue becomes increasingly pronounced as the number of potential factors increases . In neuroscience , for example , identifying whether the frequency of a word presented on a subject ’ s retina modulates brain activity can be surprisingly difficult . Indeed , the frequency of words in natural language covaries with other factors such as their length ( short words are more frequent than long words ) , their categories ( determinants are more frequent than adverbs ) and so forth ( Kutas & Federmeier , 2011 ; Pegado et al. , 2014 ) . Instead of selecting a set of words that control for all of these factors simultaneously , it is thus common to use forward modeling , i.e . to train a model to predict observations ( e.g . brain activity ) from a minimal combination of competing factors ( e.g . word length , word frequency , e.g . ( Huth et al. , 2016 ) ) , and investigate , in the model , the estimated contribution of each factor ( Friston et al. , 1994 ) . The second challenge to measuring causal influence is that observations can be large and complex . The relationship between causes and effects is thus often considered in a backward manner , by training models to maximally predict causes from multidimensional observations . For example , brain activity is often recorded with hundreds or thousands of sensors simultaneously . As multiple sensors may be affected by common noise sources , it is common to use backward regression , by , for example , fitting a support vector machine across multiple sensors to decode the category of a stimulus ( Cichy et al. , 2014 ; Kriegeskorte et al. , 2008 ; Norman et al. , 2006 ) . Both forward and backward modeling have competing benefits and drawbacks . Specifically , forward modeling disentangles the independent contribution of collinear factors , but does not combine multidimensional observations . By contrast , backward modeling combines multiple observations , but does not disentangle collinear factors ( Weichwald et al. , 2015 ; Hebart & Baker , 2018 ; King et al. , 2018 ) . To combine the benefits of forward and backward modeling , several authors have proposed to use cross-decomposition techniques such as Partial Least Squares ( PLS ) and Canonical Correlation Analysis ( CCA ) ( de Cheveigne et al. , 2019 ) . CCA and PLS aim to find , from two sets of data X and Y , the components H and G were XH and Y G are maximially correlated or maximally covarying respectively . Because CCA and PLS are based on a generalized eigen decomposition , their resulting coefficients are mixing the features of X and Y in a way that makes them notoriously difficult to interpret ( Lebart et al. , 1995 ) . Here , we introduce the ‘ back-to-back regression ’ ( B2B ) , which not only combines the benefits of forward and backward modeling ( Section 2 ) , but also provide robust , interpretable and unidimensional causal coefficients for each tested factor . The present paper focuses on the restricted issue of disentangling the causal influence of linearly correlated predictors ( X ) onto noisy multivariate observations ( Y ) . The present approach thus differs from other causal discovery algorithms based on temporal-delays and/or nonlinear interactions in systems where the directionality of causation ( from X to Y or vice versa ) is unknown ( e.g . ( Peters et al. , 2017 ; Granger , 1969 ; Janzing et al. , 2013 ; Schölkopf et al. , 2016 ) . After detailing B2B method and proving its convergence ( Section 2.2 ) , we show with synthetic data that it outperforms state-of-the-art forward , backward and cross-decomposition techniques in identifying causal influence ( Section 3.1 ) . Finally , we apply B2B to a large neuroimaging dataset and reveal that distinct but collinear word features lead to distinguishable brain representations ( Section 3.5 ) . 2 BACK-TO-BACK REGRESSION . We consider the measurement of multivariate signal Y ∈ Rn×dy , generated from a set of putative causes X ∈ Rn×dx , via some unknown linear apparatus F ∈ Rdx×dy . Not all the variables in X exert a causal influence on Y . By considering a square binary diagonal matrix of causal influences E ∈ Ddx×dx , we denote by XE the causal factors of Y . In summary , the problem can be formalized as : yi = ( xiE + ni ) F ( 1 ) where i is a given sample , and ni is a sample-specific noise drawen from a centered distribution . While the triplet of variables X and N are independent , we allow each of them to have any form of covariance . In practice , we observe n samples ( X , Y ) from the model . This problem space , along with the sizes of all variables involved , is illustrated in Figure 1 . Given the model in Equation ( 1 ) , the goal of Back-to-Back Regression ( B2B ) is to estimate the matrix of causal influences E . 2.1 ALGORITHM . Back-to-Back Regression ( B2B ) consists of two steps . First , we estimate the linear regression coefficients Ĝ from Y to X , and construct the predictions X̂ = Y Ĝ . This backward regression recovers the correlations between Y and each factor of X . Second , we estimate the linear regression coefficients Ĥ fromX to X̂ . The diagonal of the regression coefficients Ĥ , denoted by Ê = diag ( Ĥ ) , is the desired estimate of the causal influence matrix E , as detailed in the A.1 . If using l2-regularized least-squares ( Hoerl , 1959 ; Rifkin & Lippert , 2007 ) , B2B has a closed form solution : Ĝ = ( Y > Y + ΛY ) −1Y > X , ( 2 ) Ĥ = ( X > X + ΛX ) −1X > Y Ĝ , ( 3 ) where ΛX and ΛY are two diagonal matrices of regularization parameters , useful to invert the covariance matrices of X and Y if these are ill-conditioned . Performing two regressions over the same data sample can result in overfitting , as spurious correlations in the data absorbed by the first regression will be leveraged by the second one . To avoid this issue , we split our sample ( X , Y ) into two splits ( X1 , Y1 ) and ( X2 , Y2 ) . Then , the first regression is performed using ( X1 , Y1 ) , and the second regression is performed using ( X2 , Y2 ) . To compensate for the reduction in sample size caused by the split , B2B is repeated over many random splits , and the final estimate Ê of the causal influence matrix is the average over the estimates associated to each split ( Breiman , 1996 ) . To accelerate this ensembling procedure , we implemented an efficient leave-one-out cross-validation scheme as detailed in ( Rifkin & Lippert , 2007 ) , as follows : Ŷ = ( ΣXGY − diag ( ΣXG ) Y ) / diag ( I − ΣXG ) ( element-wise division ) ( 4 ) where ΣX is the X kernel matrix and where G is computed with an eigen decomposition of X : ΣX = QV Q T G = Q ( V + λI ) −1QT ( 5 ) where Q , V and λ are the eigen vectors , eigen values and regularization , respectively . We summarize the B2B procedure in Algorithm 1 . The rest of this section provides a theoretical guarantee on the correctness of B2B . Algorithm 1 : Back-to-back regression . Input : input data X ∈ Rn×dx , output data Y ∈ Rn×dy , number of repetitions m ∈ N. Output : estimate of causal influences Ê ∈ Ddx×dx . 1 Ê ← 0 ; 2 for i = 1 , . . . , m do 3 ( X , Y ) ← ShuffleRows ( ( X , Y ) ) ; 4 ( X1 , Y1 ) , ( X2 , Y2 ) ← SplitRowsInHalf ( ( X , Y ) ) ; 5 Ĝ = LinearRegression ( Y1 , X1 ) ; . Ĝ = ( Y > 1 Y1 + ΛY ) −1Y > 1 X1 6 Ĥ = LinearRegression ( X2 , Y2Ĝ ) ; . Ĥ = ( X > 2 X2 + ΛX ) −1X > 2 Y2Ĝ 7 Ê ← Ê + diag ( Ĥ ) ; 8 end 9 Ê ← Ê/m ; 10 Ŵ ← LinearRegression ( XÊ , Y ) ; 11 return Ê , Ŵ 2.2 THEORETICAL GUARANTEES . Theorem 1 ( B2B consistency - general case ) . Consider the B2B model from Equation Y = ( XE + N ) F , N centred and full rank noise . Let Img ( M ) refers to the image of the matrix M . If F and X are full-rank on the Img ( E ) , then , the solution of B2B , Ĥ , will minimize minH ‖X −XH‖2 + ‖NH‖2and satisfy EĤ = Ĥ Proof . See Appendix A.1 . Since EĤ = Ĥ , we have Ĥ = arg min H ‖X −XEH‖2 + ‖NEH‖2 = ( EX > XE + EN > NE ) †EXX > . ( 6 ) Assuming , without loss of generality , that the active features in E are the k ∈ Z : k ∈ [ 0 , dx ] first features , and rewritingX = ( X1 , X2 ) andN = ( N1 , N2 ) ( X1 andN1 containing the k first features ) , we have : X > X = ( ΣX1X1 ΣX1X2 ΣX1X2 ΣX2X2 ) , N > N = ( ΣN1N1 ΣN1N2 ΣN1N2 ΣN2N2 ) , ( 7 ) where ΣAB is the covariance of A and B , and : Ĥ = ( ( ΣX1X1 + ΣN1N1 ) −1ΣX1X1 ( ΣX1X1 + ΣN1N1 ) −1ΣX1X2 0 0 ) ( 8 ) diagk ( Ĥ ) = diag ( ( ΣX1X1 + ΣN1N1 ) −1ΣX1X1 ) = diag ( ( I + Σ −1 X1X1 ΣN1N1 ) −1 ) ( 9 ) In the absence of noise , we have ΣN1N1 = 0 , and so diagk ( Ĥ ) = I , and diag ( Ĥ ) = diag ( E ) Therefore , we recover E from Ĥ . In the presence of noise , the causal factors of E correspond to the positive elements of diag ( Ĥ ) . The methods to recover them are presented in the Appendix A.4 . 3 EXPERIMENTS . We perform two sets of experiments to evaluate B2B : one on controlled synthetic data , and a second one on a real , large-scale magneto-encephalography ( MEG ) dataset . We use scikit-learn ’ s PLS and RidgeCV ( Pedregosa et al. , 2011 ) as well as pyrcca ’ s regularized canonical component analysis ( RegCCA , ( Bilenko & Gallant , 2016 ) ) objects to compare B2B against the standard baselines . 3.1 SYNTHETIC DATA . We evaluate the performance of B2B throughout a series of experiments on controlled synthetic data . The purpose of these experiments is to evaluate the ability of B2B in terms of prediction of independent and identically distributed data , as well as a method to recover causal factors . The data generating process for each experiment constructs n = 1000 training examples according to the model Y = ( hXE + N ) F , where h is a scalar that modulates the signal-to-noise ratio . Here , F ∈ Rdx×dy contains entries drawn from N ( 0 , σ2 ) where σ2 is inversely proportional to dx , X ∈ Rn×dx contains rows drawn from N ( 0 , ΣX ) , N ∈ Rn×dx contains rows drawn from N ( 0 , ΣN ) , E ∈ Rdx×dx is a binary diagonal matrix containing nc ones , ΣX = AA > where A ∈ Rdx×dx contains entries drawn from N ( 0 , σ2 ) , ΣN = BB > where B ∈ Rdx×dx contains entries drawn from N ( 0 , σ2 ) , and the factor h ∈ R+ . To simulate a wide range of experimental conditions , we sample 10 values in log-space for dx , dy ∈ [ 10 , 100 ] , nc ∈ [ 3 , 63 ] , h ∈ [ 0.001 , 10 ] . We discard the cases where nc > dx , limit dx , dy to 100 to keep the running time under 2 hours for each condition , and average over 5 random seeds . We compare the performance of B2B against four competing methods , all implemented in scikit-learn ( Pedregosa et al. , 2011 ) and pyrcca ( Bilenko & Gallant , 2016 ) :

Summary: The authors introduce a method (B2B) for disentangling effects of correlated predictors in the context of high dimensional outcomes. Their model assumes the outcomes are constructed by a linear transformation of a set of true causes plus measurement noise. Specifically, they fit the model Y=(XE+N)F, where X are the predictors, E is a binary matrix indicating the true causes, N is a noise term, and F is a mixing term. They provide a closed form solution the model fit based on a pair of l2-regularized regressions. They simulate from the given model and provide comparisons against least-squares regression, canonical correlation analysis, and partial least squares. They also apply the method to brain imaging data containing 102 individuals reading 120 sentences plus scrambled sentences, with the goal of inferring which features of the words have an effect on imaging results.