Title: Synthetic Combinations: A Causal Inference Framework for Combinatorial Interventions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causal Inference

Matrix Completion

Combinatorial Learning

Ranking
Abstract: We consider a setting where there are $N$ heterogeneous units and $p$ interventions. Our goal is to learn unit-specific potential outcomes for any combination of these $p$ interventions, i.e., $N \times 2^p$ causal parameters. Choosing a combination of interventions is a problem that naturally arises in a variety of applications such as factorial design experiments and recommendation engines (e.g., showing a set of movies that maximizes engagement for a given user). Running $N \times 2^p$ experiments to estimate the various parameters is likely expensive and/or infeasible as $N$ and $p$ grow. Further, with observational data there is likely confounding, i.e., whether or not a unit is seen under a combination is correlated with its potential outcome under that combination. We study this problem under a novel model that imposes latent structure across both units and combinations of interventions. Specifically, we assume latent similarity in potential outcomes across units (i.e., the matrix of potential outcomes is approximately rank $r$) and regularity in how combinations of interventions interact (i.e., the coefficients in the Fourier expansion of the potential outcomes is approximately $s$ sparse). We establish identification for all $N \times 2^p$ parameters despite unobserved confounding. We propose an estimation procedure, Synthetic Combinations, and establish finite-sample consistency under precise conditions on the observation pattern. We show that Synthetic Combinations is able to consistently estimate unit-specific potential outcomes given a total of $\text{poly}(r) \times \left( N + s^2p\right)$ observations. In comparison, previous methods that do not exploit structure across both units and combinations have poorer sample complexity scaling as $\min(N \times s^2p, \ \ r \times (N + 2^p))$.

Synthetic Combinations: ACausal Framework for Combinatorial Interventions

Abstract

We consider a setting where there are N heterogeneous units and p interventions. Our goal is to learn unit-specific potential outcomes for any combination of these p interventions, i.e., N × 2 p causal parameters. Choosing a combination of interventions is a problem that naturally arises in a variety of applications such as factorial design experiments and recommendation engines (e.g., showing a set of movies that maximizes engagement). Running N × 2 p experiments to estimate the various parameters is likely expensive and/or infeasible as N and p grow. Further, with observational data there is likely confounding, i.e., whether or not a unit is seen under a combination is correlated with its potential outcome. We study this problem under a novel model that imposes latent structure across both units and combinations of interventions. Specifically, we assume latent similarity in potential outcomes across units (i.e., the matrix of potential outcomes is rank r ) and regularity in how combinations of interventions interact (i.e., the coefficients in the Fourier expansion of the potential outcomes is s sparse). We establish identification for all N × 2 p parameters despite unobserved confounding. We propose an estimation procedure, Synthetic Combinations, and establish finite-sample consistency under precise conditions on the observation pattern. We show that Synthetic Combinations is able to consistently estimate unit-specific potential outcomes given a total of poly ( r ) × ( N + s 2 p ) observations. In comparison, previous methods that do not exploit structure across both units and combinations have poorer sample complexity that scales as min( N × s 2 p, poly ( r ) × ( N +2 p )) .

1 Introduction

Modern-day decision makers-in settings from e-commerce to public policy to medicine-often must select a combination of actions, and would like to do so in a highly personalized manner. Examples include recommending a curated basket of items to customers on a commerce platform, deciding on a combination of therapies for a medical patient, enacting a collection of socio-economic policies for a specific geographic location, conjoint analysis in surveys, selecting important feature sets for machine learning models, etc. Despite the ubiquity of this setting, it comes with significant empirical challenges: with p interventions and N units, a decision maker must evaluate N × 2 p potential combinations in order to confirm the optimal personalized policy. With large N and even with relatively small p (due to exponential dependence), it becomes infeasible to run that many experiments; in observational data there is the additional challenge of potential unobserved confounding. Current methods tackle this problem by following one of two approaches: (i) they impose structure on how combinations of interventions interact, or (ii) they assume latent similarity

Amazon, Core AI

in potential outcomes across units. However, as we discuss in detail below, these approaches require a large number of observations to estimate all N × 2 p potential outcomes because they do not exploit structure across both units and combinations. This motivates the question: how can one effectively share information across both units and combinations of interventions?

Contributions. Our contributions may be summarized as follows. (1) For a unit n ∈ [ N ] , we represent its potential outcomes over the 2 p combinations as a Boolean function from {-1 , 1 } p to R , expressed in the Fourier basis. To impose structure across combinations, we assume that for a unit n , the linear coefficients α n ∈ R 2 p induced by this Fourier basis representation is s -sparse, i.e., has at most s nonzero entries. To impose structure across units, we assume that this matrix of Fourier coefficients across units A =[ α n ] n ∈ [ N ] ∈ R N × 2 p has rank r . This simultaneous sparsity and low-rank assumption allows the researcher to share information across both units and combinations. (2) We establish identification for the N × 2 p potential outcomes of interest, which requires that any confounding is mediated by the (unobserved) matrix of Fourier coefficients A . (3) We design a two-step algorithm 'Synthetic Combinations' and prove it consistently estimates the various causal parameters, despite potential unobserved confounding. The first step of Synthetic Combinations, termed 'horizontal regression', learns the structure across combinations of interventions-assuming sparsity in the Fourier coefficients-via the Lasso. The second step, termed 'vertical regression', learns the structure across units-assuming the matrix of Fourier coefficients are low-rank-via principal component regression (PCR). (4) We show that Synthetic Combinations is able to consistently estimate unit-specific potential outcomes given a total of poly ( r ) × ( N + s 2 p ) observations (ignoring logarithmic factors). This improves over previous methods that do not exploit structure across both units and combinations, which have sample complexity scaling as min( N × s 2 p, poly ( r ) × ( N +2 p )) . A summary of the sample complexities required for different methods can be found in Table 1. A key technical challenge in our proofs is analyzing how the error induced in the first step of Synthetic Combinations percolates through to the second step. To tackle it, we reduce the problem to that of high-dimensional error-in-variables regression with linear model misspecification, and do a novel analysis of this statistical setting.

2 Related Work

Learning over Combinations. To place structure on the space of combinations, we use tools from the theory of learning (sparse) Boolean functions, in particular the Fourier transform. Sparsity of the Fourier transform was proposed as a complexity measure to characterize and design learning algorithms for low-depth trees, low-degree polynomials, and small circuits [12, 26]. Learning Boolean functions is now a central topic in learning theory, and is closely related to many important questions in MLmorebroadly; see e.g. [31] for discussion of the k -Junta problem and its relation to relevant feature selection. We refer to O'Donnell [34, Chapter 3] for further background on this area. In this paper, we focus on [32] which showed that Lasso can be used to efficiently learn sparse Boolean functions, an essential property in our setting since the dimension of the function class grows exponentially.

Matrix Completion. We build on the observation that imputing counterfactual outcomes in the presence of a latent factor structure can be equivalently expressed as low-rank matrix completion [9, 8, 5]. The observation that low-rank matrices may typically be recovered from a small fraction of the entries by nuclear-norm minimization has had a major impact on modern statistics [15, 35, 13]. In the noisy setting, proposed estimators have generally proceeded by minimizing risk subject to a nuclear-norm penalty, such as in the SoftImpute algorithm of [30], or minimizing risk subject to a rank constraint as in the hard singular-value thresholding (HSVT) algorithms analyzed by [27, 28, 23, 16]. We refer the reader to [19, 33] for a comprehensive overview of this vast literature.

Econometrics/causal inference. There is a rich literature on how to learn personalized treatment effects for heterogeneous units. This problem is of particular importance in the social sciences and in medicine, where experimental data is limited, and has led to several promising approaches including instrumental variables, difference-in-differences, regression discontinuity, and others; see [7, 25] for an overview. Of particular interest to us here is the 'synthetic control' method [1], which exploits an underlying factor structure to effectively 'share' counterfactual information between treatment and control units. Building on [1], recent work has shown that the same underlying structure can be used to estimate treatment effects with multiple treatments and heterogeneous units despite unmeasured confounding. The resulting framework, called 'synthetic interventions' (SI), uses the factor representation to efficiently share information across similar (yet distinct) units and treatments [5]. We generalize the SI framework to settings where treatments are combinations of interventions and most treatments have no units that receive it, but there is structure across combinations of interventions. In doing so, we enable its use in highly practical cases where multiple interventions are delivered simultaneously, such as recommender systems, medical treatment regimens, and factorial design experiments.

3 Setup and Model

In this section, we first describe requisite notation, background on the Fourier expansion of real-valued functions over booleans, and how it relates to potential outcomes over combinations.

3.1 Notation

Representation of Combinations as Binary Vectors . Let [ p ] = { 1 , ... p } denote the set of p interventions. Denote by Π the power set of [ p ] , i.e., the set of all possible combinations of p interventions, where we note | Π | =2 p . Then, any given combination π ∈ Π induces the following binary representation v ( π ) ∈{-1 , 1 } p defined as follows: v ( π ) i =2 1 { i ∈ π }-1 .

̸

Fourier Expansion of Boolean Functions. Let F bool = { f : {-1 , 1 } p → R } be the set of all realvalued functions defined on the hypercube {-1 , 1 } p . Then F bool forms a Hilbert space defined by the following inner product: for any f,g ∈F bool , ⟨ f,g ⟩ B = 1 2 p ∑ x ∈{-1 , 1 } p f ( x ) g ( x ) . This inner product induces the norm ⟨ f,f ⟩ B := ∥ f ∥ 2 B = 1 2 p ∑ x ∈{-1 , 1 } p f 2 ( x ) . We construct an orthonormal basis for F bool as follows: for each subset S ⊂ [ p ] , define a basis function χ S ( x )= ∏ i ∈ S x i where x i is the i th coefficient of x ∈{-1 , 1 } p . One can verify that for any S ⊂ [ p ] that ∥ χ S ∥ B =1 , and that ⟨ χ S ,χ S ′ ⟩ B = 0 for any S ′ = S . Since |{ χ S : S ⊂ [ p ] }| =2 p , the functions χ S are an orthonormal basis of F bool .

Hence, any f ∈ F bool can be expressed via the following 'Fourier' decomposition: f ( x ) = ∑ S ⊂ [ p ] α S χ S ( x ) , where the Fourier coefficient α S is given by computing α S = ⟨ f, χ S ⟩ B . We will refer to χ S as the Fourier character. Define α f = [ α S ] S ∈ [ p ] ∈ R 2 p and χ ( x )=[ χ S ( x )] S ∈ [ p ] ∈{-1 , 1 } 2 p as the vector of Fourier coefficients and characters respectively. Hence any function f : {-1 , 1 } p → R can be re-expressed as follows: f ( x )= ⟨ α f , χ ( x ) ⟩ . For π ∈ Π , abbreviate χ S ( v ( π )) and χ ( v ( π )) as χ π S and χ π respectively.

Observed and potential outcomes. Let Y ( π ) n ∈ R denote the potential outcome for unit n under combination π and Y nπ ∈{ R ∪ ⋆ } as the observed outcome , where ⋆ indicates a missing value, i.e., the outcome associated with the unit-combination pair ( n,π ) was not observed. Let Y =[ Y nπ ] ∈ { R ∪ ⋆ } N × 2 p . Let D⊂ [ N ] × [2 p ] , refer to the subset of unit-combination pairs we do observe, i.e.,

Note that (1) implies stable unit treatment value assignment (SUTVA) holds. Let Π S ⊆ Π denote a subset of combinations. For a given unit n , let Y Π S ,n =[ Y nπ i : π i ∈ Π S ] ∈{ R ∪ ⋆ } | Π S | represent the vector of observed outcomes for all π ∈ Π S . Similarly, let Y (Π S ) n =[ Y ( π i ) n : π i ∈ Π S ] ∈ R | Π S | represent the vector of potential outcomes. Denote χ (Π S )=[ χ π i : π i ∈ Π S ] ∈{-1 , 1 } | Π S |× 2 p .

3.2 Model & Target Causal Parameter

Define Y ( · ) n : π → R as a real-valued function over the hypercube {-1 , 1 } p associated with unit n . It takes as input a combination π , converts it to a p -dimensional binary vector v ( π ) , and outputs a real number Y ( π ) n . Given the discussion in Section 3.1, it follows that Y ( π ) n always has the representation ⟨ α n , χ π ⟩ for some α n ∈ R 2 p . Thus, without any loss of generality, the α n are unit-specific latent variables (Fourier coefficients) encoding the treatment response function. Below, we state our key assumption on these induced Fourier coefficients.

Assumption 3.1 (Potential Outcome Model) . For any unit-combination pair ( n,π ) , we assume it has the following representation,

where α n ∈ R 2 p and χ π ∈{-1 , 1 } 2 p are the Fourier coefficients and characters, respectively. We assume the following properties: (a) low-rank: the matrix A =[ α n ] n ∈ [ N ] has rank r ∈ [min { N, 2 p } ] ; (b) sparsity: α n is s -sparse (i.e. ∥ α n ∥ 0 ≤ s , where s ∈ [2 p ] ) for every unit n ∈ [ N ] ; (c) ϵ π n is a residual term specific to ( n,π ) which satisfies E [ ϵ π n | A ] =0 .

The assumption is then that each α n s -sparse and A is rankr ; ϵ π n is the residual from this sparse and low-rank approximation, and it serves as the source of uncertainty in our model. Given E [ ϵ π n | A ] =0 , the matrix E [ Y (Π) N ] = [ E [ Y (Π) n ] : n ∈ [ N ]] ∈ R 2 p × N also is rank r , where the expectation is defined with respect to ϵ π n . This is because E [ Y (Π) N ] can be written as E [ Y (Π) N ] = χ (Π) A T , and since χ (Π) is an invertible matrix, rank ( E [ Y (Π) N ]) = rank ( A ) . The low-rank property places structure across units ; that is, we assume there is sufficient similarity across units so that E [ Y (Π) n ] for any unit n can be written as a linear combination of r other rows of E [ Y (Π) N ] . This is a standard assumption used to encode latent similarity across units in matrix completion and its related applications (e.g., recommendation engines).

Sparsity establishes unit-specific structure; that is, we assume that the potential outcomes for a given user only depend on a small subset of the functions { χ S : S ⊂ [ p ] } . We emphasize that this subset of functions can be different across units. As discussed in Section 2, sparsity is commonly employed when studying the learnability of Boolean functions. In the context of recommendation engines, sparsity is implied if the ratings for a set of goods only depend on a small number of combinations of items within that set. Sparsity is also often assumed implicitly in factorial design experiments, where analysts typically only include pairwise interaction effects between interventions and ignore higher-order interactions [24]. We discuss further applications of combinatorial inference in greater detail in Appendix A.

Next, we present an assumption that formalizes the dependence (i.e., confounding) between the missingness pattern induced by the treatment assignments D and the potential outcomes Y ( π ) n , and provide an interpretation of the induced data generating process (DGP) for potential outcomes.

Assumption 3.2 (Selection on Fourier coefficients) . For all n ∈ [ N ] and π ∈ Π , Y ( π ) n ⊥ ⊥ D | A .

DGP. Given Assumptions 3.1 and Assumption 3.2, the DGP can be summarized as follows: (i) unitspecific latent Fourier coefficients A are either deterministic or sampled from an unknown distribution; we will condition on this quantity throughout. (ii) Given A , we sample mean-zero random variables ϵ π n , and generate potential outcomes according to our model Y ( π ) n = ⟨ α n ,χ π ⟩ + ϵ π n . (iii) D is allowed to depend on unit-specific latent Fourier coefficients A (i.e. D = f ( A ) ). We define all expectations w.r.t. noise, ϵ π n . This DGP introduces unobserved confounding since Y ( π ) n ̸⊥ ⊥ D . However, this DGP does imply that Assumption 3.2 holds, i.e., conditional on the Fourier coefficients A , the potential outcomes are independent of the treatment assignments D . This conditional independence condition can be thought of as 'selection on latent Fourier coefficients', which generalizes the widely made assumption of 'selection on observables.' The latter requires that potential outcomes are independent of treatment assignments conditional on observed covariates-we reemphasize that A is unobserved.

Target parameter. For any unit-combination pair ( n,π ) , we aim to estimate E [ Y ( π ) n | A ] , where the expectation is w.r.t. ϵ π n , and we condition on the set of Fourier coefficients A .

4 Identification of Potential Outcomes

We show that E [ Y ( π ) n | A ] can be written as a function of observed outcomes, i.e., we establish identification of our target causal parameter. As discussed earlier, our model allows for unobserved confounding : whether or not a unit is seen under a combination may be correlated with its potential outcome under that combination due to unobserved factors, as long as certain conditions are met. We introduce necessary notation and assumption required for our result. For a unit n ∈ [ N ] , denote the subset of combinations we observe them under as Π n ⊆ Π . For π ∈ Π , let ˜ χ π n ∈ R 2 p denote the vector where we zero out all coordinates of χ π ∈ R 2 p that correspond to the coefficients of α n which are zero. For example, if α n =(1 , 1 , 0 , 0 ,... 0) and χ π =(1 , 1 ,... 1) , then ˜ χ π n =(1 , 1 , 0 ,... 0) . We the make the following assumption.

Assumption 4.1 (Donor Units) . We assume there exists a set of 'donor units' I ⊂ [ N ] , such that the following two conditions hold:

(a) Horizontal span inclusion: For any donor unit u ∈ I and combination π ∈ Π , suppose ˜ χ π u ∈ span ( ˜ χ π i u : π i ∈ Π u ) . That is, there exists exists β π Π u ∈ R | Π u | such that ˜ χ π u = ∑ π i ∈ Π u β π π i ˜ χ π i u .

(b) Linear span inclusion: For any unit n ∈ [ N ] \I , suppose α n ∈ span ( α u : u ∈I ) . That is, there exists w n such that α n = ∑ u ∈I w n u α u

Horizontal span inclusion requires that the set of observed combinations for any donor unit is 'diverse' enough that the projection of the Fourier characteristic for a target intervention is in the span of characteristics of observed interventions. Linear span inclusion requires that the donor set is diverse enough such that the Fourier coefficient of any unit is in the span of the Fourier coefficients of the donor set.

4.1 Identification Result

Given these assumptions, we now present our identification theorem.

Theorem 4.2. Let Assumptions 3.1, 3.2, 4.1 hold. Given β π Π u and w n u defined in Assumption 4.1, we have

(a) Donor units: For u ∈I , and π ∈ Π , E [ Y ( π ) u | A ] = ∑ π u ∈ Π u β π π u E [ Y u,π u | A , D ] .

(b) Non-donor units: For n ∈ [ N ] \I , and π / ∈ Π n , E [ Y ( π ) n | A ] = ∑ u ∈I ,π u ∈ Π u w n u β π π u E [ Y u,π u | A , D ] .

Theorem 4.2 gives conditions under which the donor set I and the observation pattern D are sufficient to recover the full set of unit specific potential outcomes E [ Y ( π ) n |A ] in the noise-free limit. Part (a) establishes that for every donor unit u ∈I , the causal estimand can be written as a function of its own observed outcomes E [ Y Π u ] , given knowledge of β π Π u . Part (b) states that the target causal estimand E [ Y ( π ) n ] for a non-donor unit and combination π can be written as a linear combination of the outcomes of the donor set I , given knowledge of w n n . Previous works that establish identification under a latent factor model requires a growing number of donor units to be observed under all treatments [6]. This is infeasible in our setting because the vast majority of combinations have no units that receive it . As a result, we have to first to identify the outcomes of donor units under all combinations (part (a)), before transferring them to non-donor units (part (b)). In order to do so,

Theorem 4.2 suggests that the key quantities in estimating E [ Y ( π ) n ] for any unit-combination pair ( n,π ) are β π Π u and w n u . In the following section, we provide an algorithm to estimate both β π Π u and w n u , as well as concrete ways of determining the donor set I .

5 The Synthetic Combinations Estimator

We now describe the Synthetic Combinations estimator, a simple and flexible two-step procedure for estimating our target causal parameter. See Figure 1 for a pictorial representation.

Step 1: Horizontal Regression. We denote the vector of observed responses Y n, Π n =[ Y nπ : π ∈ Π n ] ∈ R | Π n | for any unit n as Y Π n . Then, for every unit u in the donor set I , we estimate E [ Y ( π ) u ]

via the Lasso, i.e., by solving the following convex program with penalty parameter λ u :

where recall that χ (Π u ) = [ χ π : π ∈ Π u ] ∈ R | Π u |× 2 p . Then, for any donor unit-combination pair ( u,π ) , let ˆ E [ Y ( π ) u ] = ⟨ ˆ α u , χ π ⟩ denote the estimate of the potential outcome E [ Y ( π ) u ] .

Step 2: Vertical Regression. Next, we estimate potential outcomes for all units n ∈ [ N ] \I . To do so, we define some required notation. For Π S ⊆ Π , define the vector of estimated potential outcomes ˆ E [ Y (Π S ) u ] =[ ˆ E [ Y ( π ) u ] : π ∈ Π S ] ∈ R | Π S | . Additionally, let ˆ E [ Y (Π S ) I ] =[ ˆ E [ Y (Π S ) u ] : u ∈I ] ∈ R | Π S |×|I| .

Step 2(a): Principal Component Regression. Perform a singular value decomposition (SVD) of ˆ E [ Y (Π n ) I ] to get ˆ E [ Y (Π n ) I ] = ∑ min( | Π n | , |I| ) l =1 ˆ s l ˆ µ l ˆ ν T l . Using a hyper-parameter κ ≤ min( | Π n | , |I| ) 3 , compute ˆ w n ∈ R |I| as follows:

Step 2(b): Estimation. Using ˆ w n =[ ˆ w n u : u ∈I ] , we have the following estimate for any π ∈ Π

IIn

π

u

[y(]

n

*

米

W

-n

IY

[Y]

YIu

[Y(])

Iu

(a) Observation pattern

(b) Horizontal regression

(c) Vertical regression

Suitability of Lasso and PCR. Lasso is appropriate for the horizontal regression because α u is sparse. However, Synthetic Combinations allows for any ML algorithm (e.g., neural networks, random forests) to be used in the first step. This flexibility allows an analyst to tailor the horizontal learning procedure, and include prior information. However, we leave this model-agnostic analysis as future work, and focus on the Lasso for simplicity. PCR is appropriate for the vertical regression because A is low rank. As [3, 4] show, PCR implicitly regularizes the regression by adapting to the rank of the covariates ( Y Π n ), i.e., the out-of-sample error of PCR scales with r rather than the ambient covariate dimension.

Determining Donor Set I . Synthetic Combinations requires the existence of a subset of units I ⊂ [ N ] such that we are able to (i) accurately estimate their potential outcomes under all possible combinations, and (ii) transfer these estimated outcomes to a unit n ∈ [ N ] \I . Theoretically, we detail sufficient conditions on the observation pattern such that we are able to perform (i) and (ii) accurately via the Lasso and PCR respectively. In practice, we recommend the following to determine I . For every unit n ∈ [ N ] , learn a separate Lasso model α n and assess its performance through crossvalidation (CV). Assign units with low CV error (with a pre-determined threshold) as the donor set I ,

and estimate outcomes ˆ E [ Y ( π ) u ] for every unit u ∈I and π ∈ Π . For non-donor units, PCR performance can also be assessed via k-fold CV. For units with low PCR error, linear span inclusion (Assumption 4.1(b)) and the assumptions required for the generalization for PCR likely hold, and hence we estimate their potential outcomes as in (5). For units with large PCR error, it is either unlikely that these set of assumptions holds or that | Π n | is not large enough (i.e., additional experiments need to be run for this unit), and hence we do not recommend estimating their counterfactuals.

6 Synthetic Combinations Theoretical Analysis

In this section, we establish finite-sample consistency of Synthetic Combinations, starting with a discussion of the additional assumptions required for our results.

6.1 Additional Assumptions

Assumption 6.1 (Bounded Potential Outcomes) . We assume that E [ Y ( π ) n ] ∈ [ -1 , 1] for any unit-combination pair ( n,π ) .

Assumption6.2 (Sub-Gaussian Noise) . Conditioned on A , for any unit-combination pair ( n,π ) , ϵ π n are independent zero-mean sub-Gaussian random variables with Var [ ϵ π n | A ] ≤ σ 2 and ∥ ϵ π n | A∥ ψ 2 ≤ Cσ for some constant C> 0 .

Assumption 6.3 (Incoherence of Donor Fourier characteristics) . For every unit u ∈I , assume χ (Π u ) satisfies incoherence: ∥ ∥ ∥ χ (Π u ) T χ (Π u ) | Π u | -I 2 p ∥ ∥ ∥ ∞ ≤ C ′ s , for a universal constant C ′ > 0 .

To define our next set of assumptions, we introduce necessary notation. For any subset of combinations Π S ⊂ Π , let E [ Y (Π S ) I ] =[ E [ Y (Π S ) u ] : u ∈I ] ∈ R | Π S |×|I| .

Assumption 6.4 (Donor Unit Balanced Spectrum) . For a given unit n ∈ [ N ] \ I , let r n and s 1 ...s r n denote the rank and non-zero singular values of E [ Y (Π n ) I | A ] . We assume that the singular values are well-balanced, i.e., for universal constants c, c ′ > 0 , we have that s r n /s 1 ≥ c , and ∥ E [ Y (Π n ) I | A ] ∥ 2 F ≥ c ′ | Π n ||I| .

Assumption 6.5 (Subspace Inclusion) . For a given unit n ∈ [ N ] \I and intervention π ∈ Π \ Π n , assume that E [ Y ( π ) I ] lies within the row-span of E [ Y (Π n ) I ]

Assumption 6.3 is necessary for finite-sample consistency when estimating α n via the Lasso estimator (3), and is commonly made when studying the Lasso [36]. Incoherence can also seen as a inclusion criteria for a unit n to be included in the donor set I . Assumption 6.3 for example holds (with high probability) if Π u is chosen uniformly at random and grows as ω ( s 2 p ) as shown in Lemma 2 of [32]. Assumption 6.4 requires that the non-zero singular values of E [ Y (Π n ) I | A ] are well-balanced. This assumption is standard when studying PCR [3, 5], and within the econometrics literature [10, 22]. It can also be empirically validated by plotting the spectrum of ˆ E [ Y (Π n ) I ] ; if the singular spectrum of ˆ E [ Y (Π n ) I ] displays a natural elbow point, then Assumption 6.4 is likely. Assumption 6.5 is also commonly made when analyzing PCR [4, 5, 2]. It can be thought of as a 'causal transportability' condition from the model learnt using Π n to the interventions π ∈ Π \ Π n . That is, subspace inclusion allows us to generalize well, and accurately estimate E [ Y ( π ) n | A ] using ⟨ ˆ E [ Y (Π n ) I ] , ˆ w n ⟩ .

6.2 Finite Sample Consistency

The following result establishes finite-sample consistency of Synthetic Combinations. Without loss of generality, we will focus on estimating the pair of quantities ( E [ Y ( π ) u ] , E [ Y ( π ) n ]) for a given donor unit u ∈I , and non-donor unit n ∈ [ N ] \I under treatment assignment π ∈ Π . To simplify notation, we will use O p notation: for any sequence of random vectors X n , X n = O p ( γ n ) if, for any ϵ > 0 , there exists constants c ϵ and n ϵ such that P ( ∥ X n ∥ 2 ≥ c ϵ γ n ) ≤ ϵ for every n ≥ n ϵ . Similarly, we define ˜ O p ( γ n ) which suppresses logarithmic terms. We will further absorb dependencies on σ into ˜ O p ( · ) .

Theorem 6.6 (Finite Sample Consistency of Synthetic Combinations) . Conditioned on A , let Assumptions 3.1-4.1, and 6.1-6.5 hold. Then, the following statements hold.

(a) For the given donor unit-combination pair ( u,π ) , let the Lasso regularization parameter satisfy λ u =Ω( √ p | Π u | ) . Then, we have that: | ˆ E [ Y ( π ) u ] -E [ Y ( π ) u ] | = ˜ O p (√ s 2 p | Π u | ) .

(b) For the given unit-combination pair ( n,π ) where n ∈ [ N ] \I , let κ = rank ( E [ Y (Π n ) I ]) := r n . Then, provided that min u ∈I | Π u | := M = ω ( r 2 n s 2 p ) , we have that:

Establishing Theorem 6.6 requires a novel analysis of error-in-variables (EIV) linear regression. Specifically, the general EIV linear model is as follows: Y = X β + ϵ , Z = X + H , where Y and Z are observed. In our case, Y = Y Π n , X = E [ Y (Π n ) I ] , β = w n , Z = ˆ E [ Y (Π n ) I ] , and H is the error arising in estimating E [ Y (Π n ) I ] via the Lasso. Typically one assumes that H is is a matrix of independent sub-gaussian noise. Our analysis requires a novel worst-case analysis of H (due to the 2-step regression of Lasso and then PCR), in which each entry of is H simply bounded.

Next, we describe the conditions placed on Π n , M , I to achieve consistent estimation of ( E [ Y ( π ) u ] , E [ Y ( π ) n ]) . To simplify the discussion, we assume that min {| Π n | , |I|} = |I| . This condition can be enforced in practice by simply picking a subset of donor units such that |I| ≤ | Π n | when performing PCR (i.e., step 2 of Synthetic Combinations). Given this assumption, it can be verified that | Π n | and M need to scale as ω ( r 4 n ) and ω ( r 4 n s 2 p ) respectively to achieve max ( | ˆ E [ Y ( π ) u ] -E [ Y ( π ) u ] | , | ˆ E [ Y ( π ) n ] -E [ Y ( π ) n ] | ) =˜ o p (1) . Next, we present a corollary that discusses how quickly the parameters r n , s , and p can grow with the number of observations | Π n | and M .

Corollary 6.7. With the set-up of Theorem 6.6, if (a) r n = o ( | Π n | 1 / 4 ) and (b) s = o ( √ M/ ( p max {| Π n | , |I|} ) ) then max ( | ˆ E [ Y ( π ) u ] -E [ Y ( π ) u ] | , | ˆ E [ Y ( π ) n ] -E [ Y ( π ) n ] | ) = ˜ o p (1) as M, | Π n | , |I|→∞ .

Corollary 6.7 quantifies how s,r n can scale with the number of observations to achieve consistency. That is, Corollary 6.7 reflects the maximum 'complexity' allowed for a given sample size.

6.3 Sample Complexity

We discuss the sample complexity of Synthetic Combinations to estimate all N × 2 p causal parameters, and compare it to that of other methods. To ease our discussion, we will ignore dependence on logarithmic factors and σ . Even if potential outcomes Y ( π ) n were observed for all unit-combination pairs, consistently estimating E [ Y (Π) N ] is not trivial. This is because, we only get to observe a single and noisy version Y nπ = ⟨ α n , χ π ⟩ + ϵ π n . Hypothetically, if we observe K independent samples of Y nπ for a given ( n,π ) , denoted by Y 1 nπ ,...,Y K nπ , the maximum likelihood estimator would be the empirical average 1 K ∑ K i =1 Y i nπ . The empirical average would concentrate around E [ Y ( π ) n ] at a rate O (1 / √ K ) and hence would require K =Ω( δ -2 ) samples to estimate E [ Y ( π ) n ] within error O ( δ ) . Therefore, this naive (unimplementable) solution would require N × 2 p × δ -2 observations to estimate E [ Y (Π) N ] .

On the other hand, Synthetic Combinations produces consistent estimates of the potential outcome despite being given at most only a single noisy sample of each potential outcome. As the discussion after Theorem 6.6 shows, if |I|≤| Π n | , Synthetic Combinations requires |I|× r 4 s 2 p/δ 2 observations for the donor set, and ( N -|I| ) × r 4 /δ 4 observations for the non-donor units to achieve an estimation error of O ( δ ) for all N × 2 p causal parameters. Hence, we have that the number of observations required to achieve an estimation error of O ( δ ) for all pairs ( n,π ) scales as O ( poly ( r/δ ) × ( N + s 2 p )) .

Sample Complexity Comparison to Other Methods. Horizontal regression: An alternative algorithm would be to learn an individual model for each unit n ∈ [ N ] . That is, run a separate horizontal regression via the Lasso for every unit. This alternative algorithm has sample complexity that scales at least as O ( N × s 2 p/δ 2 ) rather than O ( poly ( r ) /δ 4 × ( N + s 2 p )) required by Synthetic Combinations. It suffers because it does not utilize any structure across units (i.e., the low-rank property of A ), whereas Synthetic Combinations captures the similarity between units via PCR.

Matrix completion: Synthetic Combinations can be thought of as a matrix completion method; estimating E [ Y ( π ) n ] is equivalent to imputing ( n,π ) -th entry of the observation matrix Y ∈{ R ∪ ⋆ } N × 2 p , where recall ⋆ denotes an unobserved unit-combination outcome. Under the low-rank property (Assumption 3.1(b)) and various models of missingness (i.e., observation patterns), recent works on matrix completion [14, 29, 6] (see Section 2 for an overview) have established that estimating E [ Y ( π ) n ] to an accuracy O ( δ ) requires at least O ( poly ( r/δ ) × ( N +2 p )) samples. This is because matrix completion techniques do not leverage the sparsity of α n . Moreover, matrix completion results typically report error in the Frobenius norm, whereas we give entry-wise guarantees. This leads to an extra factor of s in our analysis as it requires proving convergence for ∥ ˆ α n -α n ∥ 1 rather than ∥ ˆ α n -α n ∥ 2 . Hence, Synthetic Combinations combines the best of both approaches by leveraging both the structure of the potential outcomes and the similarity across units.

Natural Lower Bound on Sample Complexity. We provide an informal discussion on the lower bound sample-complexity to estimate all N × 2 p potential outcomes. As established in Lemma E.1, A has at most rs non-zero columns. Counting the parameters in the singular value decomposition of A , only r × ( N + rs ) free parameters are required to be estimated. Hence, a natural lower bound on the sample complexity scales as O ( Nr + r 2 s ) . Hence, Synthetic Combinations is only sub-optimal by a factor (ignoring logarithmic factors) of sp and poly ( r ) . As discussed earlier, an additional factor of s can be removed if we focus on deriving Frobenius norm error bounds. It remains as interesting future work to derive estimation procedures that are able to achieve this lower bound.

7 Conclusion

In this work, we formulate a causal inference framework for combinatorial interventions, a setting that is ubiquitous in practice. We propose a model that imposes both unit-specific structure, and latent similarity across units. Under this model, we propose an estimation procedure, Synthetic Combinations, that exploits the sparsity and low-rankness of the Fourier coefficients to efficiently estimate all N × 2 p causal parameters. We formally establish finite-sample consistency of Synthetic Combinations in an observational setting. Our work also naturally suggests future directions for research such as extending Synthetic Combinations to permutations over items (i.e., rankings), or providing an analysis of Synthetic Combinations that is agnostic to the horizontal regression algorithm used. A related line of work to the question above is also deriving estimation algorithms that can achieve the sample complexity lower bound discussed in Section 6.3.

References

A. Abadie, A. Diamond, and J. Hainmueller. Synthetic control methods for comparative case studies: Estimating the effect of california's tobacco control program. Journal of the American statistical Association , 105(490):493-505, 2010.

A. Agarwal and R. Singh. Causal inference with corrupted data: Measurement error, missing values, discretization, and differential privacy. arXiv preprint arXiv:2107.02780 , 2021.

A. Agarwal, D. Shah, D. Shen, and D. Song. On robustness of principal component regression. Advances in Neural Information Processing Systems , 32, 2019.

A. Agarwal, D. Shah, and D. Shen. On principal component regression in a high-dimensional error-in-variables setting. arXiv preprint arXiv:2010.14449 , 2020.

A. Agarwal, D. Shah, and D. Shen. Synthetic interventions. arXiv preprint arXiv:2006.07691 , 2020.

A. Agarwal, M. Dahleh, D. Shah, and D. Shen. Causal matrix completion. arXiv preprint arXiv:2109.15154 , 2021.

J. D. Angrist, G. W. Imbens, and D. B. Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association , 91(434):444-455, 1996.

S. Athey, M. Bayati, N. Doudchenko, G. Imbens, and K. Khosravi. Matrix completion methods for causal panel data models. Journal of the American Statistical Association , pages 1-41, 2021.

J. Bai and S. Ng. Matrix completion, counterfactuals, and factor analysis of missing data. arXiv preprint arXiv:1910.06677 , 2019.

J. Bai and S. Ng. Matrix completion, counterfactuals, and factor analysis of missing data. Journal of the American Statistical Association , 116(536):1746-1763, 2021.

M. Bertrand and S. Mullainathan. Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination. American economic review , 94(4):991-1013, 2004.

Y. Brandman, A. Orlitsky, and J. Hennessy. A spectral lower bound technique for the size of decision trees and two-level and/or circuits. IEEE Transactions on Computers , 39(2):282-287, 1990.

E. Candes and B. Recht. Exact matrix completion via convex optimization. Communications of the ACM , 55(6):111-119, 2012.

E. J. Candes and Y. Plan. Matrix completion with noise. Proceedings of the IEEE , 98(6): 925-936, 2010.

E. J. Candès and T. Tao. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory , 56(5):2053-2080, 2010.

S. Chatterjee. Matrix estimation by universal singular value thresholding. 2015.

D. Chetverikov, Z. Liao, and V. Chernozhukov. On cross-validated lasso in high dimensions. The Annals of Statistics , 49(3):1300-1317, 2021.

T. Dasgupta, N. S. Pillai, and D. B. Rubin. Causal inference from 2 k factorial designs by using potential outcomes. Journal of the Royal Statistical Society: Series B: Statistical Methodology , pages 727-753, 2015.

M. A. Davenport and J. Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing , 10(4):608-622, 2016. doi: 10.1109/JSTSP.2016.2539100.

E. Duflo, R. Glennerster, and M. Kremer. Using randomization in development economics research: A toolkit. Handbook of development economics , 4:3895-3962, 2007.

S. Eriksson and D.-O. Rooth. Do employers use unemployment as a sorting criterion when hiring? evidence from a field experiment. American economic review , 104(3):1014-1039, 2014.

J. Fan, W. Wang, and Y. Zhong. An l ∞ eigenvector perturbation bound and its application to robust covariance estimation. Journal of Machine Learning Research , 18(207):1-42, 2018.

M. Gavish and D. L. Donoho. The optimal hard threshold for singular values is 4 / √ 3 . IEEE Transactions on Information Theory , 60(8):5040-5053, 2014.

E. George, W. G. Hunter, and J. S. Hunter. Statistics for experimenters: design, innovation, and discovery . Wiley, 2005.

G. W. Imbens and D. B. Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015.

M. Karpovsky. Finite Orthogonal Series in Design of Digital Devices . John Wiley & Sons, Inc., Hoboken, NJ, USA, 1976. ISBN 0471731889.

R. Keshavan, A. Montanari, and S. Oh. Matrix completion from noisy entries. Advances in neural information processing systems , 22, 2009.

R. H. Keshavan, A. Montanari, and S. Oh. Matrix completion from a few entries. IEEE transactions on information theory , 56(6):2980-2998, 2010.

W. Ma and G. H. Chen. Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under a low nuclear norm assumption. Advances in neural information processing systems , 32, 2019.

R. Mazumder, T. Hastie, and R. Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. The Journal of Machine Learning Research , 11:2287-2322, 2010.

E. Mossel, R. O'Donnell, and R. P. Servedio. Learning juntas. In Proceedings of the thirty-fifth annual ACM symposium on Theory of computing , pages 206-212, 2003.

S. Negahban and D. Shah. Learning sparse boolean polynomials. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 2032-2036. IEEE, 2012.

L. T. Nguyen, J. Kim, and B. Shim. Low-rank matrix completion: A contemporary survey. IEEE Access , 7:94215-94237, 2019.

R. O'Donnell. Some topics in analysis of boolean functions. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 569-578, 2008.

B. Recht. A simpler approach to matrix completion. Journal of Machine Learning Research , 12(12), 2011.

P. Rigollet and J.-C. Hütter. High dimensional statistics. Lecture notes for course 18S997 , 813 (814):46, 2015.

P.-Å. Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics , 12(1):99-111, 1972.

C. J. Wu and M. S. Hamada. Experiments: planning, analysis, and optimization . John Wiley &Sons, 2011.

A. Zhao and P. Ding. Regression-based causal inference with factorial experiments: estimands, model specifications and design-based properties. Biometrika , 109(3):799-815, 2022.

A Combinatorial Inference Applications

In this section, we discuss how classical models and applications could relate to our proposed potential outcome model. In particular, we discuss two well-studied models for functions over combinations: low-degree Boolean polynomials and k -Juntas. We also discuss applications such as factorial design experiments and recommendation systems.

Low-degree Boolean Polynomials. A special instance of sparse Boolean functions is a low-degree polynomial which we define as follows. For a positive integer d ≤ p , f : {-1 , 1 } p → R is a d -degree polynomial if its Fourier transform satisfies the following for any input x ∈{-1 , 1 } p

In this setting, we see that s ≤ ∑ d i =0 ( p i ) ≈ p d . That is, degree d -polynomials impose sparsity on the potential outcome by limiting the degree of interaction between interventions.

Applications of low-degree Boolean polynomials. We discuss two applications when the potential outcomes can be modeled as a low-degree Boolean polynomial.

Factorial Design Experiments. Factorial design experiments consist of p treatments where each treatment arm can take on a discrete set of values, and units are assigned different combinations of these treatments. In the special case that each treatment arm only takes on two possible values, this experimental design mechanism is referred to as a 2 p factorial experiment. Factorial design experiments are widely employed in the social sciences, agricultural and industrial applications [20, 18, 38]. A common strategy to determine treatment effects in this setting is to assume that the potential outcome only depends on main effects and pairwise interactions with higher-order interactions being negligible [11, 21, 24]. That is, analysts implicitly enforce sparsity of α n by setting d =2 . Werefer the reader to [39] for a detailed discussion of various estimation strategies and their validity in different settings. The model we propose (see Assumption 3.1) captures these various modeling choices commonly used in factorial design experiments by imposing sparsity on α n . Further, as we show later, Synthetic Combinations can adapt to low-degree polynomials without pre-specifying the degree, d , i.e., it automatically adapts to the inherent level of interactions in the data. The additional assumption we make however is that there is structure across units, i.e., the matrix A =[ α n ] n ∈ [ N ] is low-rank.

Optimal Team Selection. Another use for combinatorial inference is to find the optimal configuration for a team. For example, a soccer team can be interested in finding the configuration of positions (i.e., defensive, midfield, offensive players) that leads to the highest winning percentage. Here, units n are different teams, and combinations π represent possible configurations of the team. The potential outcomes E [ Y ( π ) n ] represent the winning percentage for a given team n under a particular configuration π , with the coefficients of α n representing the effect of different configurations on the win rate. In this setting, enforcing sparsity of α n via low-degree polynomials may be appropriate because the number of wins may depend on all the players, however only interactions between players of the same type matter while higher-order interactions between players of different types can be negligible. Returning to the example of a soccer team, the win rate can depend strongly on the synergy of the offensive players, whereas interactions between offensive and defensive players have a smaller effect. Further, since different teams might play in similar ways and can have the same optimal configuration, one potential way to capture this structure across teams is by placing a low-rank structure on A .

k -Juntas. Another special case of sparse Boolean functions are k -Juntas which only depend on k<p input variables. More formally, a function f : {-1 , 1 } p → R is a k -junta if there exists a set K ⊂ [ p ] with | K | = k such that the fourier expansion of f can be represented as follows

Therefore, in the setting of the k -Junta, the sparsity index s ≤ 2 k . In contrast to low-degree polynomials where the function depends on all p variables but limits the degree of interaction between variables, k -Juntas only depend on k variables but allow for arbitrary interactions amongst them.

Applications of k -Juntas. We discuss two applications when the potential outcomes can be modeled as a k -Junta.

Recommendation Systems. Recommendation platforms such as Netflix are often interested in recommending a combination of movies that maximizes a user's engagement with a platform. Here, units n can be individual users, and π represents combinations of different movies. The potential outcomes E [ Y ( π ) n ] are the engagement levels for a unit n when presented with a combination of movies π , with α n representing the preferences for that user. In this setting, a user's engagement with the platform may only depend on a small subset of movies. For example, a user who is only interested in fantasy will only remain on the platform if they are recommended movies such as Harry Potter . Under this behavioral model, the potential outcomes (i.e., engagement levels) can be modeled as a k -Junta with the non-zero coefficients of α n representing the combinations of the k movies that affect engagement level for a user n . Our potential outcome observational model (Assumption 3.1) captures this form of sparsity while also reflecting the low-rank structure commonly assumed when studying recommendation systems. Once again, our results show Synthetic Combinations can adapt to k -Juntas without prespecifying the subset of features K , i.e., it automatically learns the important features for a given user.

Knock-down Experiments in Genomics. A key task in genomics is to identify which set of genes are responsible for a phenotype (i.e., physical trait of interest such as blood pressure) in a given individual. To do so, geneticists use knock-down experiments which measure the difference in the phenotype after eliminating the effect of a set of genes in an individual. To encode this process in the language of combinatorial causal inference, we can think of units n as different individuals, an action as knocking a particular gene out, and π as a combination of genes that are knocked out. The potential outcomes E [ Y ( π ) n ] is the expression of the phenotype for a unit n when the combination of genes π are eliminated via knock-down experiments, and the coefficients of α n represent the effect of different combination of genes on the phenotype. A typical assumption in genomics is that phenotypes only depend on a small set of genes and their interactions. In this setting, we can model this form of sparsity by thinking of the potential outcome function as a k -junta, as well as capturing the similarity between the effect of genes on different individuals via our low-rank assumption.

B Proof of Theorem 4.2

Proof. Below, the symbol AX = and DX = imply that the equality follows from Assumption X and Definition X , respectively. We begin with the proof of Theorem 4.2 (a). Proof of Theorem 4.2 (a): For a donor unit u ∈I and π ∈ Π \ Π u , we have

Proof of Theorem 4.2 (b): For a donor unit n ∈ [ N ] \ I D and π ∈ Π \ Π n , we have

where the last equality follows from Theorem 4.2 (a).

C Proof of Theorem 6.6

C.1 Proof of Theorem 6.6 (a)

We have that

To finish the proof, we quote the following Theorem which we adapt to our notation.

Theorem C.1 (Theorem 2.18 in [36]) . Fix the number of samples n ≥ 2 . Assume that the linear model Y = X θ ∗ + ϵ , where X ∈ R n × d and ϵ is a sub-gaussian random variable with noise variance σ 2 . Moreover, assume that ∥ θ ∗ ∥ 0 ≤ k , and that X satisfies the incoherence condition (Assumption 6.3) with parameter k . Then, the lasso estimator ˆ θ L with regularization parameter defined by

satisfies with probability at least 1 -δ .

Further, as in established in the proof of Theorem 2.18 in [36], ∥ θ ∗ -ˆ θ L ∥ 1 ≤ √ k ∥ θ ∗ -ˆ θ L ∥ 2 . Note that the set-up of Theorem C.1 holds in our setting with the following notational changes: Y = Y Π u , X = χ (Π u ) , θ ∗ = α u , ˆ θ L = ˆ α u , k = s as well as our assumptions on the regularization parameter λ u and that ϵ π u is sub-gaussian (Assumption 6.2). Applying Theorem C.1 gives us

Substituting this bound into (8) yields the claimed result.

C.2 Proof of Theorem 6.6 (b)

For any matrix A with orthonormal columns, let P A = AA T denote the projection matrix on the subspace spanned by the columns of A . Define ˜ w n = P V (Π n ) I w n , where V (Π n ) I are the right singular vectors of E [ Y (Π n ) I ] . Let ∆ n w = ˆ w n -˜ w n ∈ R |I| , and ∆ π I = ˆ E [ Y ( π ) I ] -E [ Y ( π ) I ] ∈ R |I| . Denote ∆ E =max u ∈I ,π ∈ Π ∣ ∣ ∣ ˆ E [ Y ( π ) u ] -E [ Y ( π ) u ] ∣ ∣ ∣ . In order to proceed, we first state the following result,

LemmaC.2. Let the set-up of Theorem 6.6 hold. Then, we have

Using Lemma C.2, and the notation established above, we have

From Assumption 6.5, it follows that E [ Y ( π ) I ] = P V (Π n ) I E [ Y ( π ) I ] , where V (Π n ) I are the right singular vectors of E [ Y (Π n ) I ] . Using this in (10) gives us

Below, we bound the three terms on the right-hand-side of (11) separately. Before we bound each term, we state a Lemma that will be useful to help us establish the results.

Bounding Term 1. By Holder's inequality and Lemma C.3, we have that

This concludes the analysis for the first term.

Bounding Term 2 . By Cauchy-Schwarz and Assumption 6.1 we have,

We now state a lemma that will help us conclude our bound of Term 2. The proof is given in Appendix D.3.

LemmaC.4. Let the set-up of Theorem 6.6 hold. Then,

Incorporating the result of the lemma above into (13) gives us

Bounding Term 3. By Holder's inequality, we have that

(17)

We now state a proposition that will help us conclude our proof of Term 3. The proof is given in Appendix D.4.

Proposition C.5. Let the set-up of Theorem 6.6 hold. Then, conditioned on A , we have

As a result of Proposition C.5 and Lemma C.3, we have the following bound for Term 3,

Collecting Terms. Combining equations (12), (15), (19) gives us

By Theorem 6.6 (a), we have that

where we remind the reader that M = min u ∈I | Π u | . Substituting (21), and our assumption that M = ω ( r 2 n s 2 p ) into (20), we get

D Proofs of Helper Lemmas for Theorem 6.6.

In this section we provide proofs of Lemmas C.2, C.3, C.4, and Proposition C.5 which were required for the Proof of Theorem 6.6.

D.1 Proof of Lemma C.2

By Assumption 3.1 and 4.1 (b), we have

From Assumption 6.5, we have that E [ Y ( π ) I ] = P V (Π n ) I E [ Y ( π ) I ] . Substituting this into the equation above completes the proof.

D.2 Proof of Lemma C.3

For simplicity, denote E [ Y (Π n ) n ] = E [ Y (Π n ) ] . By definition, ˜ w n is the solution to the following optimization program

Let U (Π n ) I , Σ (Π n ) I , V (Π n ) I denote the SVD of E [ Y (Π n ) I ] . Further, let U (Π n ) ,r n I , Σ (Π n ) ,r n I , V (Π n ) ,r n I denote the rank r n truncation of the SVD. Then, define w r n = V (Π n ) ,r n I ( Σ (Π n ) ,r n I ) † ( U (Π n ) ,r n I ) T E [ Y (Π n ) ] , where † is pseudo-inverse. We first show that w r n is a solution to (23).

Next, we bound ∥ w r n ∥ 2 using Assumptions 6.1 and 6.4 as follows

Therefore, we have

D.3 Proof of Lemma C.4

First, we introduce some necessary notation required for the proof. Let ˆ E [ Y (Π n ) I ] = ˆ U (Π n ) I ˆ S (Π n ) I ˆ V (Π n ) I denote the rank r n SVD of ˆ E [ Y (Π n ) I ] . Then, to establish Lemma C.4, consider the following decomposition:

We bound each of these terms separately again.

Bounding Term 1 . We have

Theorem D.1 (Wedin's Theorem [37]) . Given A , B ∈ R m × n , let ( U , V ) , ( ˆ U , ˆ V ) denote their respective left and right singular vectors. Further, let ( U k , V k ) ∈ (respectively, ( ˆ U k , ˆ V k ) ) correspond to the truncation of ( U , V ) (respectively, ( ˆ U , ˆ V ) ), respectively, that retains the columns correspondng to the top k singular values of A (respectively, B ). Let s i represent the i -th singular values of A . Then, max( ∥P U k -P ˆ U k ∥ op , ∥P V k -P ˆ V k ∥ op ) ≤ 2 ∥ A -B ∥ op s k -s k +1

Applying Theorem D.1 gives us

where the last equality follows from the fact that rank ( E [ Y Π n I ]) = r n , hence s r n +1 = 0 . Now, plugging Assumption 6.4, (25) into (24) gives us

Substituting the result of Proposition C.5 and (26) into (24) gives us

To further simplify (27), we substitute the result of Lemma C.3

Substituting the Lemma C.3 into (27) gives us

Bounding Term 2. We introduce some necessary notation required to bound the second term. Let ˆ E [ Y (Π n ) ,r n I ] = ∑ r n l =1 ˆ s l ˆ µ l ˆ v ′ l denote the r n decomposition of ˆ E [ Y (Π n ) I ] . Let, ˆ E [ Y (Π n ) ,r n I ] = ˆ U (Π n ) I ˆ S (Π n ) I ( ˆ V (Π n ) I ) T . Further, define ϵ Π n =[ ϵ π n : π ∈ Π n ] ∈ R | Π n | . Then to begin, note that since ˆ V (Π n ) I is a isometry, we have that

To upper bound ∥ ( ˆ V (Π n ) I ) T ∆ n w ∥ 2 2 as follows, consider

Using the two equations above gives us

To bound the numerator in (29), note that by definition E [ Y (Π n ) ] = E [ Y (Π n ) I ] ˜ w n . Using this observation, we have

To proceed, we then use the following inequality: for any A ∈ R a × b ,v ∈ R b , we have

Substituting (30) into (29) and then applying inequality (31) gives us

Next, we bound ∥ ˆ E [ Y (Π n ) ,r n I ] ˆ w n -E [ Y (Π n ) ] ∥ 2 2 . To this end, note that Assumption 3.1 implies that

Next, we proceed by calling upon Property 4.1 of [4] which states that ˆ w n as given by (4) is the unique solution to the following convex program:

Using this property, we have that

Substituting (33) and (35) into (32) and using (31), we get

Then substituting this equation into (32) gives us

We state three lemmas that help us bound the equation above with their proofs given in Appendices D.3.1, D.3.2 and D.3.3 respectively.

LemmaD.2. Let the set-up of Theorem 6.6 hold. Then,

LemmaD.3. Let the set-up of Theorem 6.6 hold. Then for a universal constant C> 0 , we have

LemmaD.4. Let the set-up of Theorem 6.6 hold. Then,

Using the results of three lemmas above, applying Assumption 6.4 in (36), and then simplifying gives us

This concludes the proof for term 2. Using the result of Lemma C.3, we have that ∥ ˜ w n ∥ 1 ≤ √ r n . Substituting this into (39) gives us

Collecting Terms. Combining the results of (28) and (40), gives us

D.3.1 Proof of Lemma D.2

We first state Weyl's inequality which will be useful for us to establish the results.

Theorem D.5 (Weyl's Inequality) . Given two matrices A , B ∈ R m × n , let s i and ˆ s i denote the i -th singular values of A and B respectively. Then, for all, i ≤ min { n,m } , we have | s i -ˆ s i | ≤∥ A -B ∥ op

Using Weyl's inequality gives us

Using the inequality above and Assumption 6.4, we have

Then, substituting (21) into the equation above gives us that

holds with high probability for some universal constant C ≥ 0 . Finally, using the assumption that M = ω ( r n s 2 p ) yields the claimed result.

D.3.2 Proof of Lemma D.3

For notational simplicity, let ˆ U r n , ˆ S r n , ˆ V r n denote ˆ U (Π n ) I , ˆ S (Π n ) I , ˆ V (Π n ) I respectively. For a matrix A , let A .,j denote its j -th column. Additionally, denote ∆ j = ˆ E [ Y (Π n ) ,r n I ] .,j -E [ Y (Π n ) I ] .,j . Then,

We have that ( ˆ E [ Y (Π n ) ,r n I ] .,j -ˆ U r n ˆ U T r n E [ Y (Π n ) I ] .,j ) belongs to the subspace spanned by the column vectors of ˆ U r n , while ( ˆ U r n ˆ U T r n E [ Y (Π n ) I ] .,j -E [ Y (Π n ) I ] .,j ) belongs to its orthogonal complement. Therefore,

Therefore, we have

Bounding ∥ ∥ ∥ ˆ U r n ˆ U T r n E [ Y (Π n ) I ] .,j -E [ Y (Π n ) I ] .,j ∥ ∥ ∥ 2 2 . Note that E [ Y (Π n ) I ] .,j = U (Π n ) I ( U (Π n ) I ) T E [ Y (Π n ) I ] .,j . Using Assumption 6.1 and (26), we have

for a universal constant C> 0 . Combining the two bounds gives us,

for a universal constant C> 0 .

D.3.3 Proof of Lemma D.4

The proof of Lemma D.4 follows from adapting the notation of Lemma H.5 in [5] to that used in this paper. Specifically, we have the notational changes established in Table 2, as well as the following changes: r pre = r n , Y r pre pre, I ( d ) = ˆ E [ Y (Π n ) ,r n I ] , E [ Y pre, I ( d ) ] ,ϵ pre,n = ϵ Π n ,T 0 =Π n where the l.h.s of each equality is the notation used in [5] , and the r.h.s is the notation used in this work. Using the notation established, we can use the result of Lemma H.5 in [5] to give us

which completes the proof.

D.4 Proof of Proposition C.5

We prove Proposition C.5, which also serves as the finite-sample analysis of the vertical regression procedure (step 2 of Synthetic Combinations) done via principal component regression. The proof follows from adapting the notation of Proposition E.3 in [2] to that used in this paper. Specifically, we present table 2 that matches their notation to ours.

Using the table above, we have that the following holds with probability 1 -O ( ( |I|| Π n | ) -10 )

This completes the proof.

E Sparsity of Matrix of Fourier Coefficients

In this section, we prove that the matrix of Fourier coefficients A has at most rs non-zero columns. This result helps establish that the lower bound on the sample complexity for estimating all N × 2 p parameters is O ( Nr + r 2 s ) as discussed in 6.3.

LemmaE.1. The number of nonzero columns of A , which we have denoted p ′ , satisfies p ′ ≤ rs .

Proof. Let A t denote the sub-matrix of A formed by taking the first t rows. Further, let p ′ t denote the number of nonzero columns of A t . We proceed by induction, that is we show for all t ,

In the case of t =1 , either the first row is the zero vector, in which case p ′ 1 =rank( A 1 )=0 , or else rank( A 1 )=1 and p ′ 1 ≤ s since the first row is at most s -sparse. Then, for general t , note that

If rank( A t ) = rank( A t -1 ) then we must also have p ′ t -1 = p ′ t and the inductive hypothesis holds. Otherwise, rank( A t ) = rank( A t -1 )+1 . Note that the t th row of A has only s nonzero entries, so p ′ t ≤ p ′ t -1 + s . In this case, we have that

where the last inequality holds due to the inductive hypothesis. Since rank( A ) = r , we have that p ′ ≤ rs . This completes the proof.

Peer reviews:
comments: SUMMARY:
The manuscript introduces a synthetic combinations estimator, which computes causal effects for unseen treatment combinations under some reasonable assumptions on the data generating process. Building on theory from potential outcomes and synthetic interventions, the authors show how to exploit information sharing across units and treatments to identify causal effects in this challenging setting. The proposed two-step algorithm is more efficient and flexible than existing alternatives.

STRENGTHS:
The topic is timely and interesting. The manuscript is exceptionally clear and well-written, which is greatly appreciated when dealing with dense formalisms. The theoretical results are strong and convincing, rooted in established results while simultaneously going beyond the current state of the art. The analysis is sound and well-motivated.

WEAKNESSES:
My main critique of this manuscript is one I am sure the authors will have anticipated – there are no empirical results in the main text! I am aware that 9 pages is tight but the authors could have been more judicious in their selection of what to send to the appendix. I also found the material on CART to be super interesting; a shame to banish this material to the wilderness of Appendices J and K. Fortunately, the final manuscript affords one extra page. I strongly encourage the authors to move Fig. 2 to the main text and expand on this empirical evaluation. Would also be great to shoehorn in some of the CART results, but that may prove difficult. If it is impossible to squeeze everything within the limit, then the authors may want to consider repurposing this manuscript for a journal submission. There is more than enough material here for a very solid journal contribution.

QUESTIONS:
See above.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of estimating potential outcomes in the presence of a combinatorial number of intervention choices. Under some assumptions, they propose a two-phased algorithm \"Synthetic Combinations\": first exploit structure across combinations of interventions (via \"horizontal regression\") and then exploit structure across units (via \"vertical regression\"). Experiments are given in the appendix.\n\nSTRENGTHS:\nThe proposed algorithm is clean and intuitive. It also seems to scale nicely with the number of intervention combinations in experiments.\n\nWhile the theoretical guarantees rely heavily on a bunch of assumptions, Section 7 proposes an experimental design framework which ensures that an important set of assumptions (existence of donor units) will be met with high probability. In fact, I strongly propose that the authors rephrase their paper to highlight this; otherwise it is hard to believe that their work will useable as it is highly unlikely that all the required assumptions are met in practice without having control in assigning interventions to the units.\n\nWEAKNESSES:\nI did not check all the proofs in detail, but I do not see any glaring weaknesses.\n\nThere are a lot of assumptions and it is highly unlikely that all the required assumptions are met in practice (Section 7 helps to mitigate some of these concerns).\n\nI am skeptical about the low-rank assumption on the matrix of Fourier coefficients $A$. While it is true low-rank assumptions are common in prior matrix completion settings, they usually directly consider the matrix at hand and not transform it into the Fourier space first. For example, Lines 655-657 in the appendix writes \"This missingness pattern where outcomes with larger absolute values are observed is common in applications such as recommendation engines, where we are only likely to observe ratings for combinations that users either strongly like or dislike\". The corresponding missingness pattern in the problem studied here is on the $N$-by-$2^p$ matrix. It is unclear to me why it should be believable that the transformed space is low-rank. The authors ought to justify this, ideally with practical examples/settings, or risk diminishing the impact of their contributions.\n\nQUESTIONS:\nLine 83:\nBy \"equivalent\", do you mean that they proved equivalence between the two problems via reductions, or do you mean \"equivalent\" in a colloquial sense of the word?\n\nAssumption 3.1:\nAs discussed in the weaknesses, it is unclear to me why this model is interesting or justified. Of course, this work can be appreciated under the restriction of this assumption, but it will greatly weaken the contributions. I am more than happy to increase my \"contribution\" score if the authors provide sufficient justification for the low-rank assumption.\n\nType on Line 183:\ndouble \"exists\"\n\nMotivating example on Line 190:\nI don't understand why this motivates the existence of donor units when the paper has thus far repeatedly claim to allow unobserved confounding. If we allow interventions to be arbitrarily assigned to units, it is unclear why we should believe that donor units exist. The \"correct\" way to justify should be to say that there is an experimental design that ensures the existence of donor units, and then refer to Section 7.\n\nDetermining donor set on Line 246:\nThis feels very ad-hoc. As it is unlikely that donor units will exist if we allow arbitrary experiments, I feel that this paragraph could be removed once the authors reorder their paper to place more emphasis on the experimental design proposed in Section 7.\n\nSubsection on Additional Assumptions:\nI feel that \"so-and-so also has such an assumption\" is not sufficient discussion of assumptions. Firstly, \"so-and-so\" may have the assumptions under different contexts (e.g. see my complaint about low-rank assumption in the Weaknesses section) so it is unclear why such assumption is justified in the setting studied in this paper. Secondly, the discussion should explain \"what goes wrong\" if one particular assumption is violated, or why we should expect any particular assumption to hold in practice. As mentioned several times by now, one \"partial fix\" is to emphasize that experimental design of Section 7 guarantees some assumptions with high probability. That is, \"Synthetic Control\" should be used in conjunction with the experimental design proposed in Section 7.\n\nLIMITATIONS:\nNil.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal paper studies the problem of recovering $N\\times 2^p$ unit-specific outcomes for N heterogeneous units and any combination of p possible interventions from small number of experiments and observations. \nPrior to this work, the problem has been studied under assumptions of latent similarity or regularity in how combinations of interventions interact, as well as some other setups. In the setup when one assumes latent similarity, namely that the matrix of Fourier coefficients across units has rank <= r, the problem is reduced to matrix completion, and hence all causal outcomes can be recovered from $O(poly(r)\\times (N+2^p))$ observations. In the case, when regularity in intervention interactions is assumed it is known that $O(Ns^2p)$ measurement are sufficient, where s is the sparcity parameter of coefficients in the Fourier expansion of the potential outcomes.\nThis paper studies the problem in the case when both latent similarity and intervention regularity is assumed. Under both assumptions the paper proposes an algorithm that recovers all $N\\times 2^p$ causal outcomes from $O(N\\times (s^2 + p))$ measurements.\n\nSTRENGTHS:\nThe problem of estimating causal outcomes under a combination of interventions is a notoriously hard problem with various applications. One complication is that it is usually expensive\\impossible to run many experiments to measure the effects caused by interventions. Hence, understanding how to set up experiment that will require the minimum amount of measurements is of great importance.\nThis paper proposes an algorithm, called Synthetic Combinatorics, that provably recovers all  $N\\times 2^p$ unit-specific outcomes under $2^p$ interventions under a combination of two widely accepted assumptions from  $O(N\\times (s^2 + p))$, which is a significant improvement over the prior work.\n\nThe paper also provides statistical estimates for the number of samples needed for every experiment to achieve desired accuracy of the recovery.\n\nThe paper is well-written and as far as I can judge is correct, though I did not read the proofs carefully.\n\nWEAKNESSES:\nIt is not completely clear how realistic is the scenario when latent similarity and intervention regularity holds simultaneously. I can imagine that in some datasets one or the other may hold, while both assumptions at the same time may not hold. The paper will benefit significantly from experiments on real-world datasets, that can confirm that theoretical assumptions are realistic and we indeed see improvement in the number of measurements needed.\n\nQUESTIONS:\nCan you provide some intuition why you believe that the assumptions needed for Synthetic Cobminatorics to work are expect to hold for real-world datasets?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
__Disclaimer__: This is my first time reading & reviewing a paper from the field of Combinatorial Interventions. My expertise is in NLP.

This work studies latent structure across units and combinations of interventions, assuming similar outcomes across units and regular interaction. An estimation procedure, Synthetic Combinations, is proposed, establishing finite-sample consistency under precise conditions. This work also uses methods to reduce errors in variables and provides a possibility of model-agnostic analysis.

STRENGTHS:
* All the proofs and other mathematical explanations are clear, but I'm not able to understand them properly because of no expertise in that field.

WEAKNESSES:
*

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-27