Title: Optimized Covariance Design for AB Test on Social Network under Interference
Venue: NeurIPS 2023
Accepted: yes
Keywords: AB test

interference

causal inference

optimization

social network
Abstract: Online A/B tests have become increasingly popular and important for social platforms. However, accurately estimating the global average treatment effect (GATE) has proven to be challenging due to network interference, which violates the Stable Unit Treatment Value Assumption (SUTVA) and poses great challenge to experimental design. Existing network experimental design research was mostly based on the unbiased Horvitz-Thompson (HT) estimator with substantial data trimming to ensure unbiasedness at the price of high resultant estimation variance. In this paper, we strive to balance the bias and variance in designing randomized network experiments.  Under a potential outcome model with 1-hop interference, we derive the bias and variance of the standard HT estimator and reveal their relation to the network topological structure and the covariance of the treatment assignment vector. We then propose to formulate the experimental design problem as to optimize the covariance matrix of the treatment assignment vector to achieve the bias and variance balance by minimizing the mean squared error (MSE) of the estimator. An efficient projected gradient descent algorithm is presented to the implement of the desired randomization scheme. Finally, we carry out extensive  simulation studies to demonstrate the advantages of our proposed method over other existing methods in many settings, with different levels of model misspecification.

Optimized Covariance Design for AB Test on Social Network under Interference

Abstract

Online A/B tests have become increasingly popular and important for social platforms. However, accurately estimating the global average treatment effect (GATE) has proven to be challenging due to network interference, which violates the Stable Unit Treatment Value Assumption (SUTVA) and poses great challenge to experimental design. Existing network experimental design research was mostly based on the unbiased Horvitz-Thompson (HT) estimator with substantial data trimming to ensure unbiasedness at the price of high resultant estimation variance. In this paper, we strive to balance the bias and variance in designing randomized network experiments. Under a potential outcome model with 1-hop interference, we derive the bias and variance of the standard HT estimator and reveal their relation to the network topological structure and the covariance of the treatment assignment vector. We then propose to formulate the experimental design problem as to optimize the covariance matrix of the treatment assignment vector to achieve the bias and variance balance by minimizing a well-crafted upper bound of the mean squared error (MSE) of the estimator, which allows us to decouple the unknown interference effect component and the experimental design component. An efficient projected gradient descent algorithm is presented to the implement of the desired randomization scheme. Finally, we carry out extensive simulation studies 2 to demonstrate the advantages of our proposed method over other existing methods in many settings, with different levels of model misspecification.

1 Introduction

A/B test, also known as controlled experiment, is an effective way to evaluate treatment effect of new product features and facilitate data-driven decision-making [14]. Due to its simplicity and efficiency, it has become the gold standard in the industry. The theory of A/B test is based on the Stable Unit Treatment Value Assumption (SUTVA), under Neyman-Rubin's framework [33], which assumes unit's treatment effect isn't influenced by other units' treatment assignments. However, in situations where social network connects units, SUTVA is often violated since an individual's behavior can be influenced by their friends. This violation of SUTVA is referred as network interference or social interference, which brings challenges to both estimation and inference of parameters of interest. It has been shown that ignoring network effect can induce much higher bias or variance [29].

To tackle the bias and variance induced by network interference, researchers have tried to design randomization schemes in pre-treatment phase and estimators or adjustment methods in post-treatment phase. In this paper, we will focus on designing randomization scheme, namely, experiment design.

Specifically, we take the graph structure as known and leverage it to assist experiment design. Our basic scheme is cluster-level randomization, which treats cluster or community in the graph as unit of treatment and all units in the same cluster share the same treatment. [19] is an early work investigating this approach, and [35] proposes to form clusters utilizing graph. The graph cluster randomization is conceptually simple but effective in many scenes, and thus becomes a prevalent paradigm for network experiment design. Moreover, we consider experimental design on a social network in this work. Social network typically has organic structure and high clustering coefficient, which corresponds to meaningful cluster structure and also motivates us to adopt cluster-level randomization. We assume the cluster structure of the network is given. In practice, interpretable community detection algorithms such as the Louvain algorithm [4] are typically applied for cluster construction. Moreover, we stress the scalability of our method since social network is usually large-scale in reality.

Related works have shown that there exists an explicit bias-variance tradeoff [23, 41]. However, many HT-style estimators that utilize exposure indicator to ensure unbiasedness are designed on the basis of the neighborhood interference assumption that restricts global interference to 1-hop neighborhood [10, 25, 36, 39, 9], and graph-dependent generalized propensity score [1]. Nonetheless, we argue that pursuing unbiasedness by simple sample trimming leads to small effective sample size with high variance, and thus is far from optimal. Moreover, computational cost is also high in this approach since a large number of Monte Carlo simulations are usually needed for computing the generalized propensity score, or the trimming probability for constructing sample weights. Having observed the preceding drawbacks of the unbiased estimator, we consider the standard unit-level HT estimator to achieve possible better bias and variance tradeoff with less computation burden at the meantime. Based on a potential outcome model, we characterize the relationship among the bias, variance, the network topology, and the treatment assignment mechanism. This development allows us to design experimental schemes that minimizes MSE and achieve better bias-variance tradeoff.

Besides from the time-consuming Monte Carlo simulation for calculating generalized propensity score mentioned above, the computational efficiency of experiment design is indeed a limitation of current methods. For instance, the two-wave experiment [37], entails solving a mixed-integer quadratic programming problem, which can be computationally expensive. The adaptive covariates-balancing design [25] adopts a sequential process for treatment assignment and thus isn't very scalable. To address this limitation, we propose to utilize gradient-based optimization algorithm.

In this paper, we propose to treat the covariance matrix of treatment vector as decision variable in optimization, which is nontrivial as a result of interference and corresponding complex variance of estimator. Similar approach in optimization step is proposed in [17], where interference isn't considered, and making covariance covariance acting as decision variable is a natural choice owing to the simple structure of variance there. We draw attention to the fact that related works concerning experiment design mostly concentrates on a predetermined class of randomization schemes and seeks to optimize the parameters to obtain designs that minimize variance or MSE of estimator [37, 23, 5, 42]. However, we argue that these designs can all be understood as different patterns of the covariance matrix of the treatment vector, and we propose to treat the covariance matrix as decision variables rather than limiting our scope to a particular class of designs. One of major challenges with this approach is generating a randomization scheme that supports legitimate sampling, as there may not be a joint distribution that corresponds to the optimized covariance matrix in our formulation. To address this issue, we propose to adopt projected gradient descent to guarantee the validity of optimized covariance matrix. Furthermore, we also compare our method with other popular designs through extensive simulation on a real online social network. Various interference intensity levels and potential outcome models are considered. It is found that our method is robust to model misspecification and outperforms in almost all settings, especially when the interference isn't too weak compared to the direct treatment effect, which is the scene we target.

Contributions. Our contributions are summarized as follows.

We derive a well-crafted upper bound for the MSE of the HT estimator that decouples the estimation of causal mechanism and experimental design. This enables us to optimize the experimental design by minimizing this bound, in which the covariance matrix of treatment vector acts as decision variables.

We propose the formulation of covariance optimization problem with reparameterization of covariance matrix and constraints that guarantees legitimate sampling subject to optimized

covariance matrix, and a projected gradient descent algorithm is proposed for solving the optimization problem.

We conduct systematic simulation on a social network and compare our method with several methods proposed recently under a range of settings, providing credible reference for the effectiveness of our method.

Related Works. A variety of interference has been considered in related works[12], such as spatial [23], temporal [13] and network, which is the scene we target. Besides from the general network, there is also another line of work focused on a two-sided marketplace or platform, where the graph is bipartite [43, 24, 16, 30, 27].

In the field of network experiment design, most of extant research assumes that the graph structure is known and leverage it to assist in modeling the interference[2], though several graph agnostic methods are developed these days [40, 8]. Nevertheless, the interference is conducted through edges of network, and agnostic methods aren't effective enough.

Beyond unit-level randomization, cluster-level randomization on graph has attracted extensive attention, including both experimental design [23, 25, 5, 30] and estimation [26, 1, 6, 15]. The seminal paper of this line of work is [35], which proposes graph cluster randomization that treats clusters as units in randomization. This methodology is motivated by providing a better lower bound of exposure probability in HT estimator that uses exposure probability as inverse weight, aiming at controlling the variance of such an unbiased estimator. Along the same direction, [36] proposes a polynomial lower bound on exposure probability through randomized local clustering algorithm.

Irregular global interference may render the experimental design problem intractable [39], thus structural assumptions are usually introduced to rule out such arbitrary interference. The first category is partial interference [34, 19, 3, 11, 5], which assumes disjoint community structures and interference only happens within each community. The second category is neighborhood interference, which assumes the interference solely originates from units' 1-hop neighborhood [25, 36, 41, 12, 8]. This kind of assumption is more practical compared with partial interference, and thus widely applied. The third category is potential outcome model, which directly assumes the expression of potential outcome, e.g. classical Cliff-Ord auto-regressive model [7], linear potential outcome models[25, 39]. Actually, some potential outcome model implies neighborhood interference assumption in its mathematical expression. It's also worth to be noted that there also appear works trying to weaken the structural assumptions mentioned above and consider the interference of more general form [6, 22, 23, 18, 15].

2 Basic Framework

2.1 Setup

We consider a finite population of n units, and the treatment vector is represented as z = ( z 1 , z 2 , . . . , z n ) ∈ { 0 , 1 } n . In this paper, we consider the experiment design problem in which a sensible random treatment assignment mechanism of all units is designed to achieve good performance of the resultant treatment effect estimation. We follow the Neyman-Rubin's framework [33]. Without SUTVA assumption, we define the potential outcome of unit i as Y i = Y i ( z ) , which means the potential outcome of unit i can depend on the treatment assignment of all units.

The parameter of interest is the global average treatment effect (GATE), which is denoted by τ ,

Here we use [ n ] to denote set { 1 , 2 , ..., n } , and 1 ( 0 ) is the n -dimensional vector of 1 ( 0 ), which corresponds to global treatment and global control.

We denote the social network as G = ( V , E ) . The node set V is [ n ] , corresponding to all units considered above. Edge set E represents the social connection between units. Furthermore, we use d i to denote the degree of node i . We assume the graph structure G is known in advance.

In this paper, we consider graph cluster randomization [35]. Here we also assume that cluster structure is given, which is generated from certain community detection algorithm. We define S k as the k -th

cluster, where k ∈ [ K ] . Clusters compose a splitting of node set, namely,

The graph cluster randomization implies that units in the same cluster share the same treatment assignment, namely,

Since we consider randomization scheme at cluster level, it's more convenient to define cluster-level treatment vector t = ( t 1 , t 2 , ..., t K ) ∈ { 0 , 1 } K . Specifically, we consider the balanced cluster-level randomization scheme satisfying

2.2 Potential Outcome Model and Estimators

One of the key ideas of this paper is to deduce an appropriate objective function to optimize experiment design schemes under consideration. In this part, we will consider standard HT estimator, namely,

Then we introduce the considered potential outcome model.

Assumption 1 (Potential Outcome Model) The potential outcome of unit i is generated by

where N i is the 1-hop neighborhood of unit i .

In this potential outcome model, β i can be interpreted as direct treatment effect of unit i , and γ measures the interference effect, characterizing the extent to which each unit is affected by its neighborhood in the network. This model is a simplified (in interference part) version of overparameterized model in [39], in which heterogeneous interference effects are accommodated. We make a somewhat stronger assumption to simplify the structure of the bias, variance and MSE. Under this model, we can express the GATE in terms of the potential outcome model parameters,

We will then derive the bias and variance of ˆ τ under this potential outcome model. We point out that the number of unknown parameters in this model is still greater than the number of known potential outcomes, which can be understood as overparameterization, and we only utilize it to derive an optimizable MSE upper bound that's not concerned with estimation of β i and γ in this model. In fact, when the interference effect and the direct treatment effect are comparable at cluster-level, a multiplier of the MSE upper bound can be factored out, which is crucial in our methodology and will be shown in section 3.1.

2.3 Bias of HT Estimator

Based on the potential outcome model, we can derive the bias and variance of our estimator explicitly. We firstly introduce cluster-level treatment assignment, t = ( t 1 , t 2 , ..., t K ) , and matrix C summarizing edges between(within) clusters with its elements defined as

We have following proposition about bias.

Proposition 1 (Bias of HT Estimator) The bias of standard HT estimator is

where Cov[ t ] refers to the covariance matrix of treatment vector t .

Specifically, we can delineate the source of bias in our setting. If we consider graph cluster randomization as basic scheme, edges within cluster don't contribute any bias, while every edge between clusters contributes to bias according to the covariance of cluster-level treatment assignments. This enlightens us that independent treatment assignments is far from optimal, and we can reduce bias by assigning treatments with appropriate positive correlations that should be higher for more densely connected cluster pair. Furthermore, we point out that bias may be of no account when a very sparse network is considered, but it indeed counts in our context of social network.

We also note that the treatment vector is summarized by its covariance matrix in the expression of bias. Hence, we denote the bias by B (Cov[ t ]) .

2.4 Variance of HT Estimator

To derive the variance, we plug in the potential outcome model and express the estimator as

An important difference in the derivation of variance contrasting that of bias is all base levels α i is no longer negligible. Hence, we consider to target at the scenes that we can acquire knowledge of all base levels in advance.

Assumption 2 (Known Base Levels) The base level α i of unit i is known in advance for all units i .

This assumption is strong in the classic context of experiment design, while we propose to target the scene where AB tests are launched by social platforms. On the one hand, the improvement in user activity indices brought by new traits or interventions, specifically, direct treatment effect and interference, is typically much smaller than the base levels α i , which means the experiment design may be quite challenging when agnostic to base levels. On the other hand, it's very reasonable to assume that we know all α i s before conducting experiments since those indices we're concerned with are recorded and/or estimated daily or weekly by social platforms.

Utilizing this assumption, we can first adjust our estimator as

This adjustment has also been recommended by [39]. Notice that this adjustment helps us remove the influence of base level α i when we consider variance. In the following, we'll adhere to this format and proceed to derive the variance term, which is summarized in the following proposition.

Proposition 2 (Variance of HT Estimator) The variance of standard HT estimator is

where h is the vector ( ∑ i ∈ S k β i -γd i ) K k =1 .

Notice that the expression of variance contains second-order term with regard to t , namely, Cov[ t ] , as well as third and fourth-order terms involving t T Ct , a quadratic form of t . It's feasible to analyze the variance precisely with a specific randomization scheme, such as the independent block randomization [5], but it is generally not straightforward to optimize a broad set of randomization schemes in its exact form. In the following subsection, we will derive a bound of variance to transform third and fourth-order terms into second-order, which also motivates our optimized covariance design.

We comment that a bias-variance tradeoff is unveiled in the previous two propositions. Actually, C is a matrix with non-negative elements, and introducing positive covariance between treatment assignments of different clusters will reduce the bias. On the other hand, introducing positive covariance means an increment in variance. In addition, though this paper doesn't involve upstream clustering algorithm, the resolution of clustering can also impact the relative importance of bias and variance, namely, low resolution usually corresponds to a higher proportion of edges within cluster, which weakens the importance of bias and stress on variance.

3 Optimized Covariance Design

3.1 Constructing Objective for Optimization

Although we have derived the expression of bias and variance, the expression of variance is complicated due to the existence of treatment effects β i s, the interference effect γ , and the higher-order terms involving the treatment vector. We will bypass the first hurdle by adding reasonable assumption about the direct treatment effect and interference effect. The second challenge will be addressed by a upper bound argument.

We first consider the former issue, which is the relationship between β i and γ i . Since we consider the interference, we propose to restrict our scope to the situation that network interference isn't too weak, which is formulated as following assumption.

Assumption 3 (Comparability between Direct Treatment Effect and Interference) Given potential outcome model in equation (6), we assume there exists a constant ω > 0 such that

holds for each cluster k ∈ [ K ] .

This assumption actually demands the magnitude of interference is comparable to direct effect at cluster level. Namely, we allow a significant difference between direct treatment effect β i and interference γd i for certain units, but there isn't too much difference after aggregation in cluster. The constant ω can be viewed as degree of tolerance on incomparability between direct causal effect and interference and depends on the both nature of potential outcome and graph structure. It's natural that people may be more susceptible to the states of their friends in some social metrics. Since we'll utilize the MSE bound as optimization objective, this constant can also be viewed as tuning parameter that's determined by domain knowledge or pilot experiment. To demonstrate the robustness of our method, we don't tune ω according to the result and fix ω = 1 in the simulation, which corresponds to a moderate constraint on comparability. Empirically, our result isn't sensitive to the value of ω .

We then discuss the latter issue. We bound all high-order terms by second or lower order terms. This effort leads to an upper bound which allows us to simply optimize the covariance matrix.

Proposition 3 (Variance Bound) The variance of the standard HT estimator has following upper bound,

where d is the vector ( ∑ i ∈ S k d i ) K k =1 .

In brief, we choose to bound the third-order term by the sum of second and fourth-order terms, and bound the fourth-order term by the definition of binary treatment vector. The format is adjusted for our optimization formulation later, where we choose the covariance matrix as decision variables.

Actually, we can find that experiment design is summarized by the covariance matrix in this upper bound. Hence, we denote this upper bound by ¯ V ω (Cov[ t ]) . Notice that there is a common multiplier γ 2 , which is associated with the unknown interference effect, in the squared bias and variance. This appealing feature allows us to optimize the MSE upper bound without estimating the parameter of the potential outcome model.

Combining proposition 1 and 3, we derive following upper bound for MSE.

Theorem 1 The MSE of the standard HT estimator has following upper bound

3.2 Optimizing Covariance

Now we can discuss our experiment design. For a class of parameterized randomization schemes, one can always analyze the covariance and express the bias and variance with parameters in randomization schemes. Based on the derivation in previous section, we argue that these randomization schemes is actually specifying the patterns of covariance matrix, e.g., the block diagonal pattern. We thus choose to directly optimize the covariance matrix of randomization scheme, which is actually considering a much larger hypothesis space.

Since our decision variable is a positive semi-definite matrix, it's intuitive to make attempts for formulating the problem as positive semi-definite programming with the MSE bound acting as objective. However, notice that there is still one square term with regard to trace( C X ) , which appears in the B ( X ) 2 . If we want to transform it into semi-definite programming (SDP), we must modify the objective further. Moreover, there are some intrinsic disadvantages of this formulation. Primarily, after we solve the optimal covariance matrix, we can't directly sample from the multivariate Bernoulli distribution that is subject to this covariance matrix. Secondly, method based on mathematical programming is usually precise but not efficient enough.

To guarantee that we can sample treatment vector subject to the optimized covariance, we first present a lemma that is equivalent to the Grothendieck's identity [28].

Lemma 1 Suppose that ( X,Y ) follows a bivariate Gaussian distribution with correlation r , namely,

This lemma enables us to sample from bivariate Bernoulli distribution with mean ( 1 2 , 1 2 ) and any valid covariance. Specifically, once we solve out the optimal covariance matrix X ∗ , we can sample from multivariate Gaussian distribution with sin(2 πX ∗ ) acting as covariance matrix, here sin function is element-wise. Nevertheless, it must be noted that this transformation isn't omnipotent for n > 2 case, namely, arcsin( X ) / 2 π can't recover all possible covariance matrix of n -variable multivariate Bernoulli distribution when X ⪰ 0 , since otherwise the multivariate Bernoulli distribution can be uniquely determined by its mean vector and covariance matrix, which is apparently false.

Moreover, we can't guarantee sin(2 πX ∗ ) to be positive semi-definite even if X ∗ is positive semidefinite. Motivated by this, we adopt the Cholesky-based parameterization. We set our decision variable as matrix R , and we guarantee the validity of the covariance matrix of Gaussian distribution first, namely, we set the covariance matrix of Gaussian distribution as RR T and the covariance matrix of (Bernoulli) treatment vector is

Now we can propose our final formulation

̸

This optimization problem is no longer convex with the introduction of arcsin function, for which we design a projected gradient descent algorithm. Since it's a non-convex optimization problem, adaptive optimizer, such as Adam [20] can be applied to improve the performance of the gradient descent step.

we have

Algorithm 1 Optimizing Covariance Matrix

Input:

G , { S k } K k =1 , N , ω

1: Initialization: R 0 , i ← 0

2: repeat

3: Calculate objective M ( R )

4: R ← Adam ( R, ∇ R M ( R ))

5: Update R with row normalization (2-norm)

6: i ← i +1

7: until i > N

Output:

Optimized matrix R ∗

We would like to draw attention to the fact that row normalization can be viewed as a projection operator to the domain in the aforementioned formulation, with the Frobenius norm of the matrix. This fact can be verified through the Cauchy-Schwarz inequality. Additionally, we actually optimize a total of K × K parameters using gradient information, which is both efficient and easy to implement.

After the optimization problem is solved, we can sample from it directly through a reparameterization of the multivariate Gaussian distribution,

4 Simulation Study

4.1 Basic Setting

In this part, we carry out simulation based on a linear/multiplicative potential outcome model on a social network FB-Stanford3[32]. 3 . We consider following linear potential outcome model,

and multiplicative model, which is a simplified version of that in [36], with removing a covariate.

We choose to fix all parameters except for interference density, γ . Namely, we set ( α, β, c, σ ) = (1 , 1 , 0 . 5 , 0 . 1) for both models, and set γ ∈ { 0 . 5 , 1 , 2 } to construct three regimes. ϵ i ∼ N (0 , 1) .

It should be noted that these two models correspond to different model misspecification forms, aiming to demonstrate the robustness of our method.

Besides our optimized covariance design (OCD), we implement following randomization schemes. We first consider two baselines, independent Bernoulli randomization (Ber) and complete randomization (CR), both of which are cluster-level. We also implement two adaptive schemes, rerandomizedadaptive randomization (ReAR) and pairwise-sequential randomization (PSR) [25, 31], which balance heuristic covariates adaptively and act as competitive baselines, since the average degree is considered as a covariate in these two methods and exactly appear in both two of our models, explicitly. Then we implement the independent block randomization (IBR) [5], which represents methods that optimize parameterized randomization schemes. Nevertheless, we argue that an appropriate objective for optimization is crucial, and we propose a randomization scheme based on IBR with optimizing variance in equation (12) as objective. We nominate it as IBR-p since the heuristic solution of this scheme in our optimization problem is pairing cluster with cluster size, namely, each block is of size 2. Notice that all randomization schemes satisfy equation (4).

For every randomization scheme we consider, we perform Monte Carlo simulation that repeats randomization schemes 10,000 times and calculates the sample mean and variance of estimators.

Moreover, the oracular GATE are utilized to calculate bias.

At last, we set the clusters as given by Louvain algorithm with fixed random seed and resolution parameter as 10, which gives K = 192 clusters. We also consider other two resolution levels, namely, 2 and 5 , corresponding to K = 30 , 95 , respectively. The outcome under these two levels is presented in the appendix.

4.2 Analysis of Results

From table 1 to table 2, we show the bias, standard deviation, and MSE of standard HT estimator. We also present the result under different estimator (such as difference-in-means estimator), different clustering resolution and different dataset (network topology) in the appendix due to limited space. Overall, it meets our expectation that the proposed method OCD demonstrates greater advantages over other methods when interference intensity, namely, γ , increases. We emphasize the outstanding performance of our method in the multiplicative model and give the credit to handling the biasvariance tradeoff better. We also discover that our approach OCD isn't sensitive to the resolution level of upstream clustering, which isn't often the case for other methods, according to the thorough simulation results presented in the appendix. In reality, when there are certain densely connected cluster pairings that are assumed to have merged but have not, the clustering outcome is poor and resolution level may be inappropriate. Such pair would typically receive a high positive covariance from the optimization process in OCD, which could be regarded as adaptively fixing the fault.

In addition, we highlight that the bias usually dominates in MSE in many settings, and it almost scales linearly with interference intensity, which is exactly as shown in the expression of bias. According to our theoretical derivation, we attribute the dominance of bias to the density of social network. There is a trend that all other methods perform similarly with two baselines, since bias is dominating in a denser network, while this component is usually ignored in the experiment design, on account of the introduction of HT estimator with exposure indicator. Moreover, although we fix ω = 1 in presented result, we actually observe that our method is very robust to the choice of ω in related experiment.

Finally, we point out that the distribution of proportion of treated units of optimized randomization scheme is usually bimodal (e.g. peak at 45%, 55%) in our randomization scheme, with the mean still 50%. This bimodal pattern is the outcome of introducing non-zero covariance on almost all off-diagonal positions of covariance matrix. This is different from other methods we considered here, for which the proportion is usually strictly 50% or unimodal and concentrates at its mean, 50%.

5 Discussion

In this article, we derive the bias and variance of the standard HT estimator under a parametric potential outcome model and construct an optimizable objective based on the derived MSE. We then propose an appropriate formulation to transform the experiment design into an optimization problem with the covariance matrix of treatment vector acting as decision variables. Besides, we propose an algorithm based on projected gradient descent to provide optimized covariance design that supports legitimate sampling. Finally, we implement a series of methods and compare their performance in diverse settings, where the advantage of our methods is demonstrated.

We stress the significance of bias and propose our method to make attempts for balancing bias and variance better. Actually, almost all potential outcome models used in the simulation part of related works admit neighborhood interference assumption, which makes the estimator using exposure indicator unbiased or nearly unbiased. However, we argue that this assumption is reasonable but also restricts our scope. There have recently been some works considering interference beyond this assumption, where bias is also demonstrated to be important.

Although there has been a lot of variance bound in this area, few of them can guide experiment design directly, and we have actually made many attempts on finding appropriate variance bound. For example, the gradient of fourth-order term E [ ( t T C t ) 2 ] can be estimated directly through REINFORCE estimator [38] without need of upper bound. Nevertheless, the optimized covariance matrix tends to give a proportion of densely linked cluster pairs high correlation (near to 1), causing the covariance matrix to become nearly singular. This, in turn, leads to catastrophic gradient variance that can't be reduced by conventional technique such as [21]. In addition, weaker assumption on comparability between direct effect and interference is feasible. We can only demand there exists ω ′ > 0 such that ∥ h ∥ 2 ≤ ω ′ γ ∥ d ∥ 2 . This weaker assumption corresponds to spectral norm bound on the second order term h T E [ tt T ] h , which is a looser bound and finally produces a empirically worse solution. This bound is also utilized in [17] and is expressed as the worst-case variance, which is concerned with maximizing a convex objective over bounded potential outcomes. The optimal value is taken when potential outcome is taken as the endpoint value of its domain, which is too conservative to have good empirical performance. The simulation result of IBR can support this claim, since similar worst-case variance is utilized in IBR.

̸

Additionally, our work can be improved and extended in several aspects. Firstly, the optimization objective can be improved with a tighter but still optimizable upper bound on variance. Moreover, the situation of non-balanced treatment assignment, i.e., E [ z i ] = 1 2 , is worthy of exploration. Besides, the derivation for more complex estimator is kept explorable, e.g., the difference-in-means estimator, and HT estimator with exposure indicator. Our derivation is performed on the standard HT estimator, and structural assumption is introduced when the derivation encounters essential difficulties. In addition, the basic setting beyond neighborhood interference assumption also warrant further extension.

Acknowledgments and Disclosure of Funding

The research was supported by the National Natural Science Foundation of China (No.72171131, 72133002); the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grants 2020AAA0108400 and 2020AAA0108403. We would like to thank Ruizhong Qiu for beneficial discussion, and anonymous reviewers for the helpful feedback.

References

Peter M Aronow and Cyrus Samii. Estimating average causal effects under general interference, with application to a social network experiment. Annals of Applied Statistics , 11(4):1912-1947, 2017.

Guillaume W Basse and Edoardo M Airoldi. Model-assisted design of experiments in the presence of network-correlated outcomes. Biometrika , 105(4):849-858, 2018.

Rohit Bhattacharya, Daniel Malinsky, and Ilya Shpitser. Causal inference under interference and network uncertainty. In Uncertainty in Artificial Intelligence , pages 1028-1038. PMLR, 2020.

Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008.

Ozan Candogan, Chen Chen, and Rad Niazadeh. Correlated cluster-based randomized experiments: Robust variance minimization. Management Science , 2023.

Alex Chin. Regression adjustments for estimating the global treatment effect in experiments with interference. Journal of Causal Inference , 7(2), 2019.

Andrew David Cliff and J Keith Ord. Spatial processes: models & applications . Taylor & Francis, 1981.

Mayleen Cortez, Matthew Eichhorn, and Christina Yu. Staggered rollout designs enable causal inference under interference without network knowledge. In Advances in Neural Information Processing Systems , 2022.

Mayleen Cortez, Matthew Eichhorn, and Christina Lee Yu. Exploiting neighborhood interference with low order interactions under unit randomized design. arXiv preprint arXiv:2208.05553 , 2022.

Dean Eckles, Brian Karrer, and Johan Ugander. Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference , 5(1):20150021, 2016.

Laura Forastiere, Edoardo M Airoldi, and Fabrizia Mealli. Identification and estimation of treatment and interference effects in observational studies on networks. Journal of the American Statistical Association , 116(534):901-918, 2021.

Laura Forastiere, Fabrizia Mealli, Albert Wu, and Edoardo M Airoldi. Estimating causal effects under network interference with bayesian generalized propensity scores. Journal of Machine Learning Research , 23(289):1-61, 2022.

Peter W Glynn, Ramesh Johari, and Mohammad Rasouli. Adaptive experimental design with temporal interference: A maximum likelihood approach. Advances in Neural Information Processing Systems , 33:15054-15064, 2020.

Huan Gui, Ya Xu, Anmol Bhasin, and Jiawei Han. Network a/b testing: From sampling to estimation. In 24th International Conference on World Wide Web, WWW 2015 , pages 399-409. Association for Computing Machinery, Inc, 2015.

Kevin Han and Johan Ugander. Model-based regression adjustment with model-free covariates for network interference. arXiv preprint arXiv:2302.04997 , 2023.

Christopher Harshaw, Fredrik Sävje, David Eisenstat, Vahab Mirrokni, and Jean Pouget-Abadie. Design and analysis of bipartite experiments under a linear exposure-response model. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 606-606, 2022.

Christopher Harshaw, Fredrik Sävje, Daniel Spielman, and Peng Zhang. Balancing covariates in randomized experiments with the gram-schmidt walk design. arXiv preprint arXiv:1911.03071 , 2019.

Christopher Harshaw, Fredrik Sävje, and Yitan Wang. A design-based riesz representation framework for randomized experiments. arXiv preprint arXiv:2210.08698 , 2022.

Michael G Hudgens and M Elizabeth Halloran. Toward causal inference with interference. Journal of the American Statistical Association , 103(482):832-842, 2008.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019 . OpenReview.net, 2019.

Michael P Leung. Causal inference under approximate neighborhood interference. Econometrica , 90(1):267-293, 2022.

Michael P Leung. Rate-optimal cluster-randomized designs for spatial interference. The Annals of Statistics , 50(5):3064-3087, 2022.

Hannah Li, Geng Zhao, Ramesh Johari, and Gabriel Y Weintraub. Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. In Proceedings of the ACM Web Conference 2022 , pages 182-192, 2022.

Yang Liu, Yifan Zhou, Ping Li, and Feifang Hu. Adaptive a/b test on networks with cluster structures. In International Conference on Artificial Intelligence and Statistics , pages 1083610851. PMLR, 2022.

Joel A Middleton and Peter M Aronow. Unbiased estimation of the average treatment effect in cluster-randomized experiments. Statistics, Politics and Policy , 6(1-2):39-75, 2015.

Preetam Nandy, Kinjal Basu, Shaunak Chatterjee, and Ye Tu. A/b testing in dense largescale networks: design and inference. Advances in Neural Information Processing Systems , 33:2870-2880, 2020.

Frank Oertel. Grothendieck's inequality and completely correlation preserving functions-a summary of recent results and an indication of related research problems. arXiv preprint arXiv:2010.00746 , 2020.

Ben M Parker, Steven G Gilmour, and John Schormans. Optimal design of experiments on connected units with application to social networks. Journal of the Royal Statistical Society. Series C (Applied Statistics) , pages 455-480, 2017.

Jean Pouget-Abadie, Kevin Aydin, Warren Schudy, Kay Brodersen, and Vahab Mirrokni. Variance reduction in bipartite experiments through correlation clustering. Advances in Neural Information Processing Systems , 32, 2019.

Yichen Qin, Yang Li, Wei Ma, and Feifang Hu. Pairwise sequential randomization and its properties. arXiv preprint arXiv:1611.02802 , 2016.

Ryan A. Rossi and Nesreen K. Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI , 2015.

Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology , 66(5):688, 1974.

Michael E Sobel. What do randomized studies of housing mobility demonstrate? causal inference in the face of interference. Journal of the American Statistical Association , 101(476):13981407, 2006.

Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. Graph cluster randomization: Network exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 329-337, 2013.

Johan Ugander and Hao Yin. Randomized graph cluster randomization. Journal of Causal Inference , 11(1):20220014, 2023.

Davide Viviano. Experimental design under network interference. arXiv preprint arXiv:2003.08421 , 2020.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning , pages 5-32, 1992.

Christina Lee Yu, Edoardo M Airoldi, Christian Borgs, and Jennifer T Chayes. Estimating the total treatment effect in randomized experiments with unknown network structure. Proceedings of the National Academy of Sciences , 119(44):e2208975119, 2022.

Christina Lee Yu, Edoardo M Airoldi, Christian Borgs, and Jennifer T Chayes. Graph agnostic randomized experimental design. arXiv preprint arXiv:2205.12803 , 2022.

Yuan Yuan and Kristen M Altenburger. A two-part machine learning approach to characterizing network interference in a/b testing, 2022.

Qiong Zhang and Lulu Kang. Locally optimal design for a/b tests in the presence of covariates and network dependence. Technometrics , 64(3):358-369, 2022.

Corwin M Zigler and Georgia Papadogeorgou. Bipartite causal inference with interference. Statistical science: a review journal of the Institute of Mathematical Statistics , 36(1):109, 2021.

A Proofs

A.1 Proof of Proposition 1

Firstly, we notice that

Then we substitute Y i ( z ) in HT estimator

with our potential outcome model and take expectation w.r.t. treatment assignments,

Moreover, we can calculate the GATE under our potential outcome model

Hence, the bias of HT estimator is

Then we derive the matrix form with cluster-level treatment vector t .

̸

A.2 Proof of Proposition 2

Firstly we rewrite the estimator

We then expand the variance directly, and get

then we write it as matrix form with cluster-level treatment vector t ,

and expand the variance as covariance, we get the desired format

A.3 Proof of Proposition 3

Firstly we consider the estimator of matrix form again

Since our assumption can't guarantee E [ˆ τ ] > 0 always hold, we drop the square of expectation term, namely

Notice that all elements of matrix E [ tt T ] is non-negative, we have

Similarly, since all elements of matrix C , namely, C ij is also non-negative, we have

In summary, we have

By definition, we have

Thus the upper bound above is actually

At last, plug in following equation.

A.4 Proof of Lemma 1

Here we provide an intuitive proof. Utilizing Box-Muller transformation or pure algebraic analysis are also feasible.

We consider X and Y is generated from multivariate Gaussian distribution,

where g ∼ N (0 , I n ) and x, y are two n -dim real vectors. Then we know that

Then we think geometrically that sgn( ⟨ x, g ⟩ ) sgn( ⟨ y, g ⟩ ) > 0 holds iff. g lies above or below both of the hyperplanes that is orthogonal to x and y respectively.

Notice that the direction of g is uniform, it follows that

Now we put things together

which gives the desired outcome.

B Simulation Details

B.1 Methods

We consider following linear potential outcome model,

and multiplicative model, which is a simplified version of that in [36], with removing a covariate.

Moreover, we have thus

We choose to fix all parameters except for interference density, γ . Namely, we set ( α, β, c, σ ) = (1 , 1 , 0 . 5 , 0 . 1) for both models, and set γ ∈ { 0 . 5 , 1 , 2 } to construct three regimes. ϵ i ∼ N (0 , 1) .

We set the clusters as given by Louvain algorithm with fixed random seed and resolution parameter as 2 , 5 , 10 .

We consider social network FB-Stanford3[32], 4 and FB-Cornell5[32]. 5 . These two social networks provide the network topology, and we generate potential outcome for each unit with mentioned potential outcome models.

Besides our optimized covariance design (OCD), we implement following randomization schemes. We first consider two baselines, independent Bernoulli randomization (Ber) and complete randomization (CR), both of which are cluster-level. We also implement two adaptive schemes, rerandomizedadaptive randomization (ReAR) and pairwise-sequential randomization (PSR) [25, 31], which balance heuristic covariates adaptively and act as competitive baselines, since the average degree is considered as a covariate in these two methods and exactly appear in both two of our models, explicitly. Then we implement the independent block randomization (IBR) [5] and heuristic version IBR-p that creates blocks with size 2.

In the methods mentioned above, ReAR is the only one concerned with hyperparameters setting. We set ( q, B, α ) = (0 . 85 , 400 , 0 . 1) , which corresponds to the recommendation in the original paper.

We estimate GATE with standard HT estimator and difference-in-means (DIM) estimator, where the latter refers to

To summarize, we provide the bias, standard deviation and MSE of two estimators under two potential outcome models, three γ levels and two datasets. All of these three metrics are calculated by repetition of Monte Carlo simulation of randomization, 10,000 times. In the main paper we've presented the results of HT estimator on FB-Stanford3, and we'll present the detailed results in this section.

B.2 Discussion on Estimators

Here we also provide discussion on another popular estimator that's considered in existing literature, which is the HT estimator with exposure indicator. We consider the exposure condition that's fully treated or fully controlled in 1-hop neighborhood here, and the corresponding indicator is defined as δ i ( z 0 ) = I { ∑ j ∈ N i z j = d i z 0 } . The estimator is

We don't consider this estimator not only because of it's high variance resulted from low effective sample size, and but also its high calculation cost. For calculating it in our repeated simulations, we need to estimate E [ δ i (1)] , the probability of fully treated, for each unit i . We denote the number of clusters a unit i connects to by c i . For a unit i in the exterior of its cluster, namely, c i > 1 , such a quantity can very high in a social network, which is also decided by the resolution of clustering, we present an instance in figure 1.

Unfortunately, we must estimate such generalized propensity score [10] by Monte Carlo simulation for most of randomization schemes, even if it's just a little bit more complex than independent Bernoulli randomization, where such quantity can be calculated directly, (1 / 2) c i . For every simulation, we should visit every node and query the treatment assignment of its 1-hop neighborhood, whose time complexity is O ( | E | ) . Roughly, we need 2 c i repetitions of randomization to achieve effective estimation on δ i (1) , δ i (0) , which is too time-consuming to be acceptable for many units.

0

20

40

60

80

Number of Connected Clusters

0

250

500

750

1000

1250

1500

1750

2000

Nodes Count

We stress the computational issue here since the resulted variance can be reduced by selfnormalization, which corresponds to the Hájek estimator.

However, we argue that the computational cost can't be bypassed without further modification and restricts the practicality of such HT estimator with exposure indicator. [25] proposes the so-called cluster-adjusted estimator that assign the units on the exterior of same cluster the same modified propensity score, which reduces the computation complexity at the cost of unpredictable distortion of estimator, which can be viewed as heuristics.

Therefore, we choose to consider the standard HT estimator and DIM estimator in our simulation.

B.3 Detailed Simulation Results

We present the results according to following sequence: HT-DIM estimator, linear-multiplicative model, 2-5-10 clustering resolution. In every table, we report the average bias, standard deviation (SD) and MSE of randomization schemes with three γ levels. There are 24 tables in total, which are organized to demonstrate the advantage and robustness of our proposed method, optimized covariance design (OCD).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focused on designing a randomization scheme at the cluster level for the A/B test. It proposed and derived an upper bound for the MSE of the HT estimator, which was targeted to be minimized to optimize the experiment design. This article treated the covariance matrix of the treatment vector as the decision variable in optimization to represent a class of design. Besides, it proposed to adopt projected gradient descent to guarantee the validity of the optimized covariance matrix in order to generate a randomization scheme that supports legitimate sampling. Also, it stressed the significance of boas and attempted to balance bias and variance better. Systematic simulations on a real-time social network were conducted. Experiment outcomes were analyzed and different methods were compared to show the effectiveness of the newly proposed method in this article.\n\nSTRENGTHS:\n1. This paper is generally easy to follow.\n2. The problem is well formulated in a new perspective, with detailed analysis and deduction.\n3. The effectiveness of the proposed method is verified in the experiment. Horizontal comparisons with existing methods are sufficient and effective.\n\nWEAKNESSES:\n1. Insufficient background introduction. Different concepts in the area should be introduced more specifically and it is better to point out their connections. \n2. Unclear notation definitions and explanations, leading to confusion in understanding.\n3. Experiments were not comprehensive, or at least not comprehensively expressed.\n\nQUESTIONS:\n1. Where is the definition of di that appeared below line 150 in assumption 1? (Does it refer to the degree of node i?) It is hard to read when definitions are missed or distributed in different parts. It is better to summarize all of them into one table with clear definitions.\n2. In section 2.1, line 132, what are the certain community detection algorithms? It is better to list out some works with references.\n3. The experiment part seemed to be written in a hurry. Even though necessary comparison methods were involved, many other essential details such as the basic information about the dataset (i.e., edges, nodes, degrees, etc.) were missing. Also, it seems that experiments were only implemented on one dataset, which made the experiment outcome not convincing enough.\n4. What is the real-world application of this method? Please offer some direct applications such as the areas that may benefit from your method. This is important since your derivation used different assumptions when dealing with difficult parts in the optimization. Will the assumptions affect its performance in general? Is the proposed method only workable under a specific scene and what is the case?\n\nLIMITATIONS:\nLimitations of this article were clearly expressed in this article. For instance, the upper bound should be tighter in the optimization. And different other cases like the non-balanced treatment assignment have not been considered yet. The basic setting beyond nationhood interference assumption warrants a further extension. Besides, assumptions may hinder the method from applying to a more generalized case.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors present a novel experimental design to be used for randomized experiments under network interference.\nAuthors begin by considering a baseline adjusted Horvitz--Thompson estimator (which crucially requires knowledge of $Y_i(\\mathbf{0})$) and a pre-specified clusterings.\nGiven these two things, authors derive a bound on the mean squared error of the estimator under an arbitrary cluster design.\nMotivated by a Grothendeick identity, authors propose an experimental design where a normal vector is sampled from a fixed covariance matrix $\\Sigma$ and the signs of the vector become treatment assignment.\nAuthors propose to select an experimental design by (essentially) using the normal covariance matrix $\\Sigma$ as a decision variable and the upper bound on the mean squared error as an objective.\nFinally, simulations are run to investigate the performance of the method under various types of model-mispecification.\n\nSTRENGTHS:\nThe main strength of the paper is a novel method for designing randomized experiments under network interference. The novelty of the method comes from developing an objective function whose decision variables are continuous, (essentially) representing the covariance matrix of assignments. This stands in contrast to previous methods which focus on selecting clusters for independent cluster randomization designs. An additional strength of the paper is that the objective considers both bias and variance, which is uncommon: most approaches attempt to \"fix\" one of these two things, but typically not both.\n\nOne of the most exciting technical connections is the use of Grothendieck's identity, which (to the best of my knowledge) has not been used in causal inference. This offers a new tool in the design of experiments, which I suspect will be useful beyond the specific design used in this paper.\n\nMoreover, the simulations are very well formulated and executed.\nIn particular, the authors investigate the effectiveness of their design under various forms of model misspecification, which speaks to the robustness of the design.\nThis is important to investigate via simulations, as formal theory seems difficult given the black-box nature of the experimental design.\n\nWEAKNESSES:\nThere are several weaknesses of the current method.\n\n1. **Assumed Knowledge**: The paper makes some strong assumptions as to what the experimenter knows. For example, authors assume that the coefficients $\\alpha_i$ are known by the experimenter. I think most experimenters will find that knowledge of each individual $\\alpha_i$ is too strong of an assumption to be practical -- in the SUTVA setting, this implies that all individual treatment effects can be estimated perfectly without any randomization. In order to be transparent, authors should state this assumption and estimator earlier in the paper, perhaps replacing the HT estimator in (eq 5).\n2. **Pre-specified Clusters**: The paper assumes that clusters are pre-specified and little advice is given to practicioners on how to select the clusters in order to minimize MSE.\n3. **Understanding of the Variance**: In order to run power calculations, experimenters should have at least a rough understanding (e.g. asymptotic rates) of how the variance of the experimental design depends on sample size $n$ and network parameters. Because this method is based on a black-box optimization procedure, it seems hard to analyze the variance (i.e. optimal value).\n4. **Confidence Intervals**: In practice, experimenters value interval estimators (i.e. confidence intervals) more than point estimators, as they provide a method for uncertainty quantification. The necessary tools for uncertainty quantification (e.g. Central Limit Theorem, variance estimation) are not presented in this paper.\n\nWhile authors should make certain minor changes to address the above, I think that these weaknesses actually constitute further research directions on this exciting method.\nOverall, it is my opinion that the strengths of the paper outweigh the weaknesses.\n\nIn the two sections below, I discuss minor weakenesses in the technical discussions and literature review which should be addressed by authors before publication.\n\n## Technical Remarks\n\nBelow are some minor remarks on technical aspects of the paper.\nI think there are a few technical issues in the discussions, but I believe these can be easily fixed and will strengthen the technical contribution of the paper.\n\n1. (Line 137) authors write \"without loss of generaltiy, we consider the balanced cluster-level randomization scheme satisfying...$E[z_i] = 1/2$.\" I think that this restriction is perfectly fine, but I would say it is not technically correct to describe it as \"without loss of generality\". Consider the usual SUTVA setting: if the outcomes under treatment have more variation than outcomes under control, then the Horvitz--Thompson estimator can be made to have smaller variance by setting $p$ so that treatment is assigned more frequently. I think the phrase \"without loss of generality\" is not warranted and a simple fix is to just remove it.\n2. (Line 152) The word \"overparametrized\" is not quite standard in this literature so I'd recommend either informally defining it, or just saying that \"there are more unknown potential outcomes than observations\". This is true even under SUTVA.\n3. (Line 208): Authors introduce what they call \"Assumption 2\". I would refer to this as a \"condition\" rather than an assumption. The reason is that Assumption 2 only plays a role in choosing the design -- using standard techniques (CLT + variance estimator), Assumption 2 would not be necessary for, say, the validity of confidence intervals. Some readers might misinterpret the use of the word \"Assumption\" to think \"if this condition does not hold, then the estimates are no longer statistically valid in some sense\".\n4. (Line 254) Authors write \"This lemma enables us to sample from bivariates Bernoulli distribution with mean (1/2, 1/2) and any valid covariance\". While this is true for $n=2$ variables, I do not believe it to be true for general $n$ variables. It is sort of implied in the Section that this \"Grotendeick mapping\" can recover any covariance matrix of $\\pm 1$ variables. If authors have a proof of this, they should provide it; otherwise, they should clearly state that this \"Groethendeick mapping\" cannot generate all $\\pm 1$ covariance matrices. I think that clarifying this will increase understanding and appreciation of the method.\n5. (Line 294): The Monte Carlo simulation is only performed 200 times. In my experience, this is quite low. Can you increase to 10,000 Monte Carlo runs before camera ready submission? This will increase the reader's confidence in you results.\n\n## Literature Review + References\n\nThe authors have missed a few important references in the causal inference literature.\nI believe these should be easily fixable and would increase the relevancy of the paper by better situating it within the causal inference literature.\n\n1. (Line 36-37) In reference to cluster designs, authors write: \"This technique is originally developed in [31] and becomes a prevalent paradigm for network experiment design\". I completely agree that [31] was an influential paper for introducing cluster designs to the computer science community. [35] works the exposure mapping framework for interference [1] which allows for this arbitrary network interfernece. However, the so-called \"partial interference\" assumption has been used since at least Hudgens & Halloran (2008) and cluster designs were advocated for here. So, I'd at least reference some of this early work on clustering in the context of partial interference. I think this will also tie your contributions back to the vien of causal inference in the statistics literature in a stronger way.\n2. (Line 38) Authors write that \"sharing same treatment within cluster is usually necessary for characterizing the GATE\". I would remove or substantially weaken this statement. The proliferation of cluster designs is *not* because they are necessary; but rather, a conceptually simple yet effective type of experimental design.\n3. (Line 64) Authors write \"In this paper, we propose to treat the covariance matrix of treatment vector as a decision variable in optimization\". The following paper seems especially relevant and authors should draw a comparison: Harshaw et al (2019) \"Balancing Covariates in Randomized Experiments using the Gram--Schmdit Walk Design\". This paper studies experimental designs that directly control the covariance matrix Cov(z) (via discrepancy theory) to bound the variance of Horvitz--Thompson estimator by an implicit ridge regression of outcomes on covariates. A key idea in that paper is to use the operator norm as a measure of worst-case variance, which seems like an alternative to your Assumption 2. Given the similarity in the spirit of the two papers, this paper would benefit from a brief comparison discussion.\n4. (Line 94) Authors cite several papers on bipartite experiments. It seems that [15] and [16] are duplicates. To the best of my knowledge, the paper of Zigler & Papadogeorgou (2021) was the first paper to propose bipartite experiments (it has been a working paper since 2019), so a citation is warranted in that discussion.\n5. (Line 104): Authors write \"Along the same direction, [32] tries to provide...\" I recommend revising this language. The word \"tries\" gives the indication that \"[32] tries and fails\".\n6. (Line 108): In the discussion of partial interference, early work like Hudgens and Halloran (2008) is missing.\n7. (Line 116): A citation of several papers with more general forms of interference is listed. The recent paper Harshaw, Sävje, Wang (2022) \"A design-based riesz representation framework for randomized experiments\" is worth citing, as it proposes a deisgn-based framework which captures and extends previous types of interference.\n8. (Line 251): I haven't read [19], but it was my understanding that Grothendieck's identity is typically a different type of statement, where there are two fixed vectors $x$ and $y$ and the random variables are $\\textrm{sign}(\\langle z , x \\rangle)$ and $\\textrm{sign}(\\langle z , y \\rangle)$, where $z$ is uniform from the $\\ell_2$ ball. In fact, I have only seen Lemma 1 in certain course notes on Sums-of-Squares (though I'm sure it's appeared in other places). If Lemma 1 does not directly appear in [19] then authors should cite a relevant paper that derives it. In fact, this would probably be helpful to tie your work back to theoretical computer science's use of the technique.\n\n## References\n\n- Harshaw, C., Sävje, F., Spielman, D., & Zhang, P. (2019). \"Balancing covariates in randomized experiments with the Gram-Schmidt Walk design\". (arXiv:1911.03071)\n- Harshaw, C., Sävje, F., & Wang, Y. (2022). \"A design-based riesz representation framework for randomized experiments\". (arXiv:2210.08698)\n- Hudgens, M. G., & Halloran, M. E. (2008). \"Toward causal inference with interference\". Journal of the American Statistical Association, 103(482), 832–842.\n- Zigler, C. M. and Papadogeorgou, G. (2021). \"Bipartite causal inference with interference\". Statist. Sci., 36(1):109–123.\n\nQUESTIONS:\n1. One way to avoid \"picking clusters\" is to run the method when each unit is its own singleton cluster. How does the proposed method perform in this case? In other words, is the specification of clusters necessary to achieve small variance? If so, can you describe why or generally comment more on this? (this is a discussion I would find really fascinating in the paper)\n2. I claimed that the \"Grothendeick mapping\" cannot recover all covariance matrices on $\\pm 1$ matrices. Can you comment on this?\n3. Can you increase the Monte Carlo samples to 10,000?\n4. Can you comment on the comparison to the Gram--Schmidt Walk Design of Harshaw et al (2019)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new algorithm for A/B test design under network effect with cluster level randomization. By derivation of an upper MSE upper bound with bias-variance trade-off, the authors reparameterize to directly optimize the covariance of the treatment vector. An efficient PGD algorithm is proposed for efficient optimization. Simulation study on a real-world network demonstrates the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The authors study a very important and practical problem in A/B test design with reasonable assumptions on cluster level randomization.\n2. The paper is well written and the proposed method is both explained clearly, with reasonable assumptions  and well justified with detailed derivations.\n3. The proposed reparameterization and the PGD optimization algorithm makes  the algorithm practical and easy to implement.\n\nWEAKNESSES:\n1. It seems that the clustering algorithm plays a very important role in the quality of the A/B experiment design. From the covariate matrix formulation, maybe in some cases it would be better to directly merge two clusters instead of using the treatment vector to guarantee the same treatment. It would be very interesting to see how the proposed method performs under different cluster methods /parameters besides the number of clusters.\n2. As A/B experiments are usually used in real large-scale networks, it would be interesting to see scalability results in terms of number of clusters for different methods under comparison.\n3. As the authors point out in the future work, it would also be interesting to see some empirical evaluation on the tightness of the proposed upper bound on the MSE. It may provide some insight on whether further tightening the bound can bring additional benefits.\n\nQUESTIONS:\nPlease see above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an optimized covariance design for A/B tests on social networks with interference. The authors address the challenge of accurately estimating the global average treatment effect (GATE) in the presence of network interference. They propose a method to balance bias and variance in experimental design by optimizing the covariance matrix of the treatment assignment vector. The paper derives the bias and variance of the estimator and proposes an algorithm to implement the desired randomization scheme. Simulation studies demonstrate the advantages of their method over existing methods in various settings. Overall, the paper contributes an approach to improve the estimation of treatment effects in social network experiments by optimizing the covariance design.\n\nSTRENGTHS:\nThis paper presents a approach to optimize the covariance design for A/B tests on social networks with interference, which improves the estimation of treatment effects and enables more accurate decision making based on experimental results. The paper is well-written and clearly explains the problem formulation, assumptions, and the proposed method. The authors provide a thorough analysis of the bias and variance of the estimator, a upper bound for the MSE of the HT estimator, and a projected gradient descent algorithm to solve the optimization problem. The simulation studies demonstrate the effectiveness and robustness of the proposed method in various settings.\n\nWEAKNESSES:\n1.\tThe paper briefly mentions assumptions about the direct treatment effect and interference effect. However, a more thorough discussion on the validity and generalizability of these assumptions would be valuable. Additionally, discussing the limitations of the proposed method, such as its sensitivity to certain parameter values or potential biases introduced by the assumptions, would provide a more comprehensive understanding of the method's applicability and potential drawbacks.\n2.\tIn the experimental results presented in Tables 2, 5, 6, 7, 10, 11, 14, 15, 19, 22, and 23 in the appendix, it is observed that the proposed algorithm’s performance in terms of MSE is significantly weaker compared to other benchmark algorithms such as ReAR, IBR-p, and PSR. This suggests that further improvements may be necessary for the proposed algorithm to achieve competitive performance\n3.\tThe paper could benefit from a more detailed introduction and explanation of the motivation behind balancing bias and variance. Providing additional context and background information on this topic would help to strengthen the overall argument and improve the clarity of the paper.\n\nQUESTIONS:\nPlease refer to weaknesses 1, 2, and 3\n\nLIMITATIONS:\nWhile the paper briefly mentions assumptions about the direct treatment effect and interference effect, a more thorough discussion on the validity and generalizability of these assumptions would be valuable. Additionally, it would be beneficial for the authors to include a discussion on the limitations and potential negative consequences of their work. This could include an examination of the proposed method’s sensitivity to certain parameter values or potential biases introduced by the assumptions. Providing a more comprehensive understanding of the method’s applicability and potential drawbacks would strengthen the overall manuscript", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}