Optimized Covariance Design for AB Test on Social Network under Interference
Authors: Qianyi Chen
Bo Li
LU DENG
Yong Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: AB test
interference
causal inference
optimization
social network
TL;DR: We develop an efficient and effective network experiment design that optimizes covariance matrix of treatment vector through projected gradient descent.

Abstract: Online A/B tests have become increasingly popular and important for social platforms. However, accurately estimating the global average treatment effect (GATE) has proven to be challenging due to network interference, which violates the Stable Unit Treatment Value Assumption (SUTVA) and poses great challenge to experimental design. Existing network experimental design research was mostly based on the unbiased Horvitz-Thompson (HT) estimator with substantial data trimming to ensure unbiasedness at the price of high resultant estimation variance. In this paper, we strive to balance the bias and variance in designing randomized network experiments.  Under a potential outcome model with 1-hop interference, we derive the bias and variance of the standard HT estimator and reveal their relation to the network topological structure and the covariance of the treatment assignment vector. We then propose to formulate the experimental design problem as to optimize the covariance matrix of the treatment assignment vector to achieve the bias and variance balance by minimizing the mean squared error (MSE) of the estimator. An efficient projected gradient descent algorithm is presented to the implement of the desired randomization scheme. Finally, we carry out extensive  simulation studies to demonstrate the advantages of our proposed method over other existing methods in many settings, with different levels of model misspecification.

Optimized Covariance Design for AB Test on Social Network under Interference
Qianyi Chen', Bo Li"e Lu Deng", Yong Wang
School of Economics and Management, Tsinghua University, China
Tencent Inc., Shenzhen, China
cqy228mails tsinghua.edu. en, 1ibo@sem. twwngnua..cnn.m.. edu. adandeng@tencent. com, darvinvang@tencent. com
Abstract
Online A/B tests have become increasingly popular and important for social plat- forms. However, accurately estimating the global average treatment effect (GATE) has proven to be challenging due to network interference, which violates the Stable Unit Treatment Value Assumption (SUTVA) and poses great challenge to experi- mental design. Existing network experimental design research was mostly based the unbiased Horvitz-Thompee to ensure unbiasedness at the price of high resultant estimation variance. In this paper, strive to balance the bias and variance in designing randomized network experiments. Under a potential outcome model with 1-hop interference, we derive the bias and variance of the standard HT estimator and reveal their relation the network topological structure and the covariance of the treatment assignment vector. We then propose to formulate the experimental design problem as to op- timize the covariance matrix of the treatment assignment vector to achieve the bias and variance balance by minimizing well-crafted upper bound of the mean squared error (MSE) of the estimator, which allows us to decouple the unknown interference effect component and the experimental design component. An effi- cient projected gradient descent algorithm is presented to the implement of the desired randomization scheme. Finally, we carry out extensive simulation studie to demonstrate the advantages of our proposed method over other existing methods in many settings, with different levels of model misspecification.
(HT) estimator with substantial data trimming
Introduction
A/B test, also known as controlled experiment, an effective way to evaluate treatment effect of new product features and facilitate data-driven Docision-makingg [14). Due to its simplicity and efficiency, it has become the gold standard in the industry. The theory of A/B test based on the Stable Unit Treatment Value Assumption (SUTVA), under Seymann-Rubin 's framework [B], which assumes unit's 's treatment effect isn't influenced by other units' treatment assignments. However, in situations where social network connects units, SUTVA is often violated since an individual's behavior can be influenced by their friends. This violation of SUTVVA is referred as network interference or social interference, which brings challenges both estimation and inference of parameters of interest. has been shown that ignoring network effect can induce much higher bias or variance [29].
To tackle the bias and variance induced by network interference, researchers have tried to design randomization schemes in pre- pre-treatment phase and estimators or adjustment methods in post-treatment phase. In this paper, we will focus on designing randomization scheme, namely, experiment design.
"Corresponding author
"The code available at http://www...............///o/oa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Specifically, we take the graph structure as known and leverage it to assist experiment design. Our basic scheme is cluster-level randomization, which treats cluster or community in the graph as unit of treatment and all units in the same cluster share the same treatment. [9 is an early work investigating this approach, and [35] proposes to form clusters utilizing graph. The graph cluster randomization is conceptually simple but effective in many scenes, and thus becomes prevalent paradigm for network experiment design. Moreover, we consider experimental design on social network in this work. Social network typically has organic structure and high clustering coefficient, which corresponds to meaningful cluster structure and also motivates us to adopt cluster-level randomization. We assume the cluster structure of the network is given. In practice, interpretable community detection algorithms such as the Louvain algorithm 4 are typically applied for cluster construction. Moreover, we stress the scalability of our method since social network is usually large-scale in reality.
Related works have shown that there exists an explicit pias-variance tradeoff 41]. However, many sty- estimators that utilize exposure indicator to ensure unbiasedness are designed on the basis of the neighborhood interference assumption that restricts global interference to l-hop neighborhood
39 9 and graph-dependent generalized propensity score [I]. Nonetheless, we argue that pursuing unbiasedness by simple sample trimming leads to small effective sample size with high variance, and thus is far from optimal. Moreover, computational cost is also high in this approach since a large number of Monte Carlo simulations are usually needed for computing the generalized propensity score, the trimming probability for constructing sample weights. Having observed the preceding drawbacks of the unbiased estimator, we consider the standard unit-level HT estimator to achieve possible better bias and variance tradcoff with less computation burden at the meantime. Based on a potential outcome model, we characterize the relationship among the bias, variance, the network topology, and the treatment assignment mechanism. This development allows to design experimental schemes that minimizes MSE and achieve better bias-variance tradeoff.
10 25
36
Besides from the time-consuming Monte Carlo simulation for calculating generalized propensity score mentioned above, the computational efficiency of experiment design is indeed a limitation of current methods. For instance, the two-wave experiment (37], entails solving a mixed-integer quadratic programming problem, which can be computationally expensive. The adaptive covariates- balancing design (25) adopts sequential process for treatment assignment and thus isn' 't very scalable. To address this limitation, we propose to utilize gradient-based optimization algorithm.
In this paper, we propose to treat the covariance matrix of treatment vector as decision variable in optimization, which is nontrivial as a result of interference and corresponding complex variance of estimator. Similar approach in optimization step is proposed in [17], where interference isn't an't considered, and making covariance covariance acting as decision variable is a natural choice owing to the simple structure of variance there. We draw attention to the fact that related works concerning experiment design mostly concentrates on predetermined class of randomization schemes and seeks to optimize the parameters to obtain designs that minimize variance or MSE of estimator 337 23 5 (4). However, we argue that these designs can all be understood as different patterns of the covariance matrix of the treatment vector, and propose to treat the covariance matrix as decision variables rather than limiting our scope to a particular class of designs. One of major challenges with this approach is generating a randomization scheme that supports legitimate sampling, as there may not be joint distribution that corresponds to the optimized covariance matrix in our formulation. To address this issue, we propose adopt projected gradient descent to guarantee the validity of optimized covariance matrix. Furthermore, we also compare our method with other popular designs through extensive simulation on a real online social network. Various interference intensity levels and potential outcome models are considered. It is found that our method is robust to model misspecificatioon and outperforms in almost all settings, especially when the interference isn 't too weak compared to the direct treatment effect, which is the scene we target.
Contributions. Our contributions are summarized as follows.
1. We derive well-crafted upper bound for the MSE of the HT estimator that decouples the estimation of causal mechanism and experimental design. This enables us to optimize the experimental design by minimizing this bound, in which the covariance matrix of treatment vector acts as decision variables.
2. We propose the formulation of covariance optimization problem with reparameterization of covariance matrix and constraints that guarantees legitimate sampling subject to optimized
2

---

covariance matrix, and projected gradient descent algorithm is proposed for solving the optimization problem.
3. We conduct systematic simulation on a social network and compare our method with several methods proposed recently under range of settings, providing credible reference for the effectiveness of our method.
Related Works. A variety of interference has been considered in related works [12, such as spatial 23
temporal 13 and network, which is the scene we target. Besides from the general network, there is also another line of work focused on a two-sided marketplace or platform, where the graph is
bipartite [3 24
16 30 27].
In the field of network experiment design, most of extant research assumes that the graph structure is known and leverage it to assist in modeling the interference(2] though several graph agnostic methods are developed these days 40 g. Nevertheless, the interference conducted through edges of network, and agnostic methods aren 't effective enough.
Beyond unit-level randomization, cluster-level randomization on graph has attracted extensive atten- tion, including both experimental design [3 25 5 [30] and estimation 126
I 6 [5] The seminal paper of this line of work is [35], which proposes graph cluster randomization that treats clusters as units in randomization. This methodology is motivated by providing a better lower bound of exposure probability in HT estimator that uses exposure probability as inverse weight, aiming at controlling the variance of such an unbiased estimator. Along the same direction, [36] proposes a polynomial lower bound on exposure probability through randomized local clustering algorithm.
Irregular global interference may render the experimental design problem intractable (39. thus structural assumptions are usually introduced to rule out such arbitrary interference. The first category is partial interference [34] [9 3 51 which assumes disjoint community structures and interference only happens within each community. The second category neighborhood interference, which assumes the interference solely originates from units' 1-hop neighborhood [5] 36
8 This
kind of assumption is more practical compared with partial interference, and thus widely applied. The
third category is potential outcome model, which directly assumes the expression of potential outcome,
e.g. classical Cliff-Ore auto rugo-reegessive model 77]. linear potential outcome models 25 39 Actually, some potential outcome model implies neighborhood interference assumption in its mathematical
expression. It's also worth to be noted that there also appear works trying to weaken the structural assumptions mentioned above and consider the interference of more general form 6 22
18
15
Basic Framework
2.1 Setup
We consider finite population of units, and the treatment vector is represented as (21, $2..2..... 2n,) E 10. In this paper, we consider the experiment design problem in which sensible random treatment assignment mechanism of all units is designed to achieve good perfor- mance of the resultant treatment effect estimation. We follow the Neyman- Rubin's framework [33] Without SUTVA assumption, we define the potential outcome of unit as Y; Yi(z), which means the potential outcome of unit i can depend on the treatment assignment of all units.
The parameter of interest is the global average treatment effect (GATE), which is denoted by
1 Σ (Y/(1) -V(1)))((0)
1E(m]
Here we use [n]] to denote set {1, 2,..,.., ..., n). and 1(0) is the n-dimensional vector of 1(0), which corresponds to global treatment and global control.
We denote the social network g (V, E). The node set V is [ni], corresponding to all units considered above. Edge set E represents the social connection between units. Furthermore, we use d, to denote the degree of node i. We assume the graph structure G is known in advance.
In this paper, we consider graph cluster randomization [35.. Here we also assume that cluster structure is given, which is generated from certain community detection algorithm. We define S₃ as the k-th
3

---

cluster, where & E [K]. Clusters compose splitting of node set, namely,
S₁ C [m]
S ns
- Sy = [n|]
(2)
The graph cluster randomization implies that units in the same cluster share the same treatment assignment, namely.
zj j E Sk, k [K]
Since we consider randomization scheme at cluster level, it's more convenient to define cluster-level treatment vector t = (11, 12....." tx) {0, k Specifically, we consider the balanced cluster-level
randomization scheme satisfying
P(t = 1|9) =
E[2,]
(4)
2
2.2 Potential Outcome Model and Estimators
One of the key ideas of this paper is to deduce an appropriate objective function to optimize experiment design schemes under consideration. In this part, we will consider standard HT estimator, namely,
2g
(1 22) E 2i
(y((2)
(5)
[[1]]
3E[m]
Then we introduce the considered potential outcome model.
Assumption (Potential Outcome Model) The potential outcome of unit is generated by
(2(2) = a, + Bi2s +
Σ
2j
(6)
JEN,
where N₂ the I-hop neighborhood of unit
In this potential outcome model, 3; can be interpreted as direct treatment effect of unit i, and measures the interference effect, characterizing the extent to which each unit is affected by its neighborhood in the network. This model is simplified (in interference part) version of over- parameterized model in [39], which heterogeneous interference effects are accommodated. We make somewhat stronger assumption to simplify the structure of the bias, variance and MSE. Under this model, we can express the GATE in terms of the potential outcome model parameters.
1 Σ (Y((1) - ((())) V
1 Σ 7d4+ = n sEin]
(7)
sE(m]
We will then derive the bias and variance of f under this potential outcome model. We point out that the number of unknown parameters in this model is still greater than the number of known potential outcomes, which can be understood as overparameterizatioo, and we only utilize it to derive an optimizable MSE upper bound that's not concerned with estimation of B3 and in this model. In fact, when the interference effect and the direct treatment effect are comparable at cluster-level, a multiplier of the MSE upper bound can be factored out, which is crucial our methodology and will be shown in section 3.1
2.3 Bias of HT Estimator
Based on the potential outcome model, we can derive the bias and variance of our estimator ex- plicitly. We firstly introduce cluster-level treatment assignment, (11, 42, (k), and matrix C summarizing edges between( between(within) clusters with its elements defined as
Cj k(u, v) (u,v) E,u S S,H
(8)
We have following proposition about bias.

---

Proposition 1 (Bias of HT Estimator) The bias of standard HT estimator is
7 B r n 4 Cov(t))
ΣCb Ci
(9)
(JE[K]
where Cov(t, refers to the covariance matrix of treatment vector t.
Specifically, we can delineate the source bias in our setting. we consider graph cluster random- ization basic scheme, edges within cluster don't 't contribute any bias, while every edge between clusters contributes to bias according to the covariance of cluster-level treatment assignments. This enlightens that independent treatment assignments is far from optimal, and we can reduce bias by assigning treatments with appropriate positive correlations that should be higher for more densely connected cluster pair. Furthermore, we point out that bias may be of no account when very sparse network is considered, but it indeed counts in our context of social network.
We also note that the treatment vector is summarized by its covariance matrix in the expression of bias. Hence, we denote the bias by B(Cov((t) (t)).
2.4 Variance of HT Estimator
To derive the variance, we plug in the potential outcome model and express the estimator as
2
+ (3: + 2a, yd4422 + 2 242j
Σ
(10)
n
JEN,
An important difference in the derivation of variance contrasting that of bias is all base levels o; is no longer negligible. Hence, we consider target at the scenes that we can acquire knowledge of all base levels in advance.
Assumption (Known Base Levels) The base level a4 of unit is known in advance for all units i.
This assumption is strong the classic context of experiment design, while we propose to target the scene where AB tests are launched by social platforms. On the one hand, the improvement in user activity indices brought by new traits or interventions, specifically, direct treatment effect and interference, is typically much smaller than the base levels ag, which means the experiment design may quite challenging when agnostic to base levels. On the other hand, it's very reasonable to assume that know all a;s before conducting experiments since those indices we' re concerned with are recorded and/or estimated daily or weekly by social platforms.
Utilizing this assumption, we can first adjust our estimator
Zz E
E (Y,((z) -a;)
2;
(1I)
2
*E[m]
This adjustment has also been recommended by 39 Notice that this adjustment helps us remove the influence of base level a, when we consider variance. In the following, we'll adhere to this format and proceed to derive the variance term, which is summarized in the following proposition.
Proposition 2 (Variance of HT Estimator) The variance of standard HT estimator is
Var[f] = n²
(trace(hh" Cov(t)) + 4y Covih Tt, "t,,t CfCt] 41d Var Ct))
(12)
where h is the vector (Ees, -d)
Notice that the expression of variance contains second-order term with regard to f, namely, Covit], as well as third and fourth-order terms involving t" C', quadratic form of t. It's feasible to analyze the variance precisely with specific randomization scheme, such as the independent block randomization [5) but it is generally not straightforward to optimize broad set of randomization schemes in its exact form. the following subsection, we will derive bound of variance to transform third and fourth-order terms into second-order, which also motivates our optimized covariance design.

---

We comment that bias-variance tradeoff is unveiled in the previous two propositions. Actually, C' is a matrix with non-negative elements, and introducing positive covariance between treatment assignments of different clusters will reduce the bias. On the other hand, introducing positive covariance means increment in variance. In addition, though this paper doesn't 't involve upstream clustering algorithm, the resolution of clustering can also impact the relative importance of bias and variance, namely, low resolution usually corresponds to a higher proportion of edges within cluster, which weakens the importance of bias and stress variance.
3 Optimized Covariance Design
3.1 Constructing Objective for Optimization
Although have derived the expression of bias and variance, the expression of variance is com- plicated due to the existence of treatment effects 3;s, the interference effect 7, and the higher-order terms involving the treatment vector. We will bypass the first hurdle by adding reasonable assumption about the direct treatment effect and interference effect. The second challenge will be addressed by upper bound argument.
We first consider the former issue, which is the relationship between 3, and n. Since we consider the interference, we propose to restrict scope the situation that network interference isn't too weak, which is formulated as following assumption.
Assumption (Comparability between Direct Treatment Effect and Interference) Given potential outcome model in equation ( we assume there exists constant w >> such that
hxl
w)(
d:)
(13)
iES
holds for each cluster l E [K].
This assumption actually demands the magnitude of interference is comparable to direct effect at cluster level. Namely, we allow significant difference between direct treatment effect 8; and interference yyd for certain units, but there isn't too much difference after aggregation in cluster. The constant w can be viewed as degree of tolerance on incomparability between direct causal effect and interference and depends on the both nature of potential outcome and graph structure. It's natural that people may be more susceptible to the states of their friends in some social metrics. Since we'll Ill utilize the MSE bound as optimization objective, this constant can also be viewed as tuning parameter that's 's determined by domain knowledge or pilot experiment. To demonstrate the robustness of our method, we don't 't tune w according to the result and fix w = in the simulation, which corresponds to moderate constraint comparability. Empirically,
result isn't sensitive to the value of
We then discuss the latter issue. We bound all high-order terms by second or lower order terms. This effort leads to upper bound which allows us simply optimize the covariance matrix.
Proposition 3 (Variance Bound) The variance of the standard HT estimator has following upper bound,
Var[f] 8m2(w2 + 4) trace (dd" (Cov[t] + +
(14)
n²
where d is the vector (iiss d
In brief, we choose to bound the third-order term by the sum of second and fourth-order terms, and bound the fourth-order term by the definition of binary treatment vector. The format is adjusted for our optimization formulation later, where we choose the covariance matrix decision variables.
Actually, we can find that experiment design is summarized by the covariance matrix in this upper bound. Hence, denote this upper bound by (C.v it)). Notice that there is common multiplier ?2 which is associated with the unknown interference effect, in the squared bias and variance. This appealing feature allows to optimize the MSE upper bound without estimating the parameter of the potential outcome model.
Combining proposition and
derive following upper bound for MSE.

---

Theorem The MSE of the standard HT estimator has following upper bound
MSE - B(Covvtt) + V. (Cov(t))
(15)
3.2
Optimizing Covariance
Now we can discuss our experiment design. For a class of parameterized randomization schemes, one always analyze the covariance and express the bias and variance with parameters in randomization schemes. Based on the derivation in previous section, we argue that these randomization schemes is actually specifying the patterns of covariance matrix, e.g., the block diagonal pattern. We thus choose to directly optimize the covariance matrix of randomization scheme, which is actually considering a much larger hypothesis space.
Since our decision variable is a positive semi-definite matrix, it's intuitive to make attempts for formulating the problem as positive semi-definite programming with the MSE bound acting as objective. However, notice that there is still one square term with regard to trace(CCX), which appears in the ((X)². If we want to transform it into semi-definite programming (SDP), we must modify the objective further. Moreover, there are some intrinsic disadvantages can't 't directly sample from the multivariate Bernoulli
this formulation. Primarily,
after we solve the optimal covariance matrix, distribution that is subject to this covariance matrix. Secondly, method based on mathematical programming is usually precise but not efficient enough.
To guarantee that we can sample treatment vector subject to the optimized covariance, we first present a lemma that is equivalent to the Grothendieck's 's identity (28]
Lemma Suppose that (X, follows a bivariate Gaussian distribution with correlation r, namely,
~ N (
(16)
we have
Parosin(r)
Cov(sgn(X), sgn) sYn(
(17)
π
This lemma enables us to sample from bivariate Bernoulli distribution with mean (1) }) and any valid covariance. Specifically, once solve out the optimal covariance matrix X we can sample from multivariate Gaussian distribution with sin(2xX*) acting as covariance matrix, here sin function element-wise. Nevertheless, it must be noted that this transformation isn't 't omnipotent for case, namely, arcsin( X) 22t can"t 't recover all possible covariance matrix of n-variable multivariate Bernoulli distribution when X > O, since otherwise the multivariate Bernoulli distribution can be uniquely determined by its mean vector and covariance matrix, which is apparently false.
Moreover, can't guarantee sin sin(2mx**) to be positive definite. Motivated by this, we adopt the Cholesky-based parameterization. We set our decision variable matrix R, and guarantee the validity of the covariance matrix of Gaussian distribution first, namely, we set the covariance matrix of Gaussian distribution as RR and the covariance matrix of (Bernoulli) treatment vector is
even if X is positive semi-
X(R)= arcsin (RR*)
(18)
2T
Now we can propose our final formulation
min M(R) B(X(R)) + (X(()))
s.t. (RR") E [-1,1] Wi i [K]
(19)
(RR"),:11
Wi [K]
This optimization problem is no longer convex with the introduction of arcsin function, for which we design projected gradient descent algorithm. Since it's non-convex optimization problem, adaptive optimizer, such as Adam 20 can be applied to improve the performance of the gradient descent step.
7

---

Algorithm Optimizing Covariance Matrix
Input: g. G 1k .N. w
1: Initialization: Ro, i 4-
2: repeat
3: Calculate objective (R)
4: R Adam R. Vn
5: Update R with normalization (2-norm)
6:
i-i+1
7:
until N
Output: Optimized matrix H
We would like to draw attention to the fact that row normalization can be viewed as a projection operator the domain in the aforementioned formulation, with the Frobenius norm of the matrix. This fact can be verified through the Cauchy-Schward inequality. Additionally, we actually optimize a total of K parameters using gradient information, which is both efficient and easy implement.
After the optimization problem is solved, we can sample from it directly through a reparameterization of the multivariate Gaussian distribution,
1 + sgn (O, Ix)
t
(20)
2
4 Simulation Study
4.1 Basic Setting
In this part, we carry out simulation based on a linear/multiplicative potential outcome model on social network FB-Stanford3 32
We consider following linear potential outcome model,
dj + t.. e +
EjEN, dj
(2(2) = a 8 2
(21)
and multiplicative model, which is simplified version of that in (36), with removing covariate.
Yx()) = ta c.) d (1+ Bzg + Ejen 2j
(22)
di
We choose to fix all parameters except for interference density, 7. Namely, set (a, 3,c,o) (1, 1, 0.5, 0.1) for both models, and set y E {0.5, 1, to construct three regimes, e, N 1).
It should be noted that these two models correspond different model misspecification forms, aiming to demonstrate the robustness of our method.
Besides our optimized covariance design (OCD), we implement following randomization schemes. We first consider two baselines, independent Bernoulli randomization (Ber) and complete randomiza- tion (CR), both of which are cluster-level. We also implement two adaptive schemes, rerandomized- adaptive randomization (ReAR) and pairwise-sequential randomization (PSR) 25 31
which balance heuristic covariates adaptively and act as competitive baselines, since the average degree is considered as covariate these two methods and exactly appear in both two of our models, explicitly. Then we implement the independent block randomization (IBR) [5], which represents methods that optimize parameterized randomization schemes. Nevertheless, we argue that an appropriate objective for optimization is crucial, and we propose randomization scheme based on IBR with optimizing variance in equation (2) as objective. We nominate it as IBR-p since the heuristic solution this scheme in optimization problem is pairing cluster with cluster size, namely, each block is of size Notice that all randomization schemes satisfy equation (4).
For every randomization scheme we consider, we perform Monte Carlo simulation that repeats randomization schemes 10,000 times and calculates the sample mean and variance of estimators.
Network topology data can be found in httttp:://aeeeerrrrppsttto...r cww.s/t/o.cccccococtoottoooc -ateeeffdd-aaaarrr3ra php

---

Table 1: The average bias, standard deviation and MSE of HT estimator under linear model
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0,293
0.521
0.358
-0.588
0.584
0.688
-1.178
0.709
1.893
CR
-0.292
0.409
0.253
-0.586
0.459
0.554
-1.177
0.562
1.702
ReAR
-0.393
0.227
0.206
-0.700
0.251
0.554
-1.317
0.303
1.829
PSR
-0.295
0.235
0.143
-0.587
0.264
0.415
-1.179
0.323
1.496
IBR
-0,298
0.273
0.164
-0.593
0.308
0.447
-1.181
0.380
1.541
IBR-p
-0.294
0.232
0.141
-0.596
0.261
0.423
-1.185
0.318
1.507
OCD
-0.198
0.411
0.209
-0.388
0.469
0.371
-0.764
0.585
0.926
Table 2: The average bias, standard deviation and MSE of HT estimator under multiplicative model
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD>
MSE
method
Ber
-0.365
0.348
0.255
-0.736
0.394
0.698
-1,475
0.493
2.421
CR
-0.368
0.235
0.191
-0.744
0.274
0.629
-1.477
0.336
2.297
ReAR
-0,402
0.178
0.194
-0.809
0.174
0.685
-1.548
0.226
2.450
PSR
-0.366
0.134
0.152
-0.738
0.153
0.569
-1.479
0.192
2.227
IBR
-0.369
0.155
0.161
-0.737
0.178
0.576
-1.484
0.221
2.252
IBR-P
-0.368
0.163
0.163
-0.739
0.185
0.581
-.482
0.232
2.252
OCD
-0.258
0,040
0.069
-0.517
0.050
0.271
-1033
0.054
1.773
Moreover, the oracular GATE are utilized to calculate bias.
d (a + 3 d 7) (a g) +
(23)
72 a + -
At last, set the clusters given by Louvain algorithm with fixed random seed and resolution parameter as 10, which gives K = 192 clusters. We also consider other two resolution levels, namely, 2 and 5, corresponding K 30, 95, respectively. The outcome under these two levels is presented in the appendix.
4.2 Analysis of Results
From Table[to table we show the bias, standard deviation, and MSE of standard HT estimator. We also present the result under different estimator (such as difference-in- means estimator). different clustering resolution and different dataset (network topology) in the appendix due to limited space. Overall, meets our expectation that the proposed method OCD demonstrates greater advantages over other methods when interference intensity, namely, 7, increases. We emphasize the outstanding performance of our method the multiplicative model and give the credit handling the bias- variance tradeoff better. We also discover that our approach OCD isn't sensitive to the resolution level of upstream clustering, which isn't often the case for other methods, according to the thorough simulation results presented in the appendix. In reality, when there are certain densely connected cluster pairings that are assumed to have merged but have not, the clustering outcome is poor and resolution level may be inappropriate. Such pair would typically receive high positive covariance from the optimization process in OCD, which could be regarded as adaptively fixing the fault.
In addition, we highlight that the bias usually dominates in MSE in many settings, and it almost scales linearly with interference intensity, which is exactly as shown the expression of bias. According to our theoretical derivation, we attribute the dominance of bias to the density of social network. There a trend that other methods perform similarly with two baselines, since bias is dominating in denser network, while this component is usually ignored in the experiment design, on account of the introduction of HT estimator with exposure indicator. Moreover, although we fix w in presented result, we actually observe that our method is very robust to the choice of in related experiment.

---

Finally, we point out that the distribution of proportion of treated units of optimized randomization scheme is usually bimodal (e.g. peak 45%, 55%) in our randomization scheme, with the mean still 50%. This bimodal pattern is the outcome of introducing non-7ero covariance on almost all off-diagonal positions of covariance matrix. This is different from other methods we considered here, for which the proportion is usually strictly 50% or unimodal and concentrates at its mean, 50%.
5 Discussion
In this article, we derive the bias and variance of the standard HT estimator under parametric potential outcome model and construct optimizable objective based the derived MSE. We then propose an appropriate formulation to transform the experiment design into an optimization problem with the covariance matrix of treatment vector acting as decision variables. Besides, we propose an algorithm based on projected gradient descent to provide optimized covariance design that supports legitimate sampling. Finally, we implement a series of methods and compare their performance in diverse settings, where the advantage of our methods is demonstrated.
We stress the significance of bias and propose our method to make attempts for balancing bias and variance better. Actually, almost all potential outcome models used in the simulation part of related works admit neighborhood interference assumption, which makes the estimator using exposure indicator unbiased or nearly unbiased. However, we argue that this assumption is reasonable but also restricts our scope. There have recently been some works considering interference beyond this assumption, where bias is also demonstrated to be important.
Although there has been lot of variance bound in this area, few of them can guide experiment design directly, and we have actually made many attempts on finding appropriate variance bound. For exam- plc, the gradient of fourth-order term [t" Ct) "] can be estimated directly through REINFORCE estimator 388 without need of upper bound. Nevertheless, the optimized covariance matrix tends to give proportion of densely linked cluster pairs high correlation (near to 1), causing the covariance matrix to become nearly singular. This, in turn, leads to catastrophic gradient variance that can't 't be reduced by conventional technique such as (2I). In addition, weaker assumption on comparability between direct effect and interference feasible. We can only demand there exists w' > such that ||hll2 I w' This weaker assumption corresponds to spectral norm bound on the second order term h² """[tt") Jh. which is looser bound and finally produces a empirically worse solution. This bound also utilized in [17] and is expressed as the worst-case variance, which is concerned with maximizing convex objective over bounded potential outcomes. The optimal value is taken when potential outcome is taken as the endpoint value of its domain, which is too conservative to have good empirical performance. The simulation result of IBR support this claim, since similar worst-case variance is utilized in IBR.
Additionally, our work can be improved and extended in several aspects. Firstly, the optimization objective can be improved with a tighter but still optimizable upper bound on variance. Moreover, the situation of non-balanced treatment assignment, i.e., Bjz]] + 1. is worthy of exploration. Besides, the derivation for more complex estimator is kept explorable, e.g., the difference-in-meant estimator, and HT estimator with exposure indicator. Our derivation is performed on the standard HT estimator, and structural assumption is introduced when the derivation encounters essential difficulties. In addition, the basic setting beyond neighborhood interference assumption also warrant further extension.
Acknowledgments and Disclosure of Funding
The research was supported by the National Natural Science Foundation of China (No. Noo72217111., 72133002); the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grants A and 2020AAA0108400. We would like to thank Ruizhong Qiu for beneficial discussion, and anonymous reviewers for the helpful feedback.
References
[1] Peter M Aronow and Cyrus Samii. Estimating average causal effects under general interference, with application to social network experiment. Annals of Applied Statistics, 11(4):192-11947, 2017.
10

---

[2] Guillaume W Basse and Edoardo M Airoldi. Model-assisted design of experiments in the presence of network-correlated outcomes. Biometrika, 105(():849-855 2018.
[3] Rohit Bhattacharya, Daniel Malinsky, and llya Shpitser. Causal inference under interference and network uncertainty. In Uncertainty in Artificial Intelligence, pages 1028-1038, PMLR, 2020.
[4] Vincent D Blondel, Lean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10L)
2008.
[5] Ozan Candogan, Chen Chen, and Rad Niazadeh. Correlated cluster-based randomized experi- ments: Robust variance minimization, Management Science, 2023.
[6] Alex Chin. Regression adjustments for estimating the global treatment effect in experiments with interference. Journal of Causal Inference, 7(2), 2019.
[7] Andrew David Cliff and J Keith Ord. Spatial processes: models & applications. Taylor & Francis, 1981.
[8] Mayleen Cortez, Matthew Eichhorn, and Christina Yu. Staggered rollout designs enable causal inference under interference without network knowledge. In Advances in Neural Information Processing Systems, 2022.
[9] Mayleen Cortez, Matthew Eichhorn, and Christina Lee Yu. Exploiting neighborhood interfer- ence with low order interactions under unit randomized design. arXiv preprint XXix:2208.0055:3 2022.
[10] Dean Eckles, Brian Karrer, and Johan Ugander. Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference, 5(1):2015002,, 2016.
[11] Laura Forastiere, Edoardo M Airoldi, and Fabrizia Mealli, Identification and estimation of treatment and interference effects in observational studies networks. Journal of the American Statistical Association, 116653))9901-9 2021.
[12] Laura Forastiere, Fabrizia Mealli, Albert Wu, and Edoardo M Airoldi. Estimating causal effects under network interference with bayesian generalized propensity scores. Journal of Machine Learning Research, 23(289):1 61, 2022.
[13] Peter W Glynn, Ramesh Johari, and Mohammad Rasouli. Adaptive experimental design with temporal interference: A maximum likelihood approach. Advances in Neural Information Processing 3 33:15054-15064
2020.
[14] Huan Gui, Ya Xu, Anmol Bhasin, and Jiawei Han. Network a/b testing: From sampling estimation. In 24th International Conference on World Wide Web, WWW 2015, pages 399-409. Association for Computing Machinery, Inc, 2015.
[15] Kevin Han and Johan Ugander. Model-based regression adjustment with model-free covariates for network interference. arXiv preprint arXiv:2302.044: 049999990999 2023.
[16] Christopher Harshaw, Fredrik Sävje, David Eisenstat, Vahab Mirrokni, and Jean Pouget-Abadie. Design and analysis of bipartite experiments under a linear exposure-response model. In Proceedings of the 23rd ACM Conference Economics and Compulation, pages 606-606, 2022.
[17] Christopher Harshaw, Fredrik Sävje, Daniel Spielman, and Peng Zhang. Balancing covariates in randomized experiments with the gram-schmidt walk design. arXiv preprint arXiv:1911.03071,
2019.
[18] Christopher Harshaw, Fredrik Sävje, and Yitan Wang. A dasign-based riesz representation framework for randomized experiments. arXiv preprint 0rXiv:2210.08665 2022.
[19] Michael G Hudgens and Elizabeth Halloran. Toward causal inference with interference. Journal of the American Statistical Association, 103(482):832-842, 2008.

---

[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[21] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get baseline for free! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6. 2019. OpenReview.net, 2019.
[22] Michael P Leung. Causal inference under approximate neighborhood interference. Economet- rica, 90(1):267-295, 2022.
[23] Michael P Leung. Rate-optimal gluster-randomizee designs for spatial interference. The Annals of Statistics, 5065):30443087,, 2022.
[24] Hannah Li, Geng Zhao, Ramesh Johari, and Gabriel Y Weintraub. Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. In Proceedings of the ACM Web Conference 2022, pages 182-192, 2022.
[25] Yang Liu, Yifan Zhou, Ping Li, and Feifang Hu. Adaptive a/b test on networks with cluster structures. In International Conference on Artificial Intelligence and Statistics, pages 10836- 10851, PMLR, 2022.
[26] Joel Middleton and Peter M Aronow. Unbiased estimation of the average treatment effect in cluster-randomizzd experiments. Statistics, Politics and Policy, 6(1-2):39-77) 2015.
[27] Preetam Nandy, Kinjal Basu, Shaunak Chatterjee, and Ye Tu. A/b testing in dense large- scale networks: design and inference. Advances Neural Information Processing Systems, 33:2870-2880, 2020.
[28] Frank Oertel. Grothendieck's 's inequality and completely correlation preserving functions-a a summary of recent results and an indication of related research problems. arXiv preprint arXiv:2010.00746, 2020.
[29] Ben M Parker, Steven G Gilmour, and John Schormans. Optimal design of experiments on connected units with application to social networks. Journal of the Royal Statistical Society. Series C (Applied Statistics), pages 455-480, 2017.
[30] Jean Fouget-Abadie Kevin Aydin, Warren Schudy, Kay Brodersen, and Vahab Mirrokni. Variance reduction in bipartite experiments through correlation clustering. Advances in Neural Information Processing Systems, 32, 2019.
[31] Yichen Qin, Yang Li, Wei Ma, and Feifang Hu. Pairwise sequential randomization and its properties. arXiv preprint arXiv:1611, 022002022222220 2016.
[32] Ryan A. Rossi and Nesreen K. Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI, 2015.
[33] Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5);688, 1974.
[34] Michael E Sobel. What do randomized studies of housing mobility demonstrate? causal infer- ence in the face of interference. Journal of the American Statistical Association, 10166))) 1407, 2006.
[35] Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg, Graph cluster randomization: Network exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329-337, 2013.
[36] Johan Ugander and Hao Yin. Randomized graph cluster randomization. Journal of Causal Inference, 11(1):0222011, 2023.
[37] Davide Viviano. Experimental design under network interference. N0T::200030842211 2020.
arXiv preprint
[38] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Reinforcement learning, pages 5-32, 1992.
12

---

[39] Christina Lee Yu, Edoardo M Airoldi. Christian Borgs, and Jennifer T Chayes. Estimating the total treatment effect randomized experiments with unknown network structure. Proceedings of the National Academy of Sciences, 119(44))22288775119 2022.
[40] Christina Lee Yu, Edoardo M Airoldi, Christian Borgs, and Jennifer T Chayes. Graph agnostic randomized experimental design. arXiv preprint arXiv:2205. 12803, 2022.
[41] Yuan Yuan and Kristen M Altenburger. A two-part machine learning approach to characterizing network interference in a/b testing, 2022.
[42] Qiong Zhang and Lulu Kang. Locally optimal design for a/b tests in the presence of covariates and network dependence. Technometrics, 64(3):358-369) 2022.
[43] Corwin M Zigler and Georgia Papadogeorgou. Bipartite causal inference with interference. Statistical science: a review journal of the Institute of Mathematical Statistics, 36(1): 1009, 2021.
13

---

A Proofs
A.1 Proof of Proposition 1
Firstly, we notice that
=
Var[z]] = Elei]
E[2;]² =
(24)
2
Then we substitute (z) in HT estimator
2j
(1- 24) E
Yi(2)
(25)
n
E
[2
[1 - 2/]
:E[n]
with our potential outcome model and take expectation w.r.t. treatment assignments,
E[A]
B[(2 - -1-2100 21)))))((==)]
72
n
a,E|22, - 1] B
[[[22, 1)2rl
71
jEN;
(26)
Σ
[,[[2]] 2y Cov(zis
Σ zj]
71
JEN.
Σ
B + 4y Covizi,
Σ
1=1
JEN:
Moreover, we can calculate the GATE under our potential outcome model
(V((1) Y((O)) = - Σ (3: + 7d:)
(27)
Hence, the bias of HT estimator is
E
47 Cov Σ 2Y1 7d0
12
JEN.
(28)
Covlzis EjEN, 23]
Y
di
n
Var(z)
Then derive the matrix form with cluster-level treatment vector t.
Cov(z, ΣvEN, 2n] Var[2:]
7
di
1=1
7
4
Covlt,, tylC - Σ dj
{
sEin]
(29)
4 Σ Covlti, Cilp Σ Cij
[]E[K]
[JE(K]
4 tracee(C (Confe()
Σ Ci
[JE[K]
14

---

A.2 Proof of Proposition 2
Firstly we rewrite the estimator
f
Σ
(3: (B:-nda) (dx=
+ +2 Σ 212g
sEin
JEN:
(30)
Σ
(3: - >d:) 2
Σ
2jth
(.A)EE
We then expand the variance directly, and get
4
Var[7]
Σ
(3:
-d:) - yd3) Cov(za) 2s]
n²
LjE(n]
+
Σ
Σ
4r yd:)) Cov 2]]
(31)
x (AA)EE
Σ
Σ
47 Cov(zizy, zx)l
(M.)) (A,1)E
then we write it as matrix form with cluster-level treatment vector t.
x= 2
(h 2 Ct)
(32)
n
and expand the variance as covariance, we get the desired format
Var (1] n (trace(hh" Cov [t)) + 4y Cov(h' t, t CC))
(33)
+ 472 Varlt"Ct)
A.3 Proof of Proposition 3
Firstly we consider the estimator of matrix form again
2 (h L 2T("(1) Ct)
(34)
n
Since our assumption can't guarantce [[f] > always hold, we drop the square of expectation term, namely
Var(f) = [[ - B[+} < E[+*]
(35)
Notice that all elements of matrix F(4²' is nen-negative, we have
h ATEett" h < " B[t"" w2 (racoddd"E(tt") Similarly, since all elements of matrix C, namely, Cij also non-negative, we have
(36)
Bi( Ct) = (race(E CCt Ctt))
< trace CC1 trace trace(C11' )
(37)
In summary, we have
Var < [
< n²
(h " (x" I))
(38)
8n2
n² (2'traco(ddd B(1 + trace (rrace(C11CCR11
By definition, we have
C1 = d
(39)
15

---

Thus the upper bound above is actually
(w + 44
Var[f] S
(Irace(dd (dd B(((())) (((c)))
(40)
n²
At last, plug in following equation.
El(e1] = Cov(t] + 1 117
+
(41)
A.4 Proof of Lemma
Here we provide an intuitive proof. Utilizing Box-Muller transformation or pure algebraic analysis are also feasible.
We consider and Y generated from multivariate Gaussian distribution,
X (z. g)
Y
(y-g)
(42)
where g (n) and x, y two n-dim real vectors. Then we know that
Cov[X,Y] (z, v)
(43)
Moreover, we have
[X]] E(sgn(Y)) =0
(44)
thus
Cov[sgn(X), sgn. (Y) E[sgn sgn(Y)
(45)
Then we think geometrically that sgn( (x, g)) sgn (v. 9)) > 0 holds iff. lies above or below both of the hyperplanes that is orthogonal to æ and y respectively.
Notice that the direction of g uniform, it follows that
(Psgnn(ee g)) sgu( g)) > 0) _ (- - (rr-arcoos(2,))) v))
(46)
2x
Now we put things together
Elsgn(X X) sgn(Y))
= Pssgn( X) sgn (Y) > 0) - P(sgn( (X) (X)gg((Y) < 0)
2P(ssnn (X) sgn(gg (Y) 0) 1
= 2( (x - arccos (z. v\))
(47)
2
arccos (z, 8)
2
arcsin (x. y\)
which gives the desired outcome.
B
Simulation Details
B.1 Methods
consider following linear potential outcome model,
dj Yx(2) = + 3: c d to.ee + Ejen, Zj
(48)
dj
and multiplicative model, which is simplified version of that in (36), with removing a covariate.
dg (1 Bzg +
ΣjEN
{(()) (a ta
(49)
d,
16

---

We choose to fix all parameters except for interference density, Namely, we set (a, 3,c,0) = (1, 0.5, 0.1) for both models, and set y E {0.5, 1, to construct three regimes. e; -V(0,))) 1).
We set the clusters as given by Louvain algorithm with fixed random seed and resolution parameter 2,5, 10.
We consider social network TB-SStaford33 and FB-Comeell5 32 provide the network topology, and we generate potential outcome for each unit with mentioned potential outcome models.
These two social networks
Besides our optimized covariance design (OCD), we implement following randomization schemes. We first consider two baselines, independent Bernoulli randomization (Ber) and complete randomiza- tion (CR), both of which are cluster-level. We also implement two adaptive schemes, rerandomized- adaptive randomization (ReAR) and pairwise-sequentin randomization (PSR) 25 31, which balance heuristic covariates adaptively and act as competitive baselines, since the average degree is considered as covariate these two methods and exactly appear in both two of our models, explicitly. Then we implement the independent block randomization (IBR) [5] and heuristic version IBR-p that creates blocks with size 2.
In the methods mentioned above, ReAR is the only one concerned with hyperparameters setting. We set (g. B, a) (0.85, 400, 0.1). which corresponds the recommendation in the original paper. We estimate GATE with standard HT estimator and difference- in-means (DIM) estimator, where the latter refers to
(1- Ejemm 2j Eje(n - 2ÿ)
ÈDIM=
Yi(z)
(50)
iE(m)
To summarize, we provide the bias, standard deviation and MSE of two estimators under two potential outcome models, three ? levels and two datasets. All of these three metrics calculated by repetition of Monte Carlo simulation randomization, 10,000 times. In the main paper we 'ee presented the results of HT estimator IB-SStaford3, and we'll present the detailed results in this section.
B.2 Discussion on Estimators
Here we also provide discussion on another popular estimator that's considered existing literature, which is the HT estimator with exposure indicator. We consider the exposure condition that's 's fully treated or fully controlled in 1-hop neighborhood here, and the corresponding indicator is defined as 5z(z)) [[EEN, žj d. The estimator
5,(0) ({(2) (1) (O)
61(1)
Σ 8EIm]
(51)
We don't consider this estimator not only because of it's high variance resulted from low effective sample size, and but also its high calculation cost. For calculating need to estimate (1)], the probability of fully treated, for each unit é. We denote the number of clusters a unit connects to by c, For a unit in the exterior of its cluster, namely, G > 1, such a quantity can very high in a social network, which is also decided by the resolution of clustering, present an instance in figure T
our repeated simulations, we
Unfortunately, we must estimate such generalized propensity score [10 by Monte Carlo simulation for most of randomization schemes, even if it's just little bit more complex than independent Bernoulli randomization, where such quantity can be calculated directly, (1/2) For every simulation, we should visit every node and query the treatment assignment of its 1-hop neighborhood, whose time complexity is El), Roughly, we need 2 repetitions of randomization to achieve effective estimation on 6, (1), (0), which is too time-consuming be acceptable for many units.
Network topology data can be found in hwtpdn/nttt///etrrrropoooooooroo http:://aetwokrrppsittyy http://pppp.poocoo..ccoooooooora php
Strrfff--ttaaaaa ord3. .hpp
17

---

1750
Nodka Count
1250
1000
500
2.50
Number of Connected Clusters
Figure 1: Distribution of c; with 95 clusters FB- 33-Sttnford3
We stress the computational issue here since the resulted variance can be reduced by self- normalization, which corresponds to the Hájek estimator.
6,01)
8,(0)
Yi(z)
(52)
6
1,
RE|m]
However, we argue that the computational cost can't be bypassed without further modification and restricts the practicality of such HT estimator with exposure indicator. (25) proposes the so-called cluster-adjusted estimator that assign the units on the exterior of same cluster the same modified propensity score, which reduces the computation complexity at the cost of unpredictable distortion of estimator, which can be viewed as heuristics.
Therefore, we choose to consider the standard HT estimator and DIM estimator in our simulation.
B.3 Detailed Simulation Results
We present the results according to following sequence: HT-DIM estimator, iinear-mulipllccttiv model, 2-5-10 10 clustering resolution. In every table, we report the average bias, standard deviation (SD) and MSE of randomization schemes with three y levels. There are 24 tables total, which organized to demonstrate the advantage and robustness of our proposed method, optimized covariance design (OCD).
18

---

Table Simulation results of DIM estimator under linear model with resolution 2 FB-Stanford3
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.247
0.121
0.076
-0.497
0.122
0.262
-0.988
0.131
0.995
CR
-0.247
0.120
0.075
-0.494
0.120
0.259
-0.990
0.125
0.996
ReAR
-0.248
0.063
0.066
-0.482
0.065
0.237
-0.958
0.062
0.923
PSR
-0.243
0.059
0.063
-0.486
0.059
0.240
-0.974
0.062
0.954
IBR
-0.248
0.107
0.073
-0.493
0.109
0.255
-0.985
0.112
0.985
IBR-P
-0.244
0.090
0.068
-0.490
0.092
0.249
-0.980
0.094
0.969
OCD
-0.195
0.067
0.043
-0.388
0.068
0.156
-0.776
0.072
0.608
Table 4: Simulation results HT estimator under linear model with resolution 2 on FB-Stanford3
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.228
1.028
1.110
-0.489
1.148
1.558
-0.966
1.370
2.810
CR
-0.252
0.635
0.467
-0.494
0.697
0.730
-0.977
0.836
1.654
ReAR
-0.236
0.190
0.092
-0.486
0.173
0.267
-0.940
0.173
0.914
PSR
-0.241
0,233
0.113
-0.481
0.253
0.296
-0.972
0.297
1.033
IBR
-0.247
0.226
0.112
-0.494
0.252
0.308
-0.987
0.302
1.067
IBR-P
-0.246
0.154
0.085
-0.489
0.168
0.268
-0.981
0.203
1.006
OCD
-0.189
0.219
0.084
-0.385
0.252
0.212
-0.775
0.312
0.699
Table 5: Simulation results of DIM estimator under multiplicative model with resolution 2 on FB-Stanford3
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0.294
0.419
0.263
-0.603
0.479
0.593
-1.209
0.605
1.830
CR
-0,291
0.414
0.257
-0.607
0.473
0.593
-1.210
0.593
1.818
ReAR
-0.324
0.209
0.149
-0.573
0.237
0.386
-1.274
0.317
1.725
PSR
-0.307
0.208
0.138
-0.619
0.236
0.440
-1.236
0.297
1.618
IBR
-0.305
0.374
0.234
-0.605
0.427
0.549
-1.220
0.536
1.778
IBR-p
-0.304
0.315
0.192
-0.611
0.366
0.508
-1.214
0.457
1.684
OCD
-0.250
0.232
0.117
-0.507
0.265
0.328
-1.006
0.333
1.123
Table Simulation results of HT estimator under multiplicative model with resolution 2 on FB- Stanford3
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0,290
0,811
0.742
-0.578
0.923
1.187
-1.176
1.144
2.694
CR
-0.301
0.519
0.361
-0.605
0.582
0.705
-1.213
0.739
2.019
ReAR
-0.327
0.257
0.173
-0.547
0.259
0.367
-1.236
0.374
1.668
PSR
-0.303
0.322
0.196
-0.617
0.368
0.517
-1.230
0.461
1.727
IBR
-0.307
0.341
0.211
-0.611
0.384
0.521
-1.224
0.486
1.736
IBR-p
-0.307
0.294
0.181
-0.614
0.342
0.494
-1.218
0.427
1.666
OCD
-0.255
0.078
0.071
-0.510
0.089
0.268
-1.018
0.116
1.052
19

---

Table 7: Simulation results of DIM estimator under linear model with resolution 5 FB-Stanford3
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.277
0.096
0.086
-0.558
0.097
0.322
-1.116
0.101
1.257
CR
-0.279
0.095
0.087
-0.556
0.095
0.319
-1.117
0.097
1.259
ReAR
-0.319
0.030
0.103
-0.575
0.048
0.334
-1.143
0.045
1.311
PSR
-0.275
0.055
0.079
-0.554
0.056
0.311
-1.110
0.056
1.236
IBR
-0.276
0.088
0.085
-0.555
0.089
0.316
-1.110
0.090
1.241
IBR-P
-0.209
0.074
0.049
-0.490
0.074
0.246
-1.049
0.074
1.106
OCD
-0.194
0.102
0.048
-0.390
0.102
0.163
-0.782
0.104
0.623
Table 8: Simulation results HT estimator under linear model with resolution 5 on FB-Stanfor33
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.276
0.666
0.521
-0.535
0.749
0.847
-1.078
0.910
1.991
CR
-0.316
0.488
0.338
-0.610
0.540
0.664
-1.159
0.664
1.786
ReAR
-0.063
0.084
0.011
-0.410
0.219
0.216
-0.873
0.215
0.809
PSR
-0.272
0.222
0.123
-0.544
0.250
0.359
-1.108
0.310
1.326
IBR
-0.272
0.276
0.150
-0.553
0.313
0.404
-1.103
0.390
1.370
IBR-P
-0.694
0.064
0.486
-1.025
0.074
1.057
-1.688
0.097
2.861
OCD
-0.198
0.386
0.189
-0.391
0.439
0.346
-0.772
0.547
0.896
Table 9: Simulation results of DIM estimator under multiplicative model with resolution 5 on FB-Stanford3
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0.338
0.333
0.226
-0.687
0.387
0.622
-1.382
0,480
2.142
CR
-0.340
0.333
0.226
-0.680
0.387
0.613
-1.382
0.477
2.138
ReAR
-0.417
0.171
0.204
-0.860
0.163
0.767
-1.485
0.228
2.260
PSR
-0.336
0.192
0.150
-0.680
0.219
0.511
-1.375
0.277
1.968
IBR
-0.344
0.310
0.214
-0.680
0.351
0.586
-1.379
0.439
2.097
IBR-P
-0.098
0,272
0.084
-0.407
0.312
0.263
-1.025
0.392
1.205
OCD
-0.241
0.357
0.185
-0.497
0.407
0.413
-0.994
0.511
1.249
Table 10: Simulation results of HT estimator under multiplicative model with resolution on FB-Stanford3
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0,340
0.463
0.330
-0.683
0.523
0.740
-1.366
0.656
2.297
CR
-0.383
0.289
0.230
-0.733
0.328
0.645
-1.430
0.412
2.215
ReAR
-0.289
0.138
0.102
-0.685
0.121
0.484
-1.288
0.230
1.712
PSR
-0.335
0.128
0.129
-0.684
0.146
0.490
-1.373
0.182
1.919
IBR
-0.346
0.180
0.153
-0.694
0.207
0.525
-1.387
0.259
1.992
IBR-p
-0.488
0.190
0.275
-0.852
0.217
0.774
-1.581
0.271
2.574
OCD
-0.255
0.049
0.068
-0.508
0.057
0.262
-1.020
0.069
1.045
20

---

Table 1I: Simulation results of DIM estimator under linear model with resolution 10 FB-Stanford3
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.298
0.076
0.095
-0.595
0.077
0.361
-1.191
0.080
1.426
CR
-0.297
0.076
0.095
-0.595
0.076
0.360
-1.191
0.078
1.427
ReAR
-0.296
0.022
0.088
-0.589
0,019
0.348
-1.191
0.029
1.422
PSR
-0.297
0.035
0.090
-0.596
0.036
0.357
-1.193
0.039
1.425
IBR
-0.296
0.054
0.091
-0.593
0.055
0.356
-1.188
0.058
1.415
IBR-p
-0.297
0.049
0.091
-0.593
0.049
0.354
-1.186
0.052
1.411
OCD
-0.190
0.115
0.050
-0.384
0.116
0.161
-0.772
0.118
0.611
Table 12: Simulation results of HT estimator under linear model with resolution 10 on FB-Stanford3
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.293
0.521
0.358
-0.588
0.584
0.688
-1.178
0.709
1.893
CR
-0.292
0.409
0.253
-0.586
0.459
0.554
-1.177
0.562
1.702
ReAR
-0.393
0.227
0.206
-0.700
0.251
0.554
-1.317
0.303
1.829
PSR
-0.295
0.235
0.143
-0.587
0.264
0.415
-1.179
0.323
1.496
IBR
-0.298
0.273
0.164
-0.593
0.308
0.447
-1.181
0.380
1.541
IBR-P -0.294
0.232
0.141
-0.596
0.261
0.423
-1.185
0.318
1.507
OCD
-0.198
0.411
0.209
-0.388
0.469
0.371
-0.764
0.585
0.926
Table 13: Simulation results of DIM estimator under multiplicative model with resolution 10 on FB-Stanford3
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0.369
0.264
0.206
-0.731
0.301
0.626
-1.475
0.375
2.318
CR
-0,363
0.265
0.202
-0.737
0.303
0.636
-1.470
0.380
2.307
ReAR
-0.351
0.061
0.127
-0.715
0.095
0.520
-1.480
0.121
2.207
PSR
-0.369
0.122
0.151
-0.738
0.141
0.565
-1.481
0.176
2.227
IBR
-0.367
0.190
0.171
-0.737
0.218
0.592
-1.776
0.271
2.254
IBR-p
-0.364
0.168
0.161
-0.742
0.192
0.588
-1.88
0.244
2.253
OCD
-0.247
0.402
0.223
-0.501
0.459
0.463
-1.013
0.577
1.360
Table 14: Simulation results HT estimator under multiplicative model with resolution on FB-Stanford3
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0,365
0.348
0.255
-0.736
0.394
0.698
-1.475
0.493
2.421
CR
-0.368
0.235
0.191
-0.744
0.274
0.629
-1.477
0.336
2.297
ReAR
-0.402
0.178
0.194
-0.809
0.174
0.685
-1.548
0.226
2.450
PSR
-0.366
0.134
0.152
-0.738
0.153
0.569
-1.479
0.192
2.227
IBR
-0.369
0.155
0.161
-0.737
0.178
0.576
-1.484
0.221
2.252
IBR-p
-0.368
0.163
0.163
-0.739
0.185
0.581
-1.482
0.232
2.252
OCD
-0.258
0.040
0.069
-0.517
0.050
0.271
-1.034
0.054
1.073
21

---

Table 15: Simulation results of DIM estimator under linear model with resolution on FB-CornelS5
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.240
0.092
0.066
-0,480
0.093
0.239
-0.960
0.103
0.933
CR
-0.238
0.090
0.065
-0.480
0.092
0.239
-0.958
0.100
0.929
ReAR
-0.239
0.019
0.058
-0.469
0.026
0.221
-0.925
0.032
0.858
PSR
-0.235
0.056
0.059
-0.470
0.056
0.225
-0.944
0.058
0.894
IBR
-0.235
0.091
0.064
-0.477
0.093
0.237
-0.950
0.096
0.913
IBR-P
-0.235
0.103
0.066
-0.471
0.104
0.234
-0.944
0.108
0.903
OCD
-0.188
0.059
0.039
-0.374
0.060
0.144
-0.751
0.070
0.569
Table 16: Simulation results of HT estimator under linear model with resolution 2 on FB-Comeel5
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.228
1.087
1.234
-0.456
1.240
1.746
-0.913
1.469
2.993
CR
-0.228
0.913
0.885
-0.440
1.018
1.230
-0.914
1.231
2.350
ReAR
-0.209
0.110
0.056
-0.469
0.128
0.237
-0.965
0.135
0.950
PSR
-0.235
0.146
0.077
-0.473
0.164
0.251
-0.940
0.202
0.926
IBR
-0.232
0.530
0.335
-0.465
0.595
0.571
-0.942
0.720
1.407
IBR-P
-0.233
0.180
0.087
-0.472
0.209
0.268
-0.939
0.263
0.951
OCD
-0.184
0.213
0.079
-0.376
0.241
0.200
-0.752
0.302
0.657
Table 17: Simulation results of DIM estimator under multiplicative model with resolution 2 on FB-Comel15
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0.260
0.318
0.169
-0.546
0.363
0,430
-1.075
0.461
1.369
CR
-0.261
0.317
0.169
-0.533
0.362
0.416
-1.080
0.457
1.375
ReAR
-0.252
0.064
0.068
-0.515
0.090
0.273
-1.073
0.135
1.172
PSR
-0.268
0.190
0.108
-0.536
0.223
0.338
-1.069
0.276
1.221
IBR
-0.268
0.319
0.174
-0.535
0.368
0.422
-10774
0.453
1.359
IBR-p
-0.258
0.362
0.198
-0.526
0.416
0.450
-1.550
0.514
1.367
OCD
-0.215
0.201
0.087
-0.428
0.228
0.235
-0.866
0.291
0.835
Table 18: Simulation results of HT estimator under multiplicative model with resolution on FB-Comel15
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0,260
0.853
0.795
-0.517
0.973
1.214
-1.020
1.213
2.514
CR
-0.265
0.700
0.561
-0.536
0.801
0.929
-1.053
1.011
2.133
ReAR
-0.267
0.057
0.075
-0.515
0.076
0.271
-1.073
0.106
1.164
PSR
-0.269
0.151
0.095
-0.539
0.176
0.322
-1.770
0.218
1.194
IBR
-0.264
0.419
0.246
-0.526
0.487
0.515
-1.074
0.614
1.531
IBR-p
-0.264
0.243
0.129
-0.534
0.277
0.362
-1.066
0.344
1.255
OCD
-0.218
0.059
0.051
-0.436
0.068
0.195
-0.873
0.085
0.770
22

---

Table 19: Simulation results of DIM estimator under linear model with resolution 5 on FB-CornelS5
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.281
0.070
0.084
-0.560
0.070
0.319
-1.124
0.074
1.269
CR
-0.280
0.069
0.084
-0.561
0.069
0.320
-1.123
0.072
1.268
ReAR
-0.282
0.048
0.082
-0.559
0.045
0.315
-1.113
0.035
1.241
PSR
-0.274
0.052
0.078
-0.552
0.053
0.308
-1.106
0.055
1.227
IBR
-0.280
0.069
0.083
-0.560
0.070
0.319
-1.120
0.073
1.261
IBR-p
-0.278
0.086
0.085
-0.558
0.086
0.319
-1.113
0.088
1.247
OCD
-0.196
0.120
0.053
-0.393
0.120
0.169
-0.784
0.125
0.631
Table 20: Simulation results of HT estimator under linear model with resolution 5 on FB-Comell5
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.268
0.676
0.529
-0.551
0.755
0.874
-1.106
0.918
2.067
CR
-0.279
0.558
0.389
-0.553
0.622
0.693
-1.092
0.748
1.753
ReAR
-0.305
0.376
0.234
-0.608
0.427
0.553
-1.243
0.516
1.812
PSR
-0.275
0.373
0.215
-0.547
0.419
0.475
-1.088
0.511
1.446
IBR
-0.275
0.409
0.243
-0.556
0.461
0.522
-1.100
0.557
1.521
IBR-P
-0.274
0.332
0.185
-0.550
0.377
0.445
-1.112
0.466
1.454
OCD
-0.194
0.433
0.225
-0.385
0.490
0.389
-0.781
0.616
0.991
Table 21: Simulation results of DIM estimator under multiplicative model with resolution 5 on FB-Comell5
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0.314
0.239
0,156
-0.634
0.276
0.479
-1.269
0.344
1.729
CR
-0.313
0.238
0.154
-0.630
0.273
0.472
-1.273
0.344
1.741
ReAR
-0.330
0.157
0.134
-0.623
0.164
0.416
-1.256
0.181
1.613
PSR
-0.309
0.180
0.128
-0.622
0.207
0.430
-1.252
0.262
1.636
IBR
-0.312
0.244
0.157
-0.635
0.277
0.481
-1.266
0.349
1.725
IBR-p
-0.305
0.298
0.182
-0.625
0.342
0.508
-1.251
0.427
1.747
OCD
-0.212
0.417
0.219
-0.428
0.477
0.412
-0.868
0.600
1.114
Table 22: Simulation results of HT estimator under multiplicative model with resolution on FB-Comel15
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0,302
0.496
0.338
-0.629
0.548
0.696
-1.245
0.690
2.027
CR
-0.320
0.375
0.243
-0.632
0.430
0.586
-1.262
0.539
1.885
ReAR
-0.310
0.230
0.150
-0.623
0.242
0.447
-1.308
0.360
1.841
PSR
-0.307
0.203
0.136
-0.618
0.233
0.436
-1252
0.290
1.654
IBR
-0.318
0.239
0.159
-0.632
0.277
0.477
-1.274
0.349
1.747
IBR-p
-0.315
0.148
0.121
-0.634
0.168
0.431
-1.265
0.211
1.647
OCD
-0.225
0.042
0.053
-0,451
0.050
0.206
-0.004
0.059
0.821
23

---

Table 23: Simulation results of DIM estimator under linear model with resolution 10 on FB-CornelS
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.312
0.038
0.099
-0.624
0.040
0.392
-1.249
0.044
1.563
CR
-0.312
0.039
0.099
-0.624
0.040
0.392
-1.249
0.043
1.563
ReAR
-0.308
0.031
0.096
-0.625
0.024
0.392
-1.253
0.029
1.572
PSR
-0.314
0.035
0.100
-0.625
0.035
0.393
-1.252
0.038
1.569
IBR
-0.312
0.036
0.099
-0.625
0.038
0.392
-1.250
0.041
1.566
IBR-P
-0.311
0.036
0.098
-0.623
0.036
0.390
-1.246
0.042
1.555
OCD
-0.188
0.098
0.045
-0.374
0.100
0.150
-0.749
0.106
0.574
Table 24: Simulation results HT estimator under linear model with resolution 10 on FB-Comel15
gamma metric method
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
Ber
-0.311
0.388
0.247
-0.621
0.437
0.576
-1.237
0.517
1.799
CR
-0.311
0.264
0.167
-0.623
0.290
0.473
-1.243
0.345
1.665
ReAR
-0.306
0.069
0.099
-0.618
0.087
0.390
-1.238
0.098
1.544
PSR
-0.310
0.067
0.101
-0.622
0.076
0.394
-1.245
0.094
1.560
IBR
-0.312
0.105
0.108
-0.627
0.116
0.407
-1.250
0.139
1.583
IBR-P
-0.311
0.065
0.102
-0.623
0.073
0.395
-1.246
0.090
1.562
OCD
-0.182
0.345
0.152
-0.375
0.397
0.299
-0.749
0.495
0.807
Table 25: Simulation results of DIM estimator under multiplicative model with resolution 10 on FB-Comell5
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0.354
0.136
0.144
-0.702
0.155
0.518
-1.404
0.195
2.012
CR
-0.352
0.138
0.143
-0.702
0.154
0.517
-1.407
0.194
2.019
ReAR
-0.354
0.077
0.131
-0.699
0.099
0.499
-1.394
0.142
1.964
PSR
-0.355
0.119
0.141
-0.708
0.136
0.520
-1.411
0.174
2.023
IBR
-0.352
0.126
0.141
-0.704
0.145
0.517
-1.411
0.182
2.025
IBR-P
-0.350
0.124
0.138
-0.703
0.142
0.516
-1.403
0.178
2.003
OCD
-0.215
0.343
0.164
-0.438
0.395
0.348
-0.878
0.490
1.013
Table 26: Simulation results HT estimator under multiplicative model with resolution on FB-Comel15
gamma metric
0.5
1.0
2.0
Bias
SD
MSE
Bias
SD
MSE
Bias
SD
MSE
method
Ber
-0,355
0.325
0.232
-0.698
0.370
0.625
-1.396
0.469
2.171
CR
-0.349
0.238
0.179
-0.704
0.266
0.567
-1.401
0.335
2.075
ReAR
-0.353
0.077
0.131
-0.708
0.090
0.511
-1.399
0.131
1.976
PSR
-0.353
0.101
0.135
-0.707
0.116
0.513
-1.408
0.147
2.005
IBR
-0.351
0.141
0.143
-0.703
0.162
0.521
-1.409
0.207
2.029
IBR-p
-0.349
0.107
0.134
-0.703
0.122
0.509
-1.404
0.154
1.997
OCD
-0.224
0.038
0.052
-0.448
0.041
0.203
-0.895
0.053
0.805


Reviews:
[{"review_id": "bUUIxr1B6i", "reviewer": "Reviewer_2fwK", "summary": "This paper presents an optimized covariance design for A/B tests on social networks with interference. The authors address the challenge of accurately estimating the global average treatment effect (GATE) in the presence of network interference. They propose a method to balance bias and variance in experimental design by optimizing the covariance matrix of the treatment assignment vector. The paper derives the bias and variance of the estimator and proposes an algorithm to implement the desired randomization scheme. Simulation studies demonstrate the advantages of their method over existing methods in various settings. Overall, the paper contributes an approach to improve the estimation of treatment effects in social network experiments by optimizing the covariance design.", "questions": "Please refer to weaknesses 1, 2, and 3", "limitations": "While the paper briefly mentions assumptions about the direct treatment effect and interference effect, a more thorough discussion on the validity and generalizability of these assumptions would be valuable. Additionally, it would be beneficial for the authors to include a discussion on the limitations and potential negative consequences of their work. This could include an examination of the proposed method’s sensitivity to certain parameter values or potential biases introduced by the assumptions. Providing a more comprehensive understanding of the method’s applicability and potential drawbacks would strengthen the overall manuscript", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents a approach to optimize the covariance design for A/B tests on social networks with interference, which improves the estimation of treatment effects and enables more accurate decision making based on experimental results. The paper is well-written and clearly explains the problem formulation, assumptions, and the proposed method. The authors provide a thorough analysis of the bias and variance of the estimator, a upper bound for the MSE of the HT estimator, and a projected gradient descent algorithm to solve the optimization problem. The simulation studies demonstrate the effectiveness and robustness of the proposed method in various settings. ", "weaknesses": "1.\tThe paper briefly mentions assumptions about the direct treatment effect and interference effect. However, a more thorough discussion on the validity and generalizability of these assumptions would be valuable. Additionally, discussing the limitations of the proposed method, such as its sensitivity to certain parameter values or potential biases introduced by the assumptions, would provide a more comprehensive understanding of the method's applicability and potential drawbacks.\n2.\tIn the experimental results presented in Tables 2, 5, 6, 7, 10, 11, 14, 15, 19, 22, and 23 in the appendix, it is observed that the proposed algorithm’s performance in terms of MSE is significantly weaker compared to other benchmark algorithms such as ReAR, IBR-p, and PSR. This suggests that further improvements may be necessary for the proposed algorithm to achieve competitive performance\n3.\tThe paper could benefit from a more detailed introduction and explanation of the motivation behind balancing bias and variance. Providing additional context and background information on this topic would help to strengthen the overall argument and improve the clarity of the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thanks for your time and efforts in the reviewing process, and we address your concerns as follows.\n\n## Comparability assumption\n\nThe comparability assumption is predicated on the premise that the magnitude of interference remains akin to the direct effect at the cluster level. This assertion inherently excludes scenarios where interference is significantly weaker compared to the direct effect—a stance that aligns with our targeted investigative scope. It's worth noting that our primary focus is on experiment design encompassing network interference. Consequently, this assumption finds its rational grounding in this context. Assumptions on the form and intensity of interference are very common in related works. \n\nIn the second place, as the degree of tolerance on incomparability between direct causal effect and interference, $\\omega$, escalates, the constraint imposed by this assumption gradually relaxes. We posit that $\\omega$ can be perceived as a tunable parameter, encapsulating the practitioner's belief in the degree of comparability. Importantly, the parameter $\\omega$ signifies the extent to which a unit's outcomes are contingent upon the activities of its associates, each with varying degrees of influence—a manifestation of real-world complexity. This inherent variability mirrors the natural heterogeneity observed in real-world scenarios. We also recognize and discuss the sensitivity of our results to $\\omega$ in line 219 due to limited space. Moreover, the simulation section delves into the sensitivity analysis of other parameters within the potential outcome model, further reinforcing the comprehensiveness of our investigation.\n\n\n## Clarification on non-competitive performance\n\nFirst and foremost, our randomization scheme OCD is tailored to the network experiment with interference comparable to direct causal effect. As such, we think it's acceptable that our scheme exhibit relatively diminished performance in scenarios with low interference levels ($\\gamma$ set to 0.5). Notably, our method consistently excels as interference intensifies—a pattern that aligns with our intended scope and objectives.\n\nSecondly, it's imperative to contextualize our simulation settings. Both potential outcome models employed mirror instances of model misspecification, presenting considerable challenges. Furthermore, our chosen settings inadvertently favor certain methods—such as considering node degree explicitly as a covariate in both PSR and ReAR, whereas this is  embedded in our potential outcome models in simulation.  Despite these challenges, our method consistently performs commendably across a multitude of settings. \n\nIt's worth highlighting that expecting a single experiment design to outperform all alternatives across diverse scenarios is a lofty anticipation. Each randomization scheme inherently thrives within specific contexts. For instance, when γ approaches zero, complete randomization (CR) can potentially lead across all methods.\n\nLastly, while performance enhancement is indeed a noteworthy facet of our contributions, it's pivotal to recognize that our method's significance extends beyond this. In practical terms, our approach demonstrates commendable efficiency and scalability in the domain of social networks on online platforms. From a theoretical perspective, the meticulous formulation of the problem and the creativity with which the optimization problem is constructed holds paramount importance.\n\n\n## Motivation for balancing bias-variance \n\nAs underscored in our introduction, a predominant focus within existing literature centers on variance reduction. However, we advocate for a paradigm shift, asserting that Mean Squared Error (MSE) serves as a more apt optimization objective. This pivot inherently embodies the equilibrium between bias and variance—a fundamental tenet of the bias-variance tradeoff.\n\nDelving into Section 2.3, we undertake a comprehensive analysis of the sources underpinning bias. Within the realm of social networks, bias can manifest substantially, serving as a pivotal driver for our decision to holistically address both bias and variance. This awareness underscores our commitment to embracing a balanced perspective, effectively navigating the intricate interplay between these critical elements. More intuitively, when both bias and variance are considerable, then we should not expect an unbiased estimator or constant estimator to be optimal."}, {"review_id": "03LHn9wvHw", "reviewer": "Reviewer_Q5Zi", "summary": "In this paper, the authors propose a new algorithm for A/B test design under network effect with cluster level randomization. By derivation of an upper MSE upper bound with bias-variance trade-off, the authors reparameterize to directly optimize the covariance of the treatment vector. An efficient PGD algorithm is proposed for efficient optimization. Simulation study on a real-world network demonstrates the effectiveness of the proposed method.", "questions": "Please see above", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors study a very important and practical problem in A/B test design with reasonable assumptions on cluster level randomization.\n2. The paper is well written and the proposed method is both explained clearly, with reasonable assumptions  and well justified with detailed derivations.\n3. The proposed reparameterization and the PGD optimization algorithm makes  the algorithm practical and easy to implement.\n", "weaknesses": "1. It seems that the clustering algorithm plays a very important role in the quality of the A/B experiment design. From the covariate matrix formulation, maybe in some cases it would be better to directly merge two clusters instead of using the treatment vector to guarantee the same treatment. It would be very interesting to see how the proposed method performs under different cluster methods /parameters besides the number of clusters.\n2. As A/B experiments are usually used in real large-scale networks, it would be interesting to see scalability results in terms of number of clusters for different methods under comparison.\n3. As the authors point out in the future work, it would also be interesting to see some empirical evaluation on the tightness of the proposed upper bound on the MSE. It may provide some insight on whether further tightening the bound can bring additional benefits.   \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thanks for your time and efforts in the reviewing process and we appreciate your approval of the theoretical and empirical contributions of this work. We provide some comments on your interesting questions as follows.\n\n## Clustering issue\n\nThe role of clustering indeed introduces intriguing complexities within the sphere of cluster-level design. We once contemplated an end-to-end approach that seamlessly integrated upstream clustering into the design pipeline. However, the intricate interplay between algorithm settings and the resultant clustering structure—particularly when dealing with heuristics-rich community detection algorithms like Louvain—posed formidable analytical challenges. Consequently, we opted to focus on presenting numerical outcomes achieved by varying the resolution parameters of the Louvain algorithm. Notably, the resolution stands as a paramount parameter within our context, aligning with the preferences of our industrial collaborator. It's worth mentioning that a more nuanced analysis might become feasible should cluster information be incorporated in a simple format within the experiment design. An example is the independent block randomization (IBR) that retains cluster information exclusively as cluster sizes [4], which could potentially facilitate an encompassing analysis inclusive of clustering step. As contrast, the cluster structure is involved in our derived bias and variance with matrix $C$, which characterizes the connections within/between clusters.\n\nFurthermore, the prospect of cluster mergers raises pertinent considerations. In this context, we find the optimized covariance matrix to be particularly enlightening. A correlation nearing 1 (equivalent to covariance near 1/4) signifies a distinct entwinement between clusters, akin to a merger from the perspective of treatment assignment.\n\n## Scalability\n\nOur formulation's key scalability advantage is rooted in its continuous optimization structure, while explicit time complexity analysis proves challenging. Therefore, we provide some comments here. CR and Ber are the most efficient and scalable methods disregarding their performance. IBR builds on dynamic programming that partitions clusters into communities and is relatively scalable. The PSR and ReAR are competitive in performance, while PSR relies on a sequential assignment process and is thus poor on scalability. ReAR relies on PSR and rerandomization, which is difficult to implement in the industrial scene. \n\n## Empirical evaluation on bound tightness\n\nWe appreciate the valuable suggestion to delve into the empirical evaluation of bound tightness. In response, we highlight two pivotal tradeoffs that inherently shape the intricacies of our analysis. The first tradeoff surfaces between the underlying assumption and the tightness of the bound. We illustrate this interplay in lines 340 to 343 by introducing an alternative bound under a weaker comparability assumption. While this bound appears naturally, empirical evidence underscores its excessive conservatism, thereby illuminating the challenge of striking the delicate balance between assumption strength and bound tightness. The second tradeoff stems from the equilibrium between optimization complexity and bound tightness, as meticulously discussed in lines 334 to 339. \n\nMoreover, we acknowledge the significance of equation 19 in the appendix—an integral component of our methodology. This step, $\\operatorname{trace}\\left(\\mathbb{E}\\left[\\boldsymbol{C} t t^T \\boldsymbol{C} t t^T\\right]\\right) \\leq \\operatorname{trace}\\left(\\mathbb{E}\\left[\\boldsymbol{C} \\mathbf{1 1}^T \\boldsymbol{C} t t^T\\right]\\right)$, intrinsically linked to the randomization scheme behind treatment vector $t$, is variably influenced by different experiment designs. This inherent variability poses challenges in deriving a uniform conclusion, thereby rendering the attainment of a definitive outcome a complex endeavor. Nevertheless, it's worthwhile for future extension."}, {"review_id": "TZ7vsQO6xY", "reviewer": "Reviewer_1xr3", "summary": "In this paper, authors present a novel experimental design to be used for randomized experiments under network interference.\nAuthors begin by considering a baseline adjusted Horvitz--Thompson estimator (which crucially requires knowledge of $Y_i(\\mathbf{0})$) and a pre-specified clusterings.\nGiven these two things, authors derive a bound on the mean squared error of the estimator under an arbitrary cluster design.\nMotivated by a Grothendeick identity, authors propose an experimental design where a normal vector is sampled from a fixed covariance matrix $\\Sigma$ and the signs of the vector become treatment assignment.\nAuthors propose to select an experimental design by (essentially) using the normal covariance matrix $\\Sigma$ as a decision variable and the upper bound on the mean squared error as an objective.\nFinally, simulations are run to investigate the performance of the method under various types of model-mispecification.", "questions": "1. One way to avoid \"picking clusters\" is to run the method when each unit is its own singleton cluster. How does the proposed method perform in this case? In other words, is the specification of clusters necessary to achieve small variance? If so, can you describe why or generally comment more on this? (this is a discussion I would find really fascinating in the paper)\n2. I claimed that the \"Grothendeick mapping\" cannot recover all covariance matrices on $\\pm 1$ matrices. Can you comment on this?\n3. Can you increase the Monte Carlo samples to 10,000?\n4. Can you comment on the comparison to the Gram--Schmidt Walk Design of Harshaw et al (2019)?\n", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is a novel method for designing randomized experiments under network interference. The novelty of the method comes from developing an objective function whose decision variables are continuous, (essentially) representing the covariance matrix of assignments. This stands in contrast to previous methods which focus on selecting clusters for independent cluster randomization designs. An additional strength of the paper is that the objective considers both bias and variance, which is uncommon: most approaches attempt to \"fix\" one of these two things, but typically not both.\n\nOne of the most exciting technical connections is the use of Grothendieck's identity, which (to the best of my knowledge) has not been used in causal inference. This offers a new tool in the design of experiments, which I suspect will be useful beyond the specific design used in this paper.\n\nMoreover, the simulations are very well formulated and executed.\nIn particular, the authors investigate the effectiveness of their design under various forms of model misspecification, which speaks to the robustness of the design.\nThis is important to investigate via simulations, as formal theory seems difficult given the black-box nature of the experimental design.", "weaknesses": "There are several weaknesses of the current method.\n\n1. **Assumed Knowledge**: The paper makes some strong assumptions as to what the experimenter knows. For example, authors assume that the coefficients $\\alpha_i$ are known by the experimenter. I think most experimenters will find that knowledge of each individual $\\alpha_i$ is too strong of an assumption to be practical -- in the SUTVA setting, this implies that all individual treatment effects can be estimated perfectly without any randomization. In order to be transparent, authors should state this assumption and estimator earlier in the paper, perhaps replacing the HT estimator in (eq 5).\n2. **Pre-specified Clusters**: The paper assumes that clusters are pre-specified and little advice is given to practicioners on how to select the clusters in order to minimize MSE.\n3. **Understanding of the Variance**: In order to run power calculations, experimenters should have at least a rough understanding (e.g. asymptotic rates) of how the variance of the experimental design depends on sample size $n$ and network parameters. Because this method is based on a black-box optimization procedure, it seems hard to analyze the variance (i.e. optimal value).\n4. **Confidence Intervals**: In practice, experimenters value interval estimators (i.e. confidence intervals) more than point estimators, as they provide a method for uncertainty quantification. The necessary tools for uncertainty quantification (e.g. Central Limit Theorem, variance estimation) are not presented in this paper.\n\nWhile authors should make certain minor changes to address the above, I think that these weaknesses actually constitute further research directions on this exciting method.\nOverall, it is my opinion that the strengths of the paper outweigh the weaknesses.\n\nIn the two sections below, I discuss minor weakenesses in the technical discussions and literature review which should be addressed by authors before publication.\n\n## Technical Remarks\n\nBelow are some minor remarks on technical aspects of the paper.\nI think there are a few technical issues in the discussions, but I believe these can be easily fixed and will strengthen the technical contribution of the paper.\n\n1. (Line 137) authors write \"without loss of generaltiy, we consider the balanced cluster-level randomization scheme satisfying...$E[z_i] = 1/2$.\" I think that this restriction is perfectly fine, but I would say it is not technically correct to describe it as \"without loss of generality\". Consider the usual SUTVA setting: if the outcomes under treatment have more variation than outcomes under control, then the Horvitz--Thompson estimator can be made to have smaller variance by setting $p$ so that treatment is assigned more frequently. I think the phrase \"without loss of generality\" is not warranted and a simple fix is to just remove it.\n2. (Line 152) The word \"overparametrized\" is not quite standard in this literature so I'd recommend either informally defining it, or just saying that \"there are more unknown potential outcomes than observations\". This is true even under SUTVA.\n3. (Line 208): Authors introduce what they call \"Assumption 2\". I would refer to this as a \"condition\" rather than an assumption. The reason is that Assumption 2 only plays a role in choosing the design -- using standard techniques (CLT + variance estimator), Assumption 2 would not be necessary for, say, the validity of confidence intervals. Some readers might misinterpret the use of the word \"Assumption\" to think \"if this condition does not hold, then the estimates are no longer statistically valid in some sense\".\n4. (Line 254) Authors write \"This lemma enables us to sample from bivariates Bernoulli distribution with mean (1/2, 1/2) and any valid covariance\". While this is true for $n=2$ variables, I do not believe it to be true for general $n$ variables. It is sort of implied in the Section that this \"Grotendeick mapping\" can recover any covariance matrix of $\\pm 1$ variables. If authors have a proof of this, they should provide it; otherwise, they should clearly state that this \"Groethendeick mapping\" cannot generate all $\\pm 1$ covariance matrices. I think that clarifying this will increase understanding and appreciation of the method.\n5. (Line 294): The Monte Carlo simulation is only performed 200 times. In my experience, this is quite low. Can you increase to 10,000 Monte Carlo runs before camera ready submission? This will increase the reader's confidence in you results.\n\n## Literature Review + References\n\nThe authors have missed a few important references in the causal inference literature.\nI believe these should be easily fixable and would increase the relevancy of the paper by better situating it within the causal inference literature.\n\n1. (Line 36-37) In reference to cluster designs, authors write: \"This technique is originally developed in [31] and becomes a prevalent paradigm for network experiment design\". I completely agree that [31] was an influential paper for introducing cluster designs to the computer science community. [35] works the exposure mapping framework for interference [1] which allows for this arbitrary network interfernece. However, the so-called \"partial interference\" assumption has been used since at least Hudgens & Halloran (2008) and cluster designs were advocated for here. So, I'd at least reference some of this early work on clustering in the context of partial interference. I think this will also tie your contributions back to the vien of causal inference in the statistics literature in a stronger way.\n2. (Line 38) Authors write that \"sharing same treatment within cluster is usually necessary for characterizing the GATE\". I would remove or substantially weaken this statement. The proliferation of cluster designs is *not* because they are necessary; but rather, a conceptually simple yet effective type of experimental design.\n3. (Line 64) Authors write \"In this paper, we propose to treat the covariance matrix of treatment vector as a decision variable in optimization\". The following paper seems especially relevant and authors should draw a comparison: Harshaw et al (2019) \"Balancing Covariates in Randomized Experiments using the Gram--Schmdit Walk Design\". This paper studies experimental designs that directly control the covariance matrix Cov(z) (via discrepancy theory) to bound the variance of Horvitz--Thompson estimator by an implicit ridge regression of outcomes on covariates. A key idea in that paper is to use the operator norm as a measure of worst-case variance, which seems like an alternative to your Assumption 2. Given the similarity in the spirit of the two papers, this paper would benefit from a brief comparison discussion.\n4. (Line 94) Authors cite several papers on bipartite experiments. It seems that [15] and [16] are duplicates. To the best of my knowledge, the paper of Zigler & Papadogeorgou (2021) was the first paper to propose bipartite experiments (it has been a working paper since 2019), so a citation is warranted in that discussion.\n5. (Line 104): Authors write \"Along the same direction, [32] tries to provide...\" I recommend revising this language. The word \"tries\" gives the indication that \"[32] tries and fails\".\n6. (Line 108): In the discussion of partial interference, early work like Hudgens and Halloran (2008) is missing.\n7. (Line 116): A citation of several papers with more general forms of interference is listed. The recent paper Harshaw, Sävje, Wang (2022) \"A design-based riesz representation framework for randomized experiments\" is worth citing, as it proposes a deisgn-based framework which captures and extends previous types of interference.\n8. (Line 251): I haven't read [19], but it was my understanding that Grothendieck's identity is typically a different type of statement, where there are two fixed vectors $x$ and $y$ and the random variables are $\\textrm{sign}(\\langle z , x \\rangle)$ and $\\textrm{sign}(\\langle z , y \\rangle)$, where $z$ is uniform from the $\\ell_2$ ball. In fact, I have only seen Lemma 1 in certain course notes on Sums-of-Squares (though I'm sure it's appeared in other places). If Lemma 1 does not directly appear in [19] then authors should cite a relevant paper that derives it. In fact, this would probably be helpful to tie your work back to theoretical computer science's use of the technique.\n\n## References\n\n- Harshaw, C., Sävje, F., Spielman, D., & Zhang, P. (2019). \"Balancing covariates in randomized experiments with the Gram-Schmidt Walk design\". (arXiv:1911.03071)\n- Harshaw, C., Sävje, F., & Wang, Y. (2022). \"A design-based riesz representation framework for randomized experiments\". (arXiv:2210.08698)\n- Hudgens, M. G., & Halloran, M. E. (2008). \"Toward causal inference with interference\". Journal of the American Statistical Association, 103(482), 832–842.\n- Zigler, C. M. and Papadogeorgou, G. (2021). \"Bipartite causal inference with interference\". Statist. Sci., 36(1):109–123.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for such a detailed and informative review, and your feedback will undoubtedly contribute to the enhancement of our work. We address your concerns as follows.\n\n## Confidence interval\n\nWe think it poses a significant challenge due to the estimation of nuisance parameters, such as $\\gamma$ and $\\beta_i$, within the potential outcome model. These parameters play a crucial role in both bias and variance, and the lack of precise information for their estimation necessitates the introduction of additional assumptions when constructing a confidence interval for the GATE estimator. For instance, constructing a conservative confidence interval with variance bound and $\\hat\\gamma$ may be feasible.\n\n## Responses to remarks\n\nWe are deeply grateful for the technical and literature remarks, which will substantially facilitate our paper's improvement. The following comments are provided in response to some remarks:\n\nFor technical remark 5, we can increase Monte Carlo samples to 10,000 before camera-ready submission.  \n\nFor literature remark 8, the form of lemma 1 in the main body is presented for its intuitive role within our pipeline. We can understand multivariate Gaussian variables $X$ and $Y$ as $\\langle g,x\\rangle$ and $\\langle g,y\\rangle$, where $g\\sim \\mathcal{N}(0,I_n)$ and $x,y$ are two real-valued vectors. This aligns with the original form of Grothendieck identity and is exactly the form employed in our proof of this lemma. In addition, we'll adjust the citation for this identity to a more contemporary paper, wherein it's explicitly presented.\n\n## Singleton cluster\n\nWe think cluster structure isn't necessary in pursuit of variance reduction, since we have a much larger decision space without this pre-specified scheme, and our method can extend to this case naturally. However, an important component of our methodology, $C$ matrix, would be uninformative, and assumption 2 would be vulnerable, in this case. \n\nOn the other hand, we maintain that the point lies in estimation error instead of approximation error, and we think the restriction of decision space granted by cluster structures is effective for us to find such a relatively good solution. Empirically, we find that extremely high resolution (>1000, near to singleton cluster case) in clustering corresponds to an increased MSE, which is approximately 20% higher as compared to a resolution of 10. \n\nIf scalability is also taken into consideration, then the cluster structures is indispensable. \n\n## Grothendeick mapping\n\nThis is a very interesting question, and we think the answer to it is exactly no. The validity of $\\sin(2\\pi X)$ as a PSD matrix cannot be guaranteed, even when $X$ functions as a legitimate covariance matrix for a multivariate Bernoulli distribution. This observation is elaborated upon in lines 255 to 257. Moreover, if the recovery procedure is omnipotent, then we'll get the conclusion that the distribution of $n$-varaible Bernoulli vector is uniquely determined, given its mean vector and covariance matrix, since the Gaussian distribution behind is uniquely determined. Paradoxically, mean vector and covariance matrix can't uniquely determine the multivariate Bernoulli distribution for $n>2$ case. \n\nHowever, this answer bears no impact on our procedures. The parameterization method and the projection step in our algorithm collectively provide a guarantee for sampling from the corresponding Gaussian distribution and the transformed variables are subject to such an optimized covariance matrix, though at a price that we don't search in all possible covariance matrices.\n\n## Comment on comparison to Harshaw et al. (2019)\n\nWe appreciate the significance of the paper by Harshaw et al. (2019) concerning our work. In our forthcoming paper revision, we are committed to incorporating a comprehensive discussion that addresses this connection. The crux of similarity resides in the methodology of optimizing an upper bound of MSE of HT estimator. Moreover, both of them utilize the covariance matrix as decision variable in optimization. \n\nThe difference in problem setting, optimization formulation and algorithm are readily discernible, and we primarily comment on the difference in the derivation of MSE bound. We consider interference conducted through social network, where the interference is modeled as an additive term $\\gamma \\sum_{j \\in N_i} z_j$ in our potential outcome model. This component makes covariance appear in the bias of estimator, and there exist second, third, and fourth-order terms w.r.t. treatment vector $t$ in the expression of variance of estimator.  In Harshaw et al (2019), Lemma 3.2 tells us that the influence of experiment design is absolutely captured by $\\operatorname{Cov}[z]$, while in our case, the influence of experiment design is much more complex. We derive a variance bound that only contains $\\operatorname{Cov}[z]$ through careful analysis, in which we focus on guaranteeing the bound isn't too loose but still optimizable w.r.t. $\\operatorname{Cov}[z]$. The techniques and ideas to tackle with higher order term (>2) only appear in our paper, and the situation that experiment design is absolutely captured by $\\operatorname{Cov}[z]$ is far from granted for free in our case.\n\nIn lines 340~343 of our paper, we provide a discussion about an alternative upper bound of second-order term, namely, the spectral norm of $E[tt^T]$ (also, operator norm, with $l$2-norm), under a weaker comparability assumption. This is exactly the technique applied by Harshaw et al (2019) to bound MSE, while the point of view is different, here we refer to this as \"weaker comparability assumption\", and it's understood as \"worst-case MSE\" in Harshaw et al (2019). The logic behind this bound is more natural than that of assumption 2, while this worst-case bound may be too conservative, as evidenced by the performance of IBR in simulation studies, for which similar worst-case variance is utilized."}, {"review_id": "LtNqxcMRfd", "reviewer": "Reviewer_EFcj", "summary": "This paper focused on designing a randomization scheme at the cluster level for the A/B test. It proposed and derived an upper bound for the MSE of the HT estimator, which was targeted to be minimized to optimize the experiment design. This article treated the covariance matrix of the treatment vector as the decision variable in optimization to represent a class of design. Besides, it proposed to adopt projected gradient descent to guarantee the validity of the optimized covariance matrix in order to generate a randomization scheme that supports legitimate sampling. Also, it stressed the significance of boas and attempted to balance bias and variance better. Systematic simulations on a real-time social network were conducted. Experiment outcomes were analyzed and different methods were compared to show the effectiveness of the newly proposed method in this article.", "questions": "1. Where is the definition of di that appeared below line 150 in assumption 1? (Does it refer to the degree of node i?) It is hard to read when definitions are missed or distributed in different parts. It is better to summarize all of them into one table with clear definitions.\n2. In section 2.1, line 132, what are the certain community detection algorithms? It is better to list out some works with references.\n3. The experiment part seemed to be written in a hurry. Even though necessary comparison methods were involved, many other essential details such as the basic information about the dataset (i.e., edges, nodes, degrees, etc.) were missing. Also, it seems that experiments were only implemented on one dataset, which made the experiment outcome not convincing enough.\n4. What is the real-world application of this method? Please offer some direct applications such as the areas that may benefit from your method. This is important since your derivation used different assumptions when dealing with difficult parts in the optimization. Will the assumptions affect its performance in general? Is the proposed method only workable under a specific scene and what is the case?", "limitations": "Limitations of this article were clearly expressed in this article. For instance, the upper bound should be tighter in the optimization. And different other cases like the non-balanced treatment assignment have not been considered yet. The basic setting beyond nationhood interference assumption warrants a further extension. Besides, assumptions may hinder the method from applying to a more generalized case.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is generally easy to follow.\n2. The problem is well formulated in a new perspective, with detailed analysis and deduction.\n3. The effectiveness of the proposed method is verified in the experiment. Horizontal comparisons with existing methods are sufficient and effective.", "weaknesses": "1. Insufficient background introduction. Different concepts in the area should be introduced more specifically and it is better to point out their connections. \n2. Unclear notation definitions and explanations, leading to confusion in understanding.\n3. Experiments were not comprehensive, or at least not comprehensively expressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thanks for your time and efforts in the reviewing process, and we appreciate your approval on the problem formulation and content presentation of our work. We address your concerns as follows.\n\n## Not comprehensive experiments\n\nWe conduct an array of simulations with a substantial portion of the detailed results presented in the appendix. We specifically focus on highlighting the advantages of our proposed OCD method, particularly in scenarios where interference intensity, represented by γ, is elevated. We think our simulation study is systematic and well-formulated to demonstrate the efficacy and robustness of our proposed method.\n\n## Responses to questions\n\n1. **Clarification on notation**: Thank you for highlighting the oversight regarding the notation $d_i$.Yes, $d_i$ indicates the degree of unit $i$. We acknowledge that this definition is introduced in the linear potential outcome model originally within the introduction, which was finally deleted due to limited space. We will certainly address this in the upcoming version. Rest assured, other notations in the paper are accompanied by clear definitions or appropriate citations, ensuring overall clarity.\n\n2. **Community detection algorithm**: We appreciate your attention to detail, and we'll add additional notes in line 132 for better clarity. Actually, we've mentioned and cited the classical community detection algorithm (Louvain) in the introduction (line 44), and this algorithm is also mentioned in our simulation study. \n\n3. **Network statistics and dataset results**: We think network statistics is relatively unnecessary for the body of this paper, and we provide the accessible links to the panel of such network topology in the footnote of page 7 (and page 4 in the appendix), where most of statistics of network topology are provided. Secondly, we provide the results of two datasets in total, in the appendix, which is also mentioned in line 303.\n\n4. **Simulation for method applicability and scalability**: We think the comprehensive simulation has provided adequate evidence for the applicability of our method on different scenes, which includes different interference intensities, different clustering resolutions (corresponds to cluster numbers), different estimators, different potential outcome models, and different datasets. As mentioned in the introduction, experiment design for AB test on social network is widely used for social platforms such as Facebook, LinkedIn, and WeChat. The deployment of every new trait for social platform must come through a series of AB tests with increasing scale, which is a classical and important scenario for our method and mentioned in the introduction. Moreover, we've argued that the gradient-based algorithm for our continuous optimization problem is scalable for practical scenes and confirmed by our industrial collaborator. "}]

Author rebuttal:

Meta-review: The paper studies the experimental design for A/B tests in social networks, which may have interference due to individuals in social networks influencing one another. The authors propose a method to balance bias and variance and design an algorithm to achieve a better performance. The reviewers provided concrete comments and critics, for which the authors gave detailed replies. The reviewers are satisfied by the authors' rebuttal, and two of them raised their scores, indicating that the authors' rebuttals are effective, and the reviewers are confident in supporting the paper after the authors' explanations. The topic is an important one, and the study provides a nice addition to the research community on this topic. I recommend acceptance to the paper.
Decision: Accept (poster)