SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qianyi Chen $^{1}$ , Bo Li $^{1*}$ , Lu Deng $^{2}$ , Yong Wang $^{2}$
$^{1}$ School of Economics and Management, Tsinghua University, China
Tencent Inc., Shenzhen, China
cqy22@mails.tsinghua.edu.cn, libo@sem.tsinghua.edu.cn
adam Deng@tencent.com, darwinwang@tencent.com
Online A/B tests have become increasingly popular and important for social platforms. However, accurately estimating the global average treatment effect (GATE) has proven to be challenging due to network interference, which violates the Stable Unit Treatment Value Assumption (SUTVA) and poses great challenge to experimental design. Existing network experimental design research was mostly based on the unbiased Horvitz-Thompson (HT) estimator with substantial data trimming to ensure unbiasedness at the price of high resultant estimation variance. In this paper, we strive to balance the bias and variance in designing randomized network experiments. Under a potential outcome model with 1-hop interference, we derive the bias and variance of the standard HT estimator and reveal their relation to the network topological structure and the covariance of the treatment assignment vector. We then propose to formulate the experimental design problem as to optimize the covariance matrix of the treatment assignment vector to achieve the bias and variance balance by minimizing a well-crafted upper bound of the mean squared error (MSE) of the estimator, which allows us to decouple the unknown interference effect component and the experimental design component. An efficient projected gradient descent algorithm is presented to the implement of the desired randomization scheme. Finally, we carry out extensive simulation studies<sup>2</sup> to demonstrate the advantages of our proposed method over other existing methods in many settings, with different levels of model misspecification.
A/B test, also known as controlled experiment, is an effective way to evaluate treatment effect of new product features and facilitate data-driven decision-making [14]. Due to its simplicity and efficiency, it has become the gold standard in the industry. The theory of A/B test is based on the Stable Unit Treatment Value Assumption (SUTVA), under Neyman-Rubin's framework [33], which assumes unit's treatment effect isn't influenced by other units' treatment assignments. However, in situations where social network connects units, SUTVA is often violated since an individual's behavior can be influenced by their friends. This violation of SUTVA is referred as network interference or social interference, which brings challenges to both estimation and inference of parameters of interest. It has been shown that ignoring network effect can induce much higher bias or variance [29].
To tackle the bias and variance induced by network interference, researchers have tried to design randomization schemes in pre-treatment phase and estimators or adjustment methods in post-treatment phase. In this paper, we will focus on designing randomization scheme, namely, experiment design.
The code is available at https://github.com/Cqyiii/Optimized_Covariance_Design-NIPS2023
Specifically, we take the graph structure as known and leverage it to assist experiment design. Our basic scheme is cluster-level randomization, which treats cluster or community in the graph as unit of treatment and all units in the same cluster share the same treatment. [19] is an early work investigating this approach, and [35] proposes to form clusters utilizing graph. The graph cluster randomization is conceptually simple but effective in many scenes, and thus becomes a prevalent paradigm for network experiment design. Moreover, we consider experimental design on a social network in this work. Social network typically has organic structure and high clustering coefficient, which corresponds to meaningful cluster structure and also motivates us to adopt cluster-level randomization. We assume the cluster structure of the network is given. In practice, interpretable community detection algorithms such as the Louvain algorithm [4] are typically applied for cluster construction. Moreover, we stress the scalability of our method since social network is usually large-scale in reality.
Related works have shown that there exists an explicit bias-variance tradeoff [23, 41]. However, many HT-style estimators that utilize exposure indicator to ensure unbiasedness are designed on the basis of the neighborhood interference assumption that restricts global interference to 1-hop neighborhood [10, 25, 36, 39, 9], and graph-dependent generalized propensity score [1]. Nonetheless, we argue that pursuing unbiasedness by simple sample trimming leads to small effective sample size with high variance, and thus is far from optimal. Moreover, computational cost is also high in this approach since a large number of Monte Carlo simulations are usually needed for computing the generalized propensity score, or the trimming probability for constructing sample weights. Having observed the preceding drawbacks of the unbiased estimator, we consider the standard unit-level HT estimator to achieve possible better bias and variance tradeoff with less computation burden at the meantime. Based on a potential outcome model, we characterize the relationship among the bias, variance, the network topology, and the treatment assignment mechanism. This development allows us to design experimental schemes that minimizes MSE and achieve better bias-variance tradeoff.
Besides from the time-consuming Monte Carlo simulation for calculating generalized propensity score mentioned above, the computational efficiency of experiment design is indeed a limitation of current methods. For instance, the two-wave experiment [37], entails solving a mixed-integer quadratic programming problem, which can be computationally expensive. The adaptive covariates-balancing design [25] adopts a sequential process for treatment assignment and thus isn't very scalable. To address this limitation, we propose to utilize gradient-based optimization algorithm.
In this paper, we propose to treat the covariance matrix of treatment vector as decision variable in optimization, which is nontrivial as a result of interference and corresponding complex variance of estimator. Similar approach in optimization step is proposed in [17], where interference isn't considered, and making covariance covariance acting as decision variable is a natural choice owing to the simple structure of variance there. We draw attention to the fact that related works concerning experiment design mostly concentrates on a predetermined class of randomization schemes and seeks to optimize the parameters to obtain designs that minimize variance or MSE of estimator [37, 23, 5, 42]. However, we argue that these designs can all be understood as different patterns of the covariance matrix of the treatment vector, and we propose to treat the covariance matrix as decision variables rather than limiting our scope to a particular class of designs. One of major challenges with this approach is generating a randomization scheme that supports legitimate sampling, as there may not be a joint distribution that corresponds to the optimized covariance matrix in our formulation. To address this issue, we propose to adopt projected gradient descent to guarantee the validity of optimized covariance matrix. Furthermore, we also compare our method with other popular designs through extensive simulation on a real online social network. Various interference intensity levels and potential outcome models are considered. It is found that our method is robust to model misspecification and outperforms in almost all settings, especially when the interference isn't too weak compared to the direct treatment effect, which is the scene we target.
Contributions. Our contributions are summarized as follows.
1. We derive a well-crafted upper bound for the MSE of the HT estimator that decouples the estimation of causal mechanism and experimental design. This enables us to optimize the experimental design by minimizing this bound, in which the covariance matrix of treatment vector acts as decision variables.
2. We propose the formulation of covariance optimization problem with reparameterization of covariance matrix and constraints that guarantees legitimate sampling subject to optimized
covariance matrix, and a projected gradient descent algorithm is proposed for solving the optimization problem.
3. We conduct systematic simulation on a social network and compare our method with several methods proposed recently under a range of settings, providing credible reference for the effectiveness of our method.
Related Works. A variety of interference has been considered in related works[12], such as spatial [23], temporal [13] and network, which is the scene we target. Besides from the general network, there is also another line of work focused on a two-sided marketplace or platform, where the graph is bipartite [43, 24, 16, 30, 27].
In the field of network experiment design, most of extant research assumes that the graph structure is known and leverage it to assist in modeling the interference[2], though several graph agnostic methods are developed these days [40, 8]. Nevertheless, the interference is conducted through edges of network, and agnostic methods aren't effective enough.
Beyond unit-level randomization, cluster-level randomization on graph has attracted extensive attention, including both experimental design [23, 25, 5, 30] and estimation [26, 1, 6, 15]. The seminal paper of this line of work is [35], which proposes graph cluster randomization that treats clusters as units in randomization. This methodology is motivated by providing a better lower bound of exposure probability in HT estimator that uses exposure probability as inverse weight, aiming at controlling the variance of such an unbiased estimator. Along the same direction, [36] proposes a polynomial lower bound on exposure probability through randomized local clustering algorithm.
Irregular global interference may render the experimental design problem intractable [39], thus structural assumptions are usually introduced to rule out such arbitrary interference. The first category is partial interference [34, 19, 3, 11, 5], which assumes disjoint community structures and interference only happens within each community. The second category is neighborhood interference, which assumes the interference solely originates from units' 1-hop neighborhood [25, 36, 41, 12, 8]. This kind of assumption is more practical compared with partial interference, and thus widely applied. The third category is potential outcome model, which directly assumes the expression of potential outcome, e.g. classical Cliff-Ord auto-regressive model [7], linear potential outcome models[25, 39]. Actually, some potential outcome model implies neighborhood interference assumption in its mathematical expression. It's also worth to be noted that there also appear works trying to weaken the structural assumptions mentioned above and consider the interference of more general form [6, 22, 23, 18, 15].
2 Basic Framework
We consider a finite population of  $n$  units, and the treatment vector is represented as  $z = (z_{1},z_{2},\ldots ,z_{n})\in \{0,1\}^{n}$ . In this paper, we consider the experiment design problem in which a sensible random treatment assignment mechanism of all units is designed to achieve good performance of the resultant treatment effect estimation. We follow the Neyman-Rubin's framework [33]. Without SUTVA assumption, we define the potential outcome of unit  $i$  as  $Y_{i} = Y_{i}(z)$ , which means the potential outcome of unit  $i$  can depend on the treatment assignment of all units.
The parameter of interest is the global average treatment effect (GATE), which is denoted by  $\tau$ ,
$$
\tau := \frac {1}{n} \sum_ {i \in [ n ]} \left(Y _ {i} (\mathbf {1}) - Y _ {i} (\mathbf {0})\right) \tag {1}
$$
Here we use  $[n]$  to denote set  $\{1,2,\dots,n\}$ , and  $\mathbf{1}(\mathbf{0})$  is the  $n$ -dimensional vector of  $1(0)$ , which corresponds to global treatment and global control.
We denote the social network as  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ . The node set  $\mathcal{V}$  is  $[n]$ , corresponding to all units considered above. Edge set  $\mathcal{E}$  represents the social connection between units. Furthermore, we use  $d_{i}$  to denote the degree of node  $i$ . We assume the graph structure  $\mathcal{G}$  is known in advance.
In this paper, we consider graph cluster randomization [35]. Here we also assume that cluster structure is given, which is generated from certain community detection algorithm. We define  $S_{k}$  as the  $k$ -th
cluster, where  $k \in [K]$ . Clusters compose a splitting of node set, namely,
$$
S _ {k} \subset [ n ] \quad S _ {i} \cap S _ {j} = \emptyset \quad \cup_ {k = 1} ^ {K} S _ {k} = [ n ] \tag {2}
$$
The graph cluster randomization implies that units in the same cluster share the same treatment assignment, namely,
$$
z _ {i} = z _ {j} \quad \forall i, j \in S _ {k}, k \in [ K ] \tag {3}
$$
Since we consider randomization scheme at cluster level, it's more convenient to define cluster-level treatment vector  $\pmb{t} = (t_{1}, t_{2}, \dots, t_{K}) \in \{0, 1\}^{K}$ . Specifically, we consider the balanced cluster-level randomization scheme satisfying
$$
\mathbb {P} \left(t _ {k} = 1 | \mathcal {G}\right) = \frac {1}{2} \quad \mathbb {E} \left[ z _ {i} \right] = \frac {1}{2} \tag {4}
$$
2.2 Potential Outcome Model and Estimators
One of the key ideas of this paper is to deduce an appropriate objective function to optimize experiment design schemes under consideration. In this part, we will consider standard HT estimator, namely,
$$
\hat {\tau} = \frac {1}{n} \sum_ {i \in [ n ]} \left(\left(\frac {z _ {i}}{\mathbb {E} \left[ z _ {i} \right]} - \frac {\left(1 - z _ {i}\right)}{\mathbb {E} \left[ 1 - z _ {i} \right]}\right) Y _ {i} (\boldsymbol {z})\right) \tag {5}
$$
Then we introduce the considered potential outcome model.
Assumption 1 (Potential Outcome Model) The potential outcome of unit  $i$  is generated by
$$
Y _ {i} (\boldsymbol {z}) = \alpha_ {i} + \beta_ {i} z _ {i} + \gamma \sum_ {j \in N _ {i}} z _ {j} \tag {6}
$$
where  $N_{i}$  is the 1-hop neighborhood of unit  $i$ .
In this potential outcome model,  $\beta_{i}$  can be interpreted as direct treatment effect of unit  $i$ , and  $\gamma$  measures the interference effect, characterizing the extent to which each unit is affected by its neighborhood in the network. This model is a simplified (in interference part) version of overparameterized model in [39], in which heterogeneous interference effects are accommodated. We make a somewhat stronger assumption to simplify the structure of the bias, variance and MSE. Under this model, we can express the GATE in terms of the potential outcome model parameters,
$$
\tau = \frac {1}{n} \sum_ {i \in [ n ]} \left(Y _ {i} (\mathbf {1}) - Y _ {i} (\mathbf {0})\right) = \frac {1}{n} \sum_ {i \in [ n ]} \left(\beta_ {i} + \gamma d _ {i}\right) \tag {7}
$$
We will then derive the bias and variance of  $\hat{\tau}$  under this potential outcome model. We point out that the number of unknown parameters in this model is still greater than the number of known potential outcomes, which can be understood as overparameterization, and we only utilize it to derive an estimable MSE upper bound that's not concerned with estimation of  $\beta_{i}$  and  $\gamma$  in this model. In fact, when the interference effect and the direct treatment effect are comparable at cluster-level, a multiplier of the MSE upper bound can be factored out, which is crucial in our methodology and will be shown in section 3.1.
2.3 Bias of HT Estimator
Based on the potential outcome model, we can derive the bias and variance of our estimator explicitly. We firstly introduce cluster-level treatment assignment,  $\pmb{t} = (t_{1}, t_{2}, \dots, t_{K})$ , and matrix  $C$  summarizing edges between (within) clusters with its elements defined as
$$
\boldsymbol {C} _ {i j} = \left| \left\{\left(u, v\right): (u, v) \in \mathcal {E}, u \in S _ {i}, v \in S _ {j} \right\} \right| \tag {8}
$$
We have following proposition about bias.
Proposition 1 (Bias of HT Estimator) The bias of standard HT estimator is
$$
\mathbb {E} [ \hat {\tau} ] - \tau = \frac {\gamma}{n} \left(4 \operatorname {t r a c e} (\boldsymbol {C} \operatorname {C o v} [ \boldsymbol {t} ]) - \sum_ {i, j \in [ K ]} \boldsymbol {C} _ {i j}\right) \tag {9}
$$
where  $\operatorname{Cov}[t]$  refers to the covariance matrix of treatment vector  $t$ .
Specifically, we can delineate the source of bias in our setting. If we consider graph cluster randomization as basic scheme, edges within cluster don't contribute any bias, while every edge between clusters contributes to bias according to the covariance of cluster-level treatment assignments. This enlightens us that independent treatment assignments is far from optimal, and we can reduce bias by assigning treatments with appropriate positive correlations that should be higher for more densely connected cluster pair. Furthermore, we point out that bias may be of no account when a very sparse network is considered, but it indeed counts in our context of social network.
We also note that the treatment vector is summarized by its covariance matrix in the expression of bias. Hence, we denote the bias by  $B(\mathrm{Cov}[t])$ .
2.4 Variance of HT Estimator
To derive the variance, we plug in the potential outcome model and express the estimator as
$$
\hat {\tau} = \frac {2}{n} \sum_ {i = 1} ^ {n} - \alpha_ {i} + \left(\beta_ {i} + 2 \alpha_ {i} - \gamma d _ {i}\right) z _ {i} + 2 \gamma \sum_ {j \in N _ {i}} z _ {i} z _ {j} \tag {10}
$$
An important difference in the derivation of variance contrasting that of bias is all base levels  $\alpha_{i}$  is no longer negligible. Hence, we consider to target at the scenes that we can acquire knowledge of all base levels in advance.
Assumption 2 (Known Base Levels) The base level  $\alpha_{i}$  of unit  $i$  is known in advance for all units  $i$ .
This assumption is strong in the classic context of experiment design, while we propose to target the scene where AB tests are launched by social platforms. On the one hand, the improvement in user activity indices brought by new traits or interventions, specifically, direct treatment effect and interference, is typically much smaller than the base levels  $\alpha_{i}$ , which means the experiment design may be quite challenging when agnostic to base levels. On the other hand, it's very reasonable to assume that we know all  $\alpha_{i}$ s before conducting experiments since those indices we're concerned with are recorded and/or estimated daily or weekly by social platforms.
Utilizing this assumption, we can first adjust our estimator as
$$
\hat {\tau} _ {- \alpha} = \frac {1}{n} \sum_ {i \in [ n ]} \left(\left(\frac {z _ {i}}{\mathbb {E} \left[ z _ {i} \right]} - \frac {(1 - z _ {i})}{\mathbb {E} \left[ 1 - z _ {i} \right]}\right) \left(Y _ {i} (\boldsymbol {z}) - \alpha_ {i}\right)\right) \tag {11}
$$
This adjustment has also been recommended by [39]. Notice that this adjustment helps us remove the influence of base level  $\alpha_{i}$  when we consider variance. In the following, we'll adhere to this format and proceed to derive the variance term, which is summarized in the following proposition.
Proposition 2 (Variance of HT Estimator) The variance of standard HT estimator is
$$
\operatorname {V a r} [ \hat {\tau} ] = \frac {4}{n ^ {2}} \left(\operatorname {t r a c e} \left(\boldsymbol {h} \boldsymbol {h} ^ {T} \operatorname {C o v} [ \boldsymbol {t} ]\right) + 4 \gamma \operatorname {C o v} \left[ \boldsymbol {h} ^ {T} \boldsymbol {t}, \boldsymbol {t} ^ {T} \boldsymbol {C t} \right] + 4 \gamma^ {2} \operatorname {V a r} \left[ \boldsymbol {t} ^ {T} \boldsymbol {C t} \right]\right) \tag {12}
$$
where  $\pmb{h}$  is the vector  $(\sum_{i\in S_k}\beta_i - \gamma d_i)_k = 1$
Notice that the expression of variance contains second-order term with regard to  $t$ , namely,  $\mathrm{Cov}[t]$ , as well as third and fourth-order terms involving  $\boldsymbol{t}^T\boldsymbol{C}\boldsymbol{t}$ , a quadratic form of  $\boldsymbol{t}$ . It's feasible to analyze the variance precisely with a specific randomization scheme, such as the independent block randomization [5], but it is generally not straightforward to optimize a broad set of randomization schemes in its exact form. In the following subsection, we will derive a bound of variance to transform third and fourth-order terms into second-order, which also motivates our optimized covariance design.
We comment that a bias-variance tradeoff is unveiled in the previous two propositions. Actually,  $C$  is a matrix with non-negative elements, and introducing positive covariance between treatment assignments of different clusters will reduce the bias. On the other hand, introducing positive covariance means an increment in variance. In addition, though this paper doesn't involve upstream clustering algorithm, the resolution of clustering can also impact the relative importance of bias and variance, namely, low resolution usually corresponds to a higher proportion of edges within cluster, which weakens the importance of bias and stress on variance.
3 Optimized Covariance Design
3.1 Constructing Objective for Optimization
Although we have derived the expression of bias and variance, the expression of variance is complicated due to the existence of treatment effects  $\beta_{i}$ s, the interference effect  $\gamma$ , and the higher-order terms involving the treatment vector. We will bypass the first hurdle by adding reasonable assumption about the direct treatment effect and interference effect. The second challenge will be addressed by a upper bound argument.
We first consider the former issue, which is the relationship between  $\beta_{i}$  and  $\gamma_{i}$ . Since we consider the interference, we propose to restrict our scope to the situation that network interference isn't too weak, which is formulated as following assumption.
Assumption 3 (Comparability between Direct Treatment Effect and Interference) Given potential outcome model in equation (6), we assume there exists a constant  $\omega >0$  such that
$$
\left| \boldsymbol {h} _ {k} \right| \leq \omega \gamma \left(\sum_ {i \in S _ {k}} d _ {i}\right) \tag {13}
$$
holds for each cluster  $k \in [K]$ .
This assumption actually demands the magnitude of interference is comparable to direct effect at cluster level. Namely, we allow a significant difference between direct treatment effect  $\beta_{i}$  and interference  $\gamma d_{i}$  for certain units, but there isn't too much difference after aggregation in cluster. The constant  $\omega$  can be viewed as degree of tolerance on incomparability between direct causal effect and interference and depends on the both nature of potential outcome and graph structure. It's natural that people may be more susceptible to the states of their friends in some social metrics. Since we'll utilize the MSE bound as optimization objective, this constant can also be viewed as tuning parameter that's determined by domain knowledge or pilot experiment. To demonstrate the robustness of our method, we don't tune  $\omega$  according to the result and fix  $\omega = 1$  in the simulation, which corresponds to a moderate constraint on comparability. Empirically, our result isn't sensitive to the value of  $\omega$ .
We then discuss the latter issue. We bound all high-order terms by second or lower order terms. This effort leads to an upper bound which allows us to simply optimize the covariance matrix.
Proposition 3 (Variance Bound) The variance of the standard HT estimator has following upper bound,
$$
\operatorname {V a r} [ \hat {\tau} ] \leq \frac {8 \gamma^ {2} \left(\omega^ {2} + 4\right)}{n ^ {2}} \operatorname {t r a c e} \left(\boldsymbol {d} \boldsymbol {d} ^ {T} \left(\operatorname {C o v} [ \boldsymbol {t} ] + \frac {1}{4} \mathbf {1 1} ^ {T}\right)\right) \tag {14}
$$
where  $\pmb{d}$  is the vector  $(\sum_{i\in S_k}d_i)_{k = 1}^K$