Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper studies cluster randomized signs for one-sided bipartite experiments under network interferences. The paper discussed many different models, but I will only define the main one studied. There is a weighted bipartite graph between a set of experiments and a set of interference units, with the weight w_{is} between the experiment i and interference unit s indicating relationship between i and s.  For each experiment i, we can assign either "control" (Z_i = -1) or "treat" (Z_i = 1) to i.  The outcome for the i-th experiment is a real number Y_i(Z) = alpha_i + beta_i Z_i  + gamma_i e_i(Y). The alpha_i + beta_i Z_i is a linear function of the treatment Z_i for i, and e_i is called the exposure, which captures the interferences of the (Z_j)_{j\neq i} values to the outcome for i. e_i(Y) is called the exposure mapping.

The paper propose a bipartite analogue of the neighborhood-based exposure mapping. In this model, the dose of an interference unit s is computed as the weighted average of the treatment assignment Z_i among the neighboring experiments i of s, where weights are w_{is} values. Then, the exposure e_i is the weighted average of the doses of neighboring interference units s of i, again using w_{is} values.

The average total treatment effect tau is defined as 1/N * sum_{i \in [N]} (Y_i(vector Z = all 1 vector) - Y_i(vector Z = all -1 vector)). In the difference-in-means (DIM) estimator for tau, we choose randomly N_T experiments to treat, and N_C experiments to control, where N_T and N_C are numbers chosen upfront. The estimation given by the DIM estimator is \hat \tau_DIM := \sum_{i:Z_i = 1} Y_i/ N_T - \sum_{i:Z_i = - 1} Y_i/N_C.  The bias of the estimation is defined as \tau - \hat \tau_DIM.

The main goal of the paper is to use balanced clustering to reduce the mean squared error (MSE) of the DIM estimator. Let C be a partition of the N experiments into K equally sized clusters.  A balanced k-cluster randomized design D(C) is a distribution over vectors Z \in {-1, 1}^N, such that experiments in the same cluster have the same +-1 values, and there are K_T clusters with +1 values. The authors gave the formula for the bias under the cluster design, presented some robustness results, and conducted experiments on the effectiveness of the method. 
strengths_and_weaknesses: Evaluations:  The main contribution of the paper is use of the one-sided cluster design algorithm to reduce the bias of DIM estimator.  However, after giving the formula and the optimization problem in Lemma 3.2, I fail to see why this algorithm is better than the standard one without the clustering. A lemma about some typical cases or even an example suggesting this would be helpful.  Moreover, the clustering idea was not new. It is used in cases where all units are experimental units, and where there are experimental and interference units, but the outcomes are measured in the interference units.  It is not so hard to extend the idea to the setting studied in this paper. 

I am also annoyed by the bad format of the supplementary material. There are so many places where the mathematical formulas exceed the width of a line by a lot. The authors may have submitted the paper (at least the supplementary material) in a hurry. 
questions: It would be good to compare the clustering-based algorithm and the original one (without the clustering used) theoretically.  This could be done using an example, or a theorem that covers typical cases that arise in practice. 
limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors examine a setting of experimentation they refer to as "one-side bipartite experimentation", in which treatment and outcomes are measured on one set of units, but connections between units (and, therefore, interference) operates through a distinct set of units. The authors provide theoretical results and simulations which demonstrate that clustering on a special objective function can effectively allow for inference in this setting with minimal bias.

strengths_and_weaknesses: Originality:
- The theoretical analyses are not particularly novel.
- The setting, while interesting, is pretty straightforward: the one-sided bipartite graph just implies a straightforward way to construct an exposure graph. The methods thereafter are fairly straightforward.

Quality:
+ The authors do a good job of drawing out the implications of their main clustering result
+ The focus on design rather than analysis is very welcome.

Clarity:
+ Found the paper quite easy to follow.

Significance:
+ The problem setting is interesting and relevant to many practitioners.

Section 2.1 defines a model which is, essentially, the "Linear-in-Means" model of [1]. This important related work should be discussed. In particular, [1] draws out an agnostic basis for this model. Section 2.2 also has a discussion of exposure maps which greatly parallels section 3.1 of [1]. This is not an accusation of plagiarism, to be clear, but the paper would be improved by drawing more heavily on this.

Greater discussion of the estimand under less favorable data generating processes (in the vein of [2]), could be an interesting direction to make the work more useful in practice. The simple SBM of section 5.2, for example, obviously aligns nicely with a clustering-based design.

[1] A. Chin, Regression adjustments for estimating the global treatment effect in experiments with interference, Journal of Causal Inference, May 2019. https://arxiv.org/abs/1808.08683
[2] Fredrik Sävje. Peter M. Aronow. Michael G. Hudgens. "Average treatment effects in the presence of unknown interference." Ann. Statist. 49 (2) 673 - 701, April 2021. https://doi.org/10.1214/20-AOS1973
questions: The authors claim "variance is inconsequential in this setting". Surely there are scope conditions or assumptions necessary for this statement to hold. I think the authors may be assuming that methods compared are methods of clustered-RA (or unit-level RA where bias is extremely large)? If all clustering methods use the same number of clusters, this would be true, but I don't see why that would necessarily be the case if one were choosing the number of clusters as a practitioner would be (e.g. through finding an elbow, etc). The appropriate number of clusters could look quite different depending on the clustering objective. In any event, I'd appreciate the authors to clarify this (also in the text).
limitations: Methods like this which allow for better and more accurate experimentation are a key way to detect and analyze biases in complex real-world systems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
In this work, the authors study a new experimental design problem under interference. The underlying graph capturing the interference is a bipartite graph with two types of units (corresponding to the two partitions): experimental and interference units. The goal is to assign the experimental units to treatment or control to estimate the total treatment effect on the experiment units. The only purpose of the interference units is to facilitate interactions between experimental units. The authors motivate this setting with marketplace experiments, where buyers interact with sellers.

For this problem, they study the performance of difference-in-means estimators for cluster-randomized balanced designs, under a suitable linear exposure model. The main claim is a min-max optimal equivalence between treatment effect estimation and identifying a good clustering. In other words, they showed that the clustering that minimizes the maximum bias will result in identifying a partitioning of a weighted graph, with weights representing the strength of interference between the units. They demonstrate the performance of their approaches empirically, along with a robustness study, when the underlying data deviates from the studied exposure model.



strengths_and_weaknesses: The presentation in the paper is excellent, containing extensive discussions on their choice of experimental designs and estimators. 

They relate two important quantities of interest to a graph partitioning objective: bias of the estimator and robustness of the design. The robustness is captured using covariance of the exposure vector (under the interference model) and the design vector. From the proofs in the appendix, it is evident that the bias is a constant factor away from the graph partitioning objective, which is computationally challenging to solve exactly, and for which many approximation algorithms are already known in the literature. These new relationships/equivalences are novel and could be helpful in furthering our understanding of experimental designs under interference.

It certainly would be an important addition to strengthen the current work by considering confidence interval bounds for the bias of the considered estimator. 

The authors restrict themselves to balanced designs -- it is not immediately clear if this is necessary. They claim that it reduces the variance in the estimator. It would be helpful to understand the exact dependence of parameters here. Moreover, the theoretical claims in the paper do not compare directly with respect to a simple baseline, such as unit-level randomization. As stated, they show the equivalence in terms of the exact clustering, but not the objective values themselves.

Although min-max optimal results are useful, it would be interesting to look at specific cases, for which one can do better than off-the-self graph partitioning (clustering) algorithms from prior literature.

questions: Can the exposure model be tied into some of the earlier literature works? Isn't there double counting with respect to edge weights? (It suggests that the model was chosen after discovering the correct clustering objective)

In Lemma 3.2, are the terms corresponding to gamma0 and gamma1 missing? Although the proof in the appendix states it clearly, I would suggest fixing it to avoid any confusion.

It would be interesting to see the differences in the relative bias (Table 1) with respect to Tr(Var(d)) and H(C) for other synthetic graphs, such as power-law, which exhibit skewed distributions (unlike the stochastic block model used for simulations).


limitations: As this is a theoretical paper, with simulated experiments, they did not discuss the potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies cluster randomized designs for one-sided bipartite experiments. The authors assume a linear potential outcome model, and a normalized exposure mapping model, and focus on difference-in-means estimators to estimate the average total treatment effect. They propose to find the design that minimizes the bias among balanced K-cluster randomized designs. The authors then demonstrate the robustness of the design to deviations from the assumed potential outcome model/the exposure mapping model. Finally, simulations studies are conducted to show the empirical performance of the design. 
strengths_and_weaknesses: Strengths

-	The paper is very well-written. The proposed designed is motivated and explained in a very nice way. 

-	The paper addresses an interesting and important question: estimating the total treatment effect in the setting of one-sided bipartite experiments. 

-	The proposed design appears to have good performance both theoretically and empirically. 

-	The paper does a great job discussing robustness to model misspecifications both theoretically and empirically.

Weaknesses

-	I would love to see more detailed discussions on 1. Comparison with two-sided designs. 2. By how much balancedness helps reduce variance. 3. Implementation details (See questions section for more details)

questions: -	I’m curious to see how the proposed design compares to the two-sided designs proposed in [27, 28] (I used the same references as in the paper) Do you believe the proposed design can outperform the two-sided ones? I understand that in practice, it would require different platforms to run one-sided experiments/two-sided ones, so it is not 100% comparable, but I’d love to see some more comparisons and discussions. Thank you!

-	Line 179-184. The authors argue that balancedness helps reduce variance. By how much does balancedness help reduce variance? It will be helpful to see some theoretical/empirical results on this. Or even some more detailed references will be very helpful. 

-	Line 238-241. Is the proposed method implemented with the algorithm in [34]? I don’t think this is made 100% clear except a brief mention in line 284, which seems to be referring to how the benchmark methods are implemented. It will be helpful to clarify what specific algorithms are used. Are exact balancedness achieved? If only partial balancedness is achieved, how close is this to exact balancedness? 

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This well-written paper proposes a possibly-new experiment-design problem where there is interference. This interference is modeled by a bipartite graph where one side has the "experimental" units and the other has "interference" units. The purpose of the interference units is to facilitate interactions between the experimental units. The goal is to assign the experimental units to "treatment" or "control" in order to estimate the total treatment effect on the experiment units. Specifically, for each experiment i, we can either assign "control" (Z_i = -1) or "treat" (Z_i = 1). The outcome for this experiment is the value Y_i(Z) = alpha_i + beta_i Z_i + gamma_i e_i(Y)---where e_i(Y) is called the "exposure mapping" that captures the interferences of the (Z_j: j distinct from i) values with the outcome for i. This setting is motivated by marketplace experiments where buyers interact with sellers.

This primarily-theoretical paper studies the performance of difference-in-means estimators for cluster-randomized balanced designs, under a "linear exposure" model. One key results is a min-max optimal equivalence between treatment effect estimation and identifying a good clustering: the clustering that minimizes maximum bias will basically yield a partitioning of a weighted graph, with weights representing the strength of interference between the units. Simulation results are also given. 

There are concerns about the novelty of the clustering objective, and the supplementary material has poor formatting, but the paper's contributions are appreciated. The authors are asked to carefully incorporate the referee-comments. 
confidence: Less certain
Decision: Accept