Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The text explores estimation under a rollout set-up, whereby the treatment of units is sequentially increased over time. This is a common setting in tech companies for example, also known as a “ramp-up” period. By assuming a polynomial function of interference (in the treatment status of the neighbors), they suggest lagrange interpolation as an estimator, which, nicely, does not require knowledge of the graph. 


strengths_and_weaknesses: The paper is nicely written and easy to follow. There are some synthetic experiments that illustrate their point, as well as a few technical results that follow nicely from their setting. The biggest strengths are

1. The usefulness of a graph agnostic estimator. Knowing graphs exactly is rare and proposing a graph agnostic estimator is useful in these settings (and uncommon in the literature on interference) 
2. The idea of leveraging roll outs is understudied for how valuable they can be to study interference.

My biggest criticisms are:
1. The related works section is short on details. Namely, I’m not sure I completely grasp the improvement over [25]. Going from linear to polynomial does not seem to be a hugely difficult step, so it would help if the authors could clarify in what way this extension is non-trivial.

2. The roll-out model assumes no time-dependent noise (or noise of any kind). This is, I imagine, what leads to the (surely true but strange) result of Theorem 6. That more observations does not reduce variance is somewhat of a red flag that the model & estimator would not be so useful in practice. Adding noise terms to these models would make for a stronger more practical estimator.

3. Finally, and this is less crucial, knowing beta is unlikely to hold in practice. And even so, ramp-ups/roll-outs rarely have more than 5-6 stages, so it is unlikely we would ever be able to estimate a polynomial of large degree, which would be necessary if neighborhoods are large. Of course, low degree polynomials can approximate large degree ones, but I did not see any results about the quality of these approximations. It is also my impression that Lagrange interpolation is not very robust to these kinds of mispecifications. 

questions: N/A
limitations: See above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a staggered rollout randomized design for estimating the total treatment effect with network data. The proposed estimator does not require knowledge of the network. Based on an assumption of neighbourhood  interference (low degree interactions among neighbours of a node), the total treatment effect (TTE) is estimated unbiasedly along with its variance. The key theoretical result depends on a relation between staggered rollout and polynomial extrapolation. The results on synthetic data show superior performance of the proposed method against baselines.
strengths_and_weaknesses: strengths

(A) staggered rollout random design to estimate TTE
(B) Novel use of Lagrange interpolation along without staggered design to propose the estimator of TTE
(C) The knowledge of the network structure is not necessary for the TTE estimator
(D) Key theoretical results showing the unbiasedness of the proposed estimator along with $O(1/n)$ variance

weaknesses


(A) The necessity of low polynomial degree assumption (Assumption 2) is not entirely clear to me. A bit more discussion about the requirement of this assumption would have been useful.
(B) Some time complexity results of the proposed potential outcomes model against the least square regression would be great!
(C) A separate subsection in Section 3 with the theorems, results and discussions on them would be appropriate
questions: Following are my queries for the author(s)

(A) Any heuristics why do the count k_1 only appears in the bound in Theorem 3 for completely randomized design with distinct treatment counts?

(B) Is there a scope of considering sparse networks (which would essentially put some restriction on the degree of the network) as real world networks are often sparse?

(C) With staggered rollout, How do the author(s) take into account the time varying effect that may gradually come in the network with time?
limitations: Yes the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an unbiased and consistent estimator for the total treatment effect under network interference (under assumptions on the neighborhood that each unit's POs get affected by). The TTE can be written as a polynomial and the estimator leverages staggered rollouts to learn the coefficients of this polynomial.
strengths_and_weaknesses: Originality and significance:
One of the main differences from prior work seems to be that the network structure for interference is not necessary, and an assumption on the neighborhood size (captured by \beta) is needed. I think the work should be interesting to the community as the staggered rollout scheme seems applicable to a variety of problems.

Clarity and quality:
Overall, the paper is well-written and the main ideas are explained clearly. There are a several typos and some notation is incomplete: so proofreading would help. For example, the PO equation after Line 135 is wrong (it should be in terms of z, j, and j'). In Line 123, d_{out} is not defined.
questions: How does the variance of the estimator depend on p_{\beta}? In other words, how close does the final staggered rollout need to be to 1 in order for the estimator to perform well? The bound in Thm. 2 is agnostic to the actual values. I think it would useful to include p_\beta in the bound to see how it scales with that value.

Why is monotonicity of the treatments in the rollout required? A discussion of this assumption would be useful given that it leads to correlations of the outcomes across time steps. It is not clear what role it plays in the estimator or why it is needed.
limitations: The estimator only uses the outcomes of the units. However, in most applications (including the ones stated in the introduction), one would have access to covariate information which can potentially be used to improve precision. The current estimator ignores this and it is not clear that it can easily be extended to account for this.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the problem of estimating the total treatment effect under network interference. The authors propose a method that utilizes a stagger rollout design without knowledge of the interference graph. The estimators are constructed using the Lagrange interpolation formula. The authors establish that when the potential outcomes are low degree polynomials, the proposed estimators are (nearly) unbiased with variance goes to zero. They demonstrate the performance of the estimators through simulations on synthetic data.
strengths_and_weaknesses: Strengths

-	The paper is very well-written. The proposed method is explained in a very nice way. The theorems are stated clearly.

-	The paper addresses an interesting and important question: estimating the total treatment effect under network interference.

-	The proposed method is very elegant and simple to implement. 

-	The proposed method appears to have good performance both theoretically and empirically. 

Weaknesses

-	I don’t think the assumption of a low dimensional polynomial is very well motivated. Is the assumption practical in real life applications? Can we verify them in practice? (See more details in the questions part) 

-	There is no discussion/experiment on the performance of the method under model misspecification. 


questions: -	Related to above, I don’t think the assumption of a low dimensional polynomial is very well motivated. The choice of low dimensional polynomial seems a bit arbitrary to me. I agree that higher degree polynomials can fully capture the structure of potential outcomes (lines 137-138), but the theoretical results can only hold for fixed beta not growing with n. Even for a beta scale like log(n), Delta_p will be small and the variance term in the theorems won’t go to zero anymore. My question is: is there any application in practice such that the low dimensional polynomial (beyond linear model) is a reasonable model for potential outcomes? Is there any way to verify this assumption in practice?
-	As pointed out by line 269-276, when the treatment probability is small, the value of interpolant at 1 can be very unstable. Is it possible to consider some other basis? For example, fourier basis, cubic splines etc. 
-	How would the method perform if the low dimensional polynomial assumption is violated? It will be very interesting to see some discussions or experiments on the performance of the method under model misspecification. 
-	Some typos: the equation between line 135-136: the variables should be z_j instead of x_i; line 139: “significantly smaller THAN …”

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper proposes a staggered rollout randomized design for estimating the total treatment effect with network data, without requiring the knowledge of the network. This is an interesting problem that appears in many real world settings. The paper has many strengths and few weaknesses. I believe that the weaknesses can be addressed based on the authors' response. I strongly advise the authors to revise the manuscript based on the received comments when submitting the final version. 
confidence: Certain
Decision: Accept