Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this work, the authors propose an approach, MIINT, for identifying infected individuals using a network-based approach. They also suggest two key properties, potency and similarity among groups, which impact the efficacy of MIINT and similar approaches. A detailed simulation framework is used to compare MIINT to relatively weak baselines. The simulation results show that the MIINT modestly outperforms the baselines; the results also confirm that all approaches degrade as expected as potency decreases and the similarity among groups increases. Experimental results on a (private) real-world dataset are somewhat mixed, but show that MIINT achieves a better true positive rate at an acceptable false positive rate.

The proposed approach formalizes an intuitive understanding of the spread of an infectious disease in (as far as I am aware) a novel way. The authors also identify and empirically evaluate the conditions under which the model breaks down. I also appreciate the effort of creating a simulation model to reflect how the spread may develop over time; I believe the simulation framework could also be useful for other researchers in this area. The model may also be relevant for other network spreader domains, such as “influencers in social media” studies.

My main concerns about the work are about the experiments. First, some sort of graph-based model should be included, such as label propagation or locally-linear embedding (followed by some classifier); of course, standard graph neural network approaches could also be included in a different, more empirical evaluation-oriented study. With the current setup, it is not clear if MIINT in particular, or any approach which accounts for the graph, would outperform the baselines.

Second, it is not clear to me of the practical significance of the “isolation policy” results. Even using an oracle, Figure 1 seems to suggest that between 30-40% of the population would need to be tested to see a reduction in the infection rate. Is that actually reasonable, even in the “closed” clinical environment recommended by the authors in the conclusion?

Minor comments
----------------

The references are not consistently formatted.

Neither the paper nor the supplement describe the features available in the real data.

It seems a bit strange that the AUROC and AUPRC in Table 1 are almost perfectly negatively correlated. Is there any explanation for that?

While I found the text descriptions and intuitions very helpful, all of the in-line equations made parsing many of the paragraphs in Sections 3, 4, and 5 difficult.

“outcomes, we refer” -> “outcomes; we refer”
“If exposed individuals” -> “If exposed, individuals”
review_title: New model and strong simulation framework, somewhat weak baselines

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper formulates the contagious disease into a missing label problem with dependence between each data point. The paper targets an important problem, especially in this pandemic, and the effort is greatly appreciated. However, the writing of this paper is confusing and it makes it hard to catch the main contribution of this paper. There are some concerns: 

1. The formulated problem sounds like a node label missing problem in a graph, where the node is patient (x, y) and the edge is whether they are contacted (e). In so, the paper is actually predicting the label of each node. If my understanding is correct, I am not sure why the authors choose the current formulation rather than graph one. 

2. The notation and wording are sometimes confusing, especially when I only have limited knowledge in healthcare. For example, even after reading section 4.1, it is still confusing what kind of special data structure the author is indicating. I recommend the author give a more intuitive or even graphic explanation in the paper. 

3. All the experiments are self-compared and make the result less convincing. For example, it is not clear whether different NN network would result in different performance because only NN is fixed here. 



review_title: Predicting infection on structured data with missing testing 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Interesting Paper; Some Confusions

This paper considers the problem of predicting the infection status of untested individuals. The authors propose a recurrent neural network-based model to impute the infection statuses over time, incorporating information on both the individuals (features of the individual) as well as a contact network that defines which individuals are related to each other.

The problem the authors describe is interesting and important. It has seen substantial attention in the ML-Healthcare community in recent years. Nevertheless there are some confusing aspects to the paper that make it hard to evaluate the efficacy of the proposed method.

Comments as they appear in order in the paper (comments with higher importance denoted with **):

The authors' use of the variable e for exposure state is a bit confusing and inconsistent. In S3P1 they define e as a sum of contacts' exposures and so it should be an integer. Later in S4P1, they describe a failure mode in which e_a = .9 and e_b = .1, which would suggest the authors are viewing e as on a spectrum from 0 (little exposure) to 1 (a lot of exposure). It's also not clear, throughout, how exposure (and the features x) relates to infection; are there example joint distributions or conditional distributions of y given e that could describe that relationship? Is it being treated merely as another feature (normalized/standardized for the purposes of the NN model) or does it have some special status in how it relates to x^bar and y?

**In S3P2, the authors define biased testing as entailing p(Xt | Ot = 1) \neq P(Xt | Ot = 0) \neq p(Xt). This is rather non-intuitive and it would make more sense for the variables to be flipped: there is bias in testing if the probability of being observed (i.e. tested) is different depending on the patient features. Can the authors clarify why they've chosen to define bias like this?

**In S3P2 the authors state that they assume i's outcome is independent of their contacts (entirely? just the contacts' outcomes?) given xi. Can the authors clarify why this might be a reasonable assumption? The authors, for instance, don't specify any notion of time in making this assumption and so really it seems to be saying y_i^t \indep y_j^t | x_i^1, x_i^2, ..., x_i^t. This is potentially problematic though since it would make sense for y_i^t to depend on y_j^{t-1} (my contact's true infection status yesterday affects my infection status today) and it's obvious that y_j^t and y_j^{t-1} should be dependent. The authors should consider providing a figure (e.g. non-causal DAG) that illustrates these assumptions. As is, I'm very concerned that either the assumption is not sufficient to eliminate network dependency biases, or that the assumption isn't valid in the data the authors are envisioning their method being applied to, or both.

S3P2: J_i^t \in D_0 \cup D_1 should be \subset (or \subseteq) rather than \in.

S3P3: The authors define the inverse probability of being tested as p(Ot = o)/p(Ot = o | Xt). This is also a bit weird of a way to define it: the probability of being tested is the authors' numerator P(Ot = o) and the conditional probability of being tested (or propensity for being tested) is the authors' denominator p(Ot = o | Xt) so it doesn't make sense that the inverse should be this fraction rather than just 1/(the numerator).

**S3P3: Using the authors' definition of the inverse probability of being tested, it's not clear how they obtain the risk in Eq. 1. Starting with the risk in the paragraph above, Rf = E[l(f(Xt), Y^{t+1})], conditioning on O = 1 corresponds to dividing by p(O = 1). To balance this, since the LHS of Eq. 1 is the same risk Rf, it makes sense that the authors multiply the loss by p(Ot = o), the numerator of w. It is not clear how dividing by the denominator of w (thus just multiplying the whole loss by w) results in the same risk. It's unclear to me whether this is a confusion on probability algebra or these some other assumption being made here (overlap should not be sufficient).

S3P4: "However, the following reweighted empirical loss is an unbiased estimator of (...)" -- how do we know it's unbiased? Is there a proof?

S4P1: What is the reason for adding 'hats' (^) to the y's in the sentence "This becomes obvious if we breakdown \hat{e}_i^t draws from Q as follows (...)". Is the idea that we're iteratively plugging in the predictions for the contacts' infection status? This seems to be the first time in the paper this sort of idea is hinted at and so it's not clear what the estimate is meant to be.

**S4.1P1 "We stress that we do not require \hat{y} to be an accuracte estimate of the true labels, but only require that there is significant separation between the imputed values (...)" Is this formalized anywhere? Can the authors please provide equations and math to provide intuition for why this might be the case if not a formal proof? Empirical evidence isn't exactly enough since it's feasible for there to be other explanations for "good" performance.

**S4.1P2 The same comment as above applies to the hand wavy description of the cluster assumption at the bottom of the next paragraph. "The clusers assumtion states that (..) exposure states tend to form near discrete clusters" -- what's meant by 'tend to'? More intution would be nice here.

S5P1 "Let A_i^t be the set of ancestors" -- in what sense? Is there a (potentially causal) graph underlying your model? If not, what is an ancestor?

S5P1 What is \mathcal{U}^t? This doesn't appear to be defined

**General: What assumptions are being made about the missingness mechanism for the true infection states. The authors should be very clear by stating something along the lines of "we're assuming missingness at random". That seems to be what the authors are getting at with their conditional independence assumption but i) it's not obvious that assumption is sufficient (see above) and ii) the authors haven't done a great job of linking that assumption to their argument for the efficacy (identification) of their model.

S5P2/3: The authors describe the method by which they determine convergence of model fit -- I'm curious, given that they cite Chernozukov 2017, what properties their model has wrt sample efficiency and convergence to the "truth" (i.e. the parameters of their model converge to the true parameters of the function that maps X to Y)? In general, I'm not aware of theory that gives convergence rates for neural networks in the way Chernozukov discusses and so I'd be concerned the authors might be picking an arbitrary convergence cutoff rather than something principled.

S6.1P1: "Our goal here is to highlight how MIINT can be used to inform testing and isolation policies that lead to reduction in infection rates" -- the authors should be _very_ careful with using this sort of language. It implies a sort of ground truth causal efficacy to the work when this is not causal work. I agree that semi-supervised approaches are very similar to causal methods but the authors do not appear to spend sufficient time considering the assumptions necessary to ensure valid causal inferences, rather than simply fitting a black-box prediction model.

Simulation study: what is the mechanism for deciding whether somewhat gets tested? It isn't obvious in the description how this selection works. Additionally, from the description, it's not exactly clear what the probability of actually being infected is, given the features (from MNIST) and the contacts (indirectly from MNIST?)

Sensitivity to the potency property: "This confirms out conjecture (...)" The authors should be careful with such strong language based on empirical, simulated evidence.

Minor:
Throughout the paper there are several instances of incorrect grammar or other writing issues that make it hard to determine the meaning of the sentence in question. For instance in the first paragraph of S2, the authors say "In the machine learning literature, previous work has relied on proxies for exposure, e.g., the prevalence of a disease in a community (citation), and implicitly assume that conditioning on individual characteristics." The authors should carefully proofread to remove these artifacts as there are some instances where they pose a more substantive impact on the readability of the paper.
review_title: Interesting Paper; Some Confusions

final_score: rating: 3: Clear rejection
review_content: This paper focuses on contagious disease prediction with the consideration of observed data bias and patient exposure. The authors present a Model for Infections under Incomplete Testing (MIINT) and the experimental results show the proposed model outperforms baselines on *some* metrics.

Problems:
1. The paper is not well-written and pretty hard to follow.  
- It is unclear why the authors use w^1(X) in eq (1). Why should t be set as 1?
- It would be better if the authors gave the definition of \hat{w}^t_i in eq (2).
- Q(D^t_1) is a set of imputed distribution. It is unclear how to generate Q.
- It would be better if the authors use consistent notations in different places. such as f(X,Y) and f(X^t, Y^{t+1})

2. The authors want to consider the exposure relationships between patients. An intuitive solution is graph-based models. It would be better if the authors compared the proposed model with graph-based models, e.g. [1]. Moreover, exposure states are not fully considered. The authors just simply use the count of exposure of observed true infection patients at the last time point. The exposure before two time-points is ignored.

[1] Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. "Semi-supervised learning using gaussian fields and harmonic functions." Proceedings of the 20th International Conference on Machine learning (ICML-03). 2003.

3. In Table 1, the two simple baselines (OM, NEM) and POM outperform the proposed model a lot on AUPRC. It is worth explaining why. 

4. Due to missing technical details (especially how to generate Q), it is hard to re-implement the proposed models. It is necessary to provide code and data as supplementary materials.
review_title: A paper lacks necessary technical details and comprehensive comparisons

Meta-review: This paper studies a timely problem and consider an interesting approach, but overall there were many concerns about technical details and the validity of the framework. The positive reviewer also mentioned concerns about the experiments, which others also found to be an insular comparison with weak baselines. Following the response period, in discussion there are additional concerns arising related to the lack of details, for instance related to possible unidentifiability of the model. As one reviewer discusses,  the authors are attempting to use RNNs to impute missing infection status labels when the missingness mechanism is assumed to be (i) not at random, (ii) playing out over time (as it is unclear whether Y^t is assumed (conditionally) independent of Y^t' with t' << t), and (iii) subject to interference (whether someone is tested is the 'treatment' here since it's a missingess problem and one person's propensity to be tested could causally affect another person's downstream infection status since apparently no Markov independence is assumed. There is also consensus that the writing quality can be greatly improved. Overall this work contains some ideas with potential in a thorough revision 
Decision: Reject