Exploiting structured data for learning contagious diseases under incomplete testing
Authors: Maggie Makar
Lauren West
David Hooper
Eric Horvitz
Erica Shenoy
John Guttag
Venue:  (2021)
Keywords: infectious diseases
neural networks
healthcare
regularization
structured data

Abstract: One of the ways that machine learning algorithms can help control the spread of an infectious disease is by building models that predict who is likely to get infected whether or not they display any symptoms, making them good candidates for preemptive isolation. In this work we ask: can we build reliable infection prediction models when the observed data is collected under limited, and biased testing that prioritizes testing symptomatic individuals? Our analysis suggests that under favorable conditions, incomplete testing might be sufficient to achieve relatively good out-of-sample prediction error. Favorable conditions occur when untested-infected individuals have sufficiently different characteristics from untested-healthy, and when the infected individuals are "potent", meaning they infect a large majority of their neighbors. We develop an algorithm that predicts infections, and show that it outperforms benchmarks on simulated data. We apply our model to data from a large hospital to predict Clostridioides difficile infections; a communicable disease that is characterized by asymptomatic (i.e., untested) carriers. Using a proxy instead of the unobserved untested-infected state, we show that our model outperforms benchmarks in predicting infections. 

Under review as a conference paper at ICLR 2021
EXPLOITING STRUCTURED DATA FOR LEARNING CON- TAGIOUS DISEASES UNDER INCOMPLETE TESTING
Anonymous authors Paper under double-blind review
ABSTRACT
One of the ways that machine learning algorithms can help control the spread of infectious disease is by building models that predict who is likely get infected making them good candidates for preemptive interventions. In this work we ask: can we build reliable infection prediction models when the observed data is collected under limited, and biased testing that prioritizes testing symptomatic individuals? Our analysis suggests that when the infection is highly contagious. incomplete testing might be sufficient to achieve good out-of-sample prediction error. Guided by this insight, we develop an algorithm that predicts infections, and show that it outperforms baselines simulated data. We apply our model to data from a large hospital to predict Clostridioides difficile infections; communicable disease that is characterized by asymptomatic (i.e., untested) carriers. Using a proxy instead of the unobserved untested-infected state, we show that our model outperforms benchmarks in predicting infections.
INTRODUCTION
Preemptively identifying individuals at a high risk of contracting a contagious infection is impor- tant for guiding treatment decisions mitigate symptoms, and preventing further spread of the contagion. In this paper, we study how to build individual-level predictive models for contagious infections while explicitly addressing the challenges inherent to contagious diseases.
Building accurate infection prediction models is hindered by two main factors. First, contagious infections defy the usual iid assumption central to most machine learning methods. This is because an individual's infection state is not independent of their contacts' infection states. Previous work has often relied on expert knowledge to construct exposure proxies (Wiens et al.) 2012] Oh et al.] 2018). It is then assumed that conditional on the exposure proxy and individual characteristics, individual outcomes are independent of one another. Second, the observed data biased due to incomplete testing. We use the term "incomplete testing" to describe the scenario where only small, biased subset of infected individuals get tested. Such a scenario is ubiquitous in the context of contagious infections for several reasons. While many individuals carry the pathogen, only a fraction display symptoms. Even in the presence of unlimited testing resources, the latter are far more likely get tested leading to biased data collection where individuals predisposed to displaying symptoms are over-represented. Incomplete testing makes learning accurate models difficult since the collected labels are missing not random leading biased, inconsistent estimates.
In this work, we treat non-independence of outcomes as a blessing rather than a curse. Our proposed approach leverages the fact that an individual's infection state provides useful information about their contacts' true infection states. This information is used to generate pscudo-labels for untested individuals, mitigating issues due to incomplete testing. The key idea behind our approach that highly structured patterns of contagion transmission can serve as complementary signal to identify even untested carriers. The stronger that signal is, the less impact that incomplete testing will have. Our contributions can be summarized data that can be exploited to mitigate the effects of incomplete testing. (2) We propose an algorithm that leverages that insight to predict the probability of untested individual carrying the disease. (3) We empirically evaluate the effectiveness of our method on both simulated data and real data for common healthcare associated infection. We show that predictions from model can be used to
follows: (1) We identify two properties of the collected

---

Under review as a conference paper at ICLR 2021
inform efficient testing and isolation policies. Using real data, we show that our model outperforms baselines in the task of predicting a hospital associated infection.
2 RELATED WORK
Infectious disease modeling. Modeling the transmission of infectious diseases has been extensively studied in the epidemiology literature using SIS/SIR models and several other variants (Kermack & McKendrick] 1 These epidemiological models focus on the aggregate levels of infections in a community, which is distinct from our approach here where we focus on predicting individual level infections. In the machine learning literature, previous work has relied on proxies for exposure, e.g., the prevalence of disease in community (Wiens et al.) 2012] Oh et al. 2018), y and implicitly assume that conditioning on individual characteristics. Similar to our approach, Fan et al. (2016) and Makar et al. (2018) take into account structured data, namely contact networks to compute infection estimates (Fan et al. 2016 Makar et al. 2018). We differ from these approachs in that (1) we do not make parametric assumptions about the joint distribution of the observed or latent variables, and instead use nonparametric models (neural networks) to model the infection states, (2) we do not assume infections will become symptomatic as is done in Fan et al. (2016), and (3) unlike the approach taken by Makar et al. (2018), we model time evolving sequences of infections taking into account the exposure states of potential asymptomatic carriers.
Semi-supervised learning. Our proposed approach relies transductive reasoning to generate labels for untested individuals. that, it is closely related semi-supervised learning methods, such as pseudo-labeling (Lee 2003), and self-training (Robinson et al. (020). However, in traditional pseudo-labeling, the transductive power comes from the fact that points similar to each other in the
input space should have similar outputs. Here, the rich structure in the data allows for more: we can construct pseudo-label for untested individuals not not by relying on their similarity to other labeled instances, but also by observing their observed contacts' infection states. Our empirical results, and analysis are similar in spirit to concepts presented in the semi-supervised literature, specifically the cluster assumption, which we discuss length later (Seeger 2000 Rigollet 2007).
Graph Neural Networks. Our proposed approach incorporates knowledge of the contact network. In that it is similar to Graph Neural Networks (GNNs), which utilize relational data to generate prediction estimates (Zhou et al. 2018). GNNs fall into two categories, the first relies on transductive reasoning and cannot generalize to new communities (e.g., Kipf Welling (2017) or inductive, which can be used to generate estimates for previously unseen graphs D..gg,Hamilton,. et al. (2017). Our work similar to the latter category with an important distinction: our approach leverages unlabeled data giving more accurate, and robust estimates.
Our work can be viewed as combining the strengths of semi-supervised learning, and GNNs to address limited testing. In addition, our approach augments the strengths of those two approaches with ideas from domain shift, and causal inference such as importance weighting (Cortes et al.) 2010) to address biased testing.
PROBLEM SETTING
Setup. Let y' 1) denote individual's true infection state time t, with y' 0 if an individual is not infected and if they are. We use x' E X to denote a vector of the individual's features at time and define J to be the set of indices of i's contacts time t. We assume that contact indices are known, i.e., that the contact network is observed. Let ef E Ryo denote i's exposure state, with ef Σje,; u5- The exposure state is fully observed only when all of i's
contacts have been tested, but otherwise either partially observed or unobserved. Define x = x'lle', where || as the concatenation operator, i.e., x² E X T2 Let of E (0. 1} denote the observation state, with of = 1 if individual's label is observed. i.c.. if the individual has been tested for the infection. We use the super-script: i to denote variables from time = 0 up to and including t, e.g., x = =[x",.. "x",...,"* x", .x.,.'"'',
Throughout, we use capital letters to denote variables, and small letters to denote their values. We use P(x Ot, y to denote the unknown distribution over the full joint. Under biased testing, we have that O( = 1) # P(X' = 0) * P(X*). We assume that 0 < P(O = X x) < 1, for all x X. and o {0, This is the same the overlap assumption
2

---

Under review as a conference paper at ICLR 2021
in causality literature. In addition, we assume that i's outcome is independent of their contacts given x,, which is itself a function of the contacts' outcomes, we refer to this as the conditional independence assumption. We consider the case where we have access to (1) labeled (i.e., tested) }} P(X yt-110' = 1), and (2) an
set of individuals D = [Di)_= = {(x,, ut),,.. ... (x Fzy
unlabeled (untested) set of individuals Do = (D))L= = (x,...., X, ~ P( = 0), such
that for each E DDUDI, and each / [0. 7). we have that J E DD.DD. will also be convenient to use Lt' to denote the set of indices of untested individuals at time L.
Learning objective. We are interested in learning / xxx"" - y - To focus the discussion on the novel component of our approach, we consider a setting where we are only interested in pre- dicting the outcomes for a single time step. It will be particularly useful to consider the task of making predictions for t 2, using data from & = 0, 1, dropping the time superscript when it can be inferred from the context. We present the full model predicting infection sequences over time in section 5 Let & be the logistic loss function. Our goal is to find ʃ E F, where F is some hypothesis space such that the risk of incorrectly classifying the infection state R2 Ex Σx (e(((x)), y is minimized. We briefly consider scenario where we have oracle access to the true exposure states but we return to the more realistic, non-oracle scenario later. Under the conditional independence assumption,
break down the risk to the sum of independent losses. Define the inverse proba- bility of being tested, w² (X) = P(O o)/p(o* = af ), following Robins (1998). and Robins et al. 2000). Due to the overlap assumption, and under biased testing, we have that:
R = R. = =x,, (x (())) Y)JO = 1],
(I)
(Cortes et al. 2010). The reweighted risk simply places higher importance the loss of individu- als who are unlikely be tested. Ry, cannot be directly computed since the expectation is defined with respect to the unobserved distribution. However, the following reweighted empirical loss is an unbiased estimator of Ry::
c(f)
Σ SED
w(((f(x).v
by Cortes et al. (2008), where w/ = p(0' = of)// xx), p(0 of) is the empirical estimate of P(O o). and gio x is the estimated probability of getting tested conditional on individual characteristics. Without oracle access to exposure states, the samples x² P( = 1) are incomplete. This is because x includes et, which is a function of v/ : Jí. We only fully observe eq. and hence x for individuals whose contacts have all been tested. To address this, define (DD)), a set of partially imputed distributions that are consistent with the labeled samples. It is the set of all possible distributions over the (partially) unobserved eç. Our risk now defined with respect to both Q, and f, and our task is to find Q and f, such that the following empirical risk minimized:
e(S
Σ (()
LED
where & = x' and ē ~ Q, and of = r(0 (2). o.). Minimizing this objective prone to extreme overfitting. To see why, consider some Q that sets of = 100 for every i of = 1, yf ++1 1, and 0 for : of = 1.
O. Since Q is essentially leaking the true label into the input space, is trivial to find some f that takes in the imputed inputs, {(X), 100, v, (f+1) " and {(x;, 0, v/ U and gives perfect performance. Such an f is clearly expected to have poor generalization error. We next consider how to leverage existing properties of the problem as efficient regularizers.
4 EXPLOITING STRUCTURE AS A REGULARIZER
We seek to constrain the candidate sets FF and ((Di) to ensure that such pathological overfitting is avoided. To do so, we exploit the structure in the data, namely the interdependence among individu- als' infection states, and the availability of unlabeled data. Recall that the exposure state is the sum of the contacts' infection states. This means that when we draw ě from Q, we are implicitly draw- ing a label for i's contacts' outcomes, by definition of ē This becomes obvious if we breakdown o
3

---

Under review as a conference paper at ICLR 2021
11 - =0 Vhg and Vi, Q, such that Vi, E [0. 1]. This breakdown immediately implies two properties that should hold for "good" Q's. First, Q should assign the same estimate for the same individual. Consider the case where two i i= a, i = come into contact with the same individual j, who has not been tested, For simplicity, suppose that a, and b have other contacts. In this case, one failure mode would be if Q assigns ea = D.. = 0.9, and ej = Hi, = 0.1, which constitutes two "votes" on the true state of j; one vote by each contact. Second, note that Q implicitly assigning pseudo-labels for the infection states of untested contacts, this means that Q's imputed labels should be similar to the labels predicted by f. Going back to the previous example, besides having Vo.5 = 0.9, Th.3 = 0.1, we also have (x²) 0.4. A good regularization method should then explicitly encourage the pseudo-labels to be similar the estimated labels from f. and hence implicitly penalizing varying estimates for the same individual. This intuition is encoded in the main loss in our proposed approach:
draws from Q as follows: e E
f.Q" = min Σ Rof=
)
E JEntrue
i (((1((1(())))) (). (3)
S.Q 1i
nu
where denotes the set cardinality, X 2 0, and are parameters to be picked via cross validation,
ih, ~ Q", and é = Ejer(s) 1 of = 1} 5f + 1 : of = 0} fh.5 When À > 0, this objective is somewhat similar to pseudo-labeling (Lee 2003). it would encourage the votes of each of j's contacts to conform with the prediction from f. and implicitly with one another. When X= = O. equation 3 prioritizes finding good predictions for the labeled data, ignoring possible structure implied by the data. Note that in the second term in equation 3 we have f(x)-1) f(x)-)) meaning we assume not imputed exposure component for contacts at time t This is only because we are considering the simple setting where we make predictions for 1+1= = 2, meaning at ( 1-1 = O, this is the beginning of the observation period and we cannot impute exposure yet. We consider more complicated settings where the contacts' inputs also include an exposure state later.
4.1 WHEN DOES STRUCTURE WORK AS A REGULARIZER?
We now ask: when do we expect equation 3 to yield models superior to those that ignore structure? First, if the imputed ÿ.j concentrates around significantly different values for j 3/ = 1, and j Yj 0, then we expect minimizing equation 3 to yield better models. We stress that we do not require i to be an accurate estimate of the true Tabels, but only require that there is significant separation between the imputed values for untested-infected individuals and untested-uninfectee individuals, i.c., they are distinguishable. This distinction means that even noisy and inaccurate estimates of ÿ can be sufficient. We expect such high separability to exist, even in settings of low and biased testing if observed data satisfies a property which we will refer to as carrier potency. The carrier potency property can be viewed as an extension of the margin condition in classification (Tyybkkvv et al., 2004] Audibert et al., al.. (007):: it states that there are few j's who have contacts with ambiguous infection states. In other words, infections cluster so that infected-untested individuals Such a condition will be satisfied if the infection highly contagious. We refer this property as the potency property.
tend to have many more infected contacts than do miifected-untester
Second, even if the imputed allows high seperability, but x makes it difficult to identify a learnable mapping from & to g, then minimizing equation 3 instead of the objective on the labeled data only does not help. Such is the case when untested-heall healthy and citested-infecter individuals "look" the same, meaning they have very similar characteristics and exposure states. This property is often referred to as the cluster assumption in semi-supervised learning literature (Rigollet 2007; [Seeger] 2000). The cluster assumption states that individual characteristics, and exposure states tend to form near discrete clusters, with homogeneous labels within each cluster. We refer to this property as the dissimilarity property.
The degree to which these two properties are satisfied in the observed data will depend on the infection being studied, the environment in which it is spreading, and the quality of data collection, among others. Importantly, we show empirically in section 6 our proposed approach "does no harm" in that in the worst case scenario, when these two properties degrade to the point of non- existence, our model performs well the best baseline.
4

---

Under review as a conference paper at ICLR 2021
5 PROPOSED METHOD
Our proposed model, Model for Infections under Incomplete Testing (MIINT) leverages la- beled and unlabeled data in order to predict sequences of infections over time. MIINT mini- mizes a slight variant of equation
which is modified to predict sequences of infections. Let A be the set of ancestors of at time t whose outcomes are unobserved, i.e., A (i) nu', A-11= U (i) n etc. The loss at time / is defined as: JEA!
C
Σ IED:
( > >..0).)))) JEAG
(4)
11
and the final objective is find f*, Q", such that: f².Q* m D C'. We assume that I does not vary over time (though that is an assumption that could be relaxed), and take F to be the space of recurrent neural networks (RNNs), which are ideal for modeling sequences of data. We propagate the predicted state forward in time, meaning takes in x", e' and ÿ' to predict gt+1. This ensures that exposures at time- < & are taken into account when predicting at time t. Note that equation caan be broken down into the independent sums of individual losses, well their ancestors' losses. This means we can use stochastic gradient descent, with gradient updates defined with respect to mani-batches is typically done. One limitation is that equation E as stated would require keeping track of all the ancestors' states since I = O, which can be prohibitive. For long observation periods, we suggest considering subset of A up reasonable time limit.
The algorithm used to train MIINT, similar to pseudo-labeling (Lee) 2003), inherently an ex- pectation maximization algorithm, where we iterate between computing the expected label for the untested samples (i.e., finding the optimal Q, and identifying the optimal of f that maximize the likelihood of the observed labels under Q) until convergence. Convergence is achieved when the change in loss defined over the samples with observed labels in held out validation set < c for some small e. For our purposes, we find it sufficient to let Q be a deterministic function rather than an actual distribution. However, our approach is trivially extendable to allow Q to be distribution, for example using techniques described in Tran et al. (2017). All models presented in this paper are implemented using Tensorflow (Abadi et al.) 2016).
Finally, recall that we need to estimate of = p(O o o.). We follow Chernozhukov et al. (017) in using independent sample to estimate g. Importantly, g depends on x. So we follow an iterative process: after every epoch of training, we use the most updated f to estimate the unobserved outcomes in the validation set, and hence get estimate for è and x for the independent weighting sample. We use these imputed values to learn an updated The updated g provides estimates for the weights of the training samples of the main prediction model, which are used to reweight the loss function for the next epoch, and so forth.
EXPERIMENTS
We evaluate our model on a simulated and a real data setting. In the simulated setting, unlike the real data setting, we have access the true infection state, which allows to evaluate the performance of the model and baselines under different patterns of infection. In both settings, we present results from our model (MIINT) and four baselines: (1) Optimistic Model (OM): model that assumes that all unobserved labels are = 0, (2) No Exposure Model (NEM), a model that ignores exposure, and attempts predict infections solely based on the individual characteristics, (3) GraphS (GNN) graph neural network that takes into account the contact network, and observed infection states (Hamilton al. 2017) but ignores untested individuals, and (4) Pseudo-Labeling (PL) a semi- supervised learning method that takes into account untested individuals but ignores the rich graph structure (Lee) 2003).
For all models, we weight the loss from each individual by the inverse of their estimated propensity to be tested, wt. which estimated using an independent sample following [Cernozhukov et al. (017). For our model, we use the iterative weighting technique outlined in section 5 We also present an ORacle Model (ORM): an unrealistic model that has oracle access to the true labels for the whole population. For all these models, we keep the neural network architecture fixed. We use cross-validate to get the values of X, т. Results from unweighted models, and details about cross-vall validation and network architecture included in the supplement.
5

---

Under review as a conference paper at ICLR 2021
6.1 SIMULATION EXPERIMENTS
Our goal here is to highlight how MIINT can be used to inform testing and isolation policies that lead to reduction infection rates, as well as empirically validate our conjectures regarding favorable properties under which MIINT is expected to be superior.
Setup. We simulate world where there are three types of people: asymptomatic if exposed (G1), and immune (G2). If exposed, individuals in group Go become infected and symptomatic, hence they are more likely to get tested. If exposed individuals in group G1 become infected without displaying symptoms. This group is unlikely to get tested. Finally, G2, the immune group is unlikely to get the infection even if exposed. To simulate individuals' characteristics (i.e., x), we map the distinct groups to distinct MNIST digits. We use MNIST images because (1) they provide a complex input space compared to randomly generated data, and (2) images can be easily classified as similar or dissimilar, which enables where the dissimilarity property can be easily manipulated as described later.
symptomatic if exposed (Go),
design experiments
Let vn denote the pixels of an MNIST image i. For Go we randomly sample without replacement 1/a-'' elements from the set {vi]id;=0 where n is the total sample size. For G1, and G2 we sample from {vg bid.ed and 1id, respectively. Note that the infection states will be different within each group, since infection also depends on the exposure state, and injected noise. We draw the edge sets (J((i))en..eejo.r according to a stochastic block model, parameterized by the matrix B, where BR, is the probability that an individual from G2: forms an edge with an individual from Gt. B is important in simulating different levels of carrier potency. When BL.t/B.a for l = {0, 1) approaches 1, members of the asymptomatic carrier group is equally likely to form edge with individuals who are infectable (Go) as with individuals who are immune (G²)). This is the unfavorable low-saparttion setting. On the other hand, if R1../B122 5, for example, individuals in Go, and G1 are 5 times more likely to form an edge with someone in a susceptible group as compared to forming an edge with an individual in G2. This is favorable, high-separation setting.
We mimic the situation where testing started after a significant proportion of the population has been exposed by randomly setting the true exposure state of 20% of the population to be 1 time t = O. Exposure for each individual e = Σje;; v/ 2 1. The true infection label y ++1 = 1ii e (Go; (i))-1(es = I]. We introduce noise by randomly flipping the labels of 1% of the population. If an individual tests positive at / < T. their label remains positive until L = T. We define Pabs to be the proportion tested, and hence their true label is observed. We pick the probability of observing i's label based on his true infection state, meaning, p(o:ly: = 1) * p(o./u = 0). For all the simulations, we set T
we draw 500 x samples for each of the training, validation, and testing sets. We simulate an independent sample to compute the weights wi, so we also draw 500 x 6 samples that will be used to train and validate weighting models. For each experiment, we draw 10 different datasets, and report the mean and standard deviation of the performance metric across the 10 draws.
Informing testing and isolation policies. Here, we highlight how our model can in- form efficient testing and isolation policies. We simulate biased and limited testing by setting p(o:l:s = 1)/poo = 0) = 5, and pobs 1 re- spectively. We set FL./BI.2 = 5, making it high potency setting where MIINT is expected to perform well. We mimic a situation where no isolation interventions are taken at training time. At test time, we fix a testing budget of at most Prant % of the total population on each time step. We use the predictions from each model to inform who gets tested by picking the top Pteat % with the highest predicted proba- bility of infection. Of those tested, individuals who are truly infected are "isolated" by setting their edges for the subsequent time steps to O. They are also taken out of the population eligi-
Rodjuctoon infection rate
0.7
Bandee
OBM
0.6
NEM
OM
0.5
Ft.
GNN
0.4
XTIV (Dves)
0.3-
0.2
0.1
0.0
0.2
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Proportion tested
Figure I: Reduction in infection rates relative to a policy that does not isolate infections (no-action policy) as the daily testing budget varies. Our model achieves the highest reductions in policy relative to all realistic (i.e., non non-oracle) models.

---

Under review as a conference paper at ICLR 2021
ble for further testing. We compute the infec- tion rate, TM for model M as TM = n '..E. max; Di,t- We define To as the infection rate under no-action policy, that is if no isolation interventions are taken. Our main metric of interest is the reduction in infection rate relative to the no-action policy = so aM /To. Figure [left) shows the reduction in infection rate the y-axis for different values of the testing budget Prest % on the axaxi.. In addition to the main baselines, we also show results from a random testing policy. The results show that for any given testing budget, our model outperforms all feasible baselines giving higher reduction in infection rates. Importantly, the results imply that our model is able to achieve near oracle infection control with only 70% testing, compared to 90% for the baselines.
In next two settings, we empirically validate our conjectures about the two properties which en- able our model to outperform others, and explore what happens as these favorable properties are weakened to the point of non-existence.
Sensitivity to the potency property. Here, we fix Pobs .1, and p(acb: 1/plox/ = 0) but sweep over carrier potency by varying the value of BLA/Bi, from I (low potency) to 5 (high potency). Figure lefft shows Bi.//B.2 on the x-ixxi and the AUROC on the y yxaxis. The plot shows that MIINT outperforms other baselines when there is high potency, and as potency declines, its performance becomes similar to that of the other baselines. This supports our conjecture that our regularization approach is advantageous when it is easy to impute the true infection states for individuals based on their contacts' infection states.
Sensitivity to the dissimilarity property. Here we examine what happens when the cluster assump- tion breaks down, meaning when untested individuals with similar characteristics have different in- fection states. We do so by moving the untested, and possibly serettediidiiiduuee to "look" similar to the untested-healthy. Specifically, we sample pairs of images { Mi)dd use VoxelMorph (Balakrishnan et al., 2018), a learning- based framework for deformable, pairwise image registration to learn a function that gives us a deformation field which we then apply it to pairs of images. moving V/ to look closer to Vj. Further details about the deformation process are in the original VoxeIMorph paper. Figure rightt shows the results of this setting. The x- p-axis can be viewed as the degree of similarity between the two untested groups with being dissimilar (i.e., the original images without any deformation) and I being very similar (i.e., all images of the digit look almost identical to 2's). The axisis is the average AUROC, We see that all models perform worse members in G₁ look more and more similar to those in G2. We also see that MIINT out- performs all baselines when the two groups are dissimilar, but performs as well as the others when difficult.
We then
the mapping from input space to label becomes
The last two experiments confirm our conjectures about the properties necessary for MIINT to per- form well, and imply that MIINT "does harm": at worst it performs comparably to alternatives, and at best it can give significantly better performance making it the superior alternative. Additional results examining the effect of bias and limited testing are in the supplement.
0.95
0.90
0.9
AUROC
AUROC
0.85
0.5
0.80
0.75
U.7
MINT (Duz)
5
0.1
0.7 Untested Population Similarity
0.3
0.5
0.9
Carvier potency
Figure 2: Left: Impact of varying levels carrier potency controlled by Br.+/B... Our model out- performs basclines, especially in cases with high potency. Right: Impact of high (=.9) and low (=.1) similarity between the characteristics of the untested- -nntttectenineeeedd and untested-infecet populations. Our model outperforms baselines when the two populations are dissimilar.
Individuals G1 only infected if they get exposed.
7

---

Under review as a conference paper at ICLR 2021
6.2 REAL DATA EXPERIMENT
Here, our task is to predict the onset of Clostridioides difficile infections (CDI) among patients in a large urban hospital. CDI is a contagious infection that attacks the gut, and causes over $00,000 in- fections annually in the US (Magill et al. 2014). As with most contagious infections, asymptomatic carriers of CDI exists and can contribute to the spread of the infection (Riggs et al. (007).
Setup. Using Electronic Medical Records, we extract daily characteristics of patients who were admitted to the hospital between 09/01/2012 and 06/01/2014. We follow similar inclusion criteria as Oh et al. (2018); Makar et al. (2018), outlined in detail in the supplement. We collect all patient characteristics available upon admission (c.g., gender, age, medical history) as well as daily charac- teristics (e.g., lab tests). We collect contact networks, where an edge exists if two patients in the same room on the same day or if they came into contact with the same nurse on the same day.
Here, we have partial access to the true infection states, since not all the patients are tested, making accurate evaluation of different models difficult. Therefore, we exploit testing protocols to construct proxy "true" label and proxy "observed" label. Whether a patient labeled as CDI positive or not is a result of two, or possibly three tests. First, an enzyme immunoassay (EIA) and Glutamate dehydrogenase (GDH) test are conducted. If the results of the two tests are discordant, a polymerase chain reaction (PCR) assay acts as tie- breaker. Previous studies comparing the outcomes of the two groups (those who have non-discordant HIA/GDH+ results vs. PCR+) have shown that the former experiences more severe complications (Origüen et al., 2018 [Polage et al.) 2015). This implies the EIA/GDH+ label can act proxy for symptomatic infections, whereas PCR+ might be picking up on patients who are carrying the bacteria but have low toxin levels and therefore mild or no symptoms.
|
For this reason, we hide the PCR+ labels during training, presenting them as untested individuals to all models. At test time, we set the target label = 1 if a patient tested positive via ELA/GDH or PCR and 0 otherwise. In addition to the baselines outlined in section 6.1 we allow one of models full access to the EIA/GDH+ and PCR+ labels, and refer to it "partial oracle" model (POM) since has access to the PCR+ labels, but not the full infection states. The latter are unavailable because the majority of patients are not tested. We also compare our results the state-of-the-ar prediction model for CDI, which is logistic regression that takes into account the varying importance of different risk factors over the hospitalization, and relies on medical knowledge construct exposure proxies. We refer to this model as the Expert driven Logistic Regression (ELR), details about the model are outlined in Wiens et al. (2012).
We split the data into 5 subsets based on time, The first subset holds 6 months of data and is used to train the main infection prediction models. The second and third subsets contain 5 months of data each, and are used for validation and testing of the main prediction model. The last 2 subsets are used for training and validation of the weighting models, and each contain 2 months worth of data. We report the AUROC, the True Positive Rate (TPR) at the threshold which achieves False Positive Rate (FPR) of 10%, as well as the Area under the Precision Recall curve (AUPR) on the test set.
Table [ shows the results of the models the test set. For several models, the unweighted model outperforms its weighted counterpart. We show the bet- ter performing version here, and index it with "-U" to denote that it is the unweighted ver- sion. Results from all mod- els, and results broken down by GDH/EIA+ vs. PCR+ are in the supplement. Standard deviations are calculated by taking 100 bootstrap replicates of the test set data. We see that MIINT outperforms all others on all reported metrics. The one exception is ELR: MIINT and ELR achieve comparable AUROCs but MIITT significantly outperforms ELR outperforms POM even though the latter has access to better labels. We hypothesize that this is because in addition to accurately estimating the PCR+ patients, it is also capturing truly untested in-
TPR @
AUROC
AUPRC
FPR=10%
POM
0.49 (0.014)
0.73 (0.003)
0.2 (0.004)
NEM-U
0.45 (0.009)
0.7 (0.006)
0.13 (0.001)
OM U
0.45 (0.012)
(0.005)
0.12 (0.004)
ELR
0.53 (0.008)
0.82 (0.006)
0.09 (0.002)
GNN
0.24 (0.005)
0.59 (0.005)
0.03 (0.007)
PL-U
0.58 (0.012)
0.78 (0.006)
0.18 (0.009)
MUNTTU
0.6 (0.007)
0.81 (0.006)
0.11 (0.002)
Table 1: Performance metrics for CDI prediction on the test set.
all other metrics. Unsurprisingly, MIINT
8

---

Under review as a conference paper at ICLR 2021
fections, and utilizing these estimates accurately impute the exposures of the EIA/GDH+ patients as well as the PCR+ patients leading to better performance metrics.
While POM, OM-U, NEM-U, and PL-U achieve higher AUPRC than model, figure 8 in the supplement shows that these models achieve a higher AUPRC than MIINT because they have high precision at low recall, whereas MIINT has the highest precision high recall values (greater than 0.5). High precision at low recall not necessarily useful in the infectious disease setting, where having low recall (meaning being unable identify the true infections) can be disastrous.
7 CONCLUSION
We presented MINT, a model that predicts contagious infections under biased and limited testing. It does by taking into account contact networks and the interdependence of individuals' out- comes, We identified two properties that determine the extent to which MIINT outperforms other approaches. The first states that the more contagious the infection, the better MIINT performs. The second is the degree to which characteristies of untested and infected individuals and characteristics of the untested and healthy individuals form discrete clusters-as an important property in general for semi- semi-supereised learning. Using simulated data, we showed that MINNT can be used to guide test- ing policies that lead to reduced infection rates, and that even if the two properties outlined above are non-existent, MIINT still performs well. We showed that MIINT outperforms baselines when applicd to real EMR data.
Because of the obvious relevance of our work to the current pandemic should note that our model is best-suited for infections which spread within clinical environments (e.g., hospitals and nursing facilities). Within clinical settings, it is easier track individuals' contacts, which is necessary for our model. While our model can theoretically be used to predict COVID-19 infections in the community, it would require unprecedented levels of contact tracing, and data collection. In conclusion, we believe this work is a first step down an important path. If predictive models are to play useful role in limiting the spread of contagious infections, they must take into account the interdependence of outcomes, and the fact that untested individuals are capable of spreading the disease before they have been diagnosed.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In Symposiun Operating Systems Design and Implementation, pp. 265-283, 2016.
Yves Audibert, Alexandre Tsybakov, et al. Fast learning rates for plug-in classifiers. The Annals of statistics, 35(2):608-633 2007.
Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. An unsu- pervised learning model for deformable medical image registration. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 9252-9260, 2018.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, and Whit- ney Newey. Double/debiaseddneymma machine learning of treatment effects. American Economic Review, 107 2017.
Corinna Cortes, Mchryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In International conference on algorithmic learning theory, pp. 38-53. Springer, 2008
Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In Advances neural information processing systems, pp. 442-450, 2010.
Kai Fan, Chunyuan Li, and Katherine Heller. A unifying variational inference framework for hi- erarchical graph-coupled hmm with an application to influenza infection. In Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, pp. 3828-3883333 2016.

---

Under review as a conference paper at ICLR 2021
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. In Advances neural information processing systems, pp. 1024-1034, 2017.
William Ogilvy Kermack and Anderson G McKendrick. A contribution to the mathematical the- ory of epidemics. Proceedings of the royal sociery of london. Series Containing papers of mathematical and physical character, 115(772):700-72) 1927.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. https: //openzeview.o www.:eiwwwwwneeoouaaaaa..aa -
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. 2003.
Shelley S Magill, Jonathan R Edwards, Wendy Bamberg, Zintars G Beldavs, Ghinwa Dumyati, Marion A Kainer, Ruth Lynfield, Meghan Maloney, Laura McAllister-Hollod, Joelle Nadle, al. Multistate point-prevalence survey of health associasee infections. New England Journal of Medicine, 370(13): 191-1198--22, 2014.
Maggie Makar, John Guttag, and Jenna Wiens. Learning the probability of activation the presence of latent spreaders. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jeeheh Oh, Maggie Makar, Christopher Fusco, Robert McCaffrey, Krishna Rao, Erin Ryan, Laraine Washer, Lauren R West, Vincent B Young, John Guttag, et al. A generalizable, data- driven approach to predict daily risk of clostridium difficile infection at two large academic health centers. infection control & hospital epidemiology, 39(4))425-433) 2018.
J Origüen, L Corbella, MA Orellana, M Fernandez-Ruiz F Lopez- soain, T Ruiz-Merlo, A Morales-Cartagent G Maestro, et al. Comparison of the clinical course of clostridium difficile infection in glutamate dehydrogenase-posie posittive toxin-negative patients di- agnosed by per to those with positive toxin test. Clinical Microbiology and Infection, 24(4): 41 414-421, 2018
San Juan, M Liza-
Christopher R Polage, Clare E Gyorke, Michael A Kennedy, Jhansi L Leslie, David L Chin, Susan Wang, Hien H Nguyen, Bin Huang, Yi-Wei Tang, Lenora W Lee, et al. Overdiagnosis of clostrid- ium difficile infection in the molecular test era. JAMA internal medicine, 175(11):1792-180) 2015.
Michelle Riggs, Ajay K Sethi, Trina F Zabarsky. Elizabeth Eckstein, Robin LP Jump, and Curtis J Donskey. Asymptomatic carriers potential source for transmission of epidemic and nonepidemic clostridium difficile strains among long-term care facility residents. Clinical infectious diseases, 45(8):992-99) 998, 2007.
Philippe Rigollet. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research, 80Jul: 2007.
James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference in epidemiology, 2000.
JM Robins. 1997 proceedings of the section on bayesian statistical science. 1998.
Joshua Robinson, Stefanie Jegelka, and Suvrit Sra. Strength from weakness: Fast learning using weak supervision. arXiv preprint arXiv:2002.08844 08483, 2020.
Matthias Seeger. Learning with labeled and unlabeled data. Technical report, 2000.
Dustin Tran, Rajesh Ranganath, and David Blei. Hicrarchical implicit models and likelihood free variational inference. In Advances in Neural Information Processing Systems, pp. 5523-5533, 2017.
Alexander B Tsybakov et al. Optimal aggregation of classifiers statistical learning. The Annals of Statistics, 135-):33 2004.
10

---

Under review as a conference paper at ICLR 2021
Jenna Wiens, Eric Horvitz, and John V Guttag. Patient risk stratification for cospital-associaan c. diff as time-series classification task. In Advances in Neural Information Processing Systems, pp. 467-475, 2012.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint 1rXiv:1812.08434. 2018.


Reviews:
[{"review_id": "v6P3aCQe0Iz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates the contagious disease into a missing label problem with dependence between each data point. The paper targets an important problem, especially in this pandemic, and the effort is greatly appreciated. However, the writing of this paper is confusing and it makes it hard to catch the main contribution of this paper. There are some concerns: \n\n1. The formulated problem sounds like a node label missing problem in a graph, where the node is patient (x, y) and the edge is whether they are contacted (e). In so, the paper is actually predicting the label of each node. If my understanding is correct, I am not sure why the authors choose the current formulation rather than graph one. \n\n2. The notation and wording are sometimes confusing, especially when I only have limited knowledge in healthcare. For example, even after reading section 4.1, it is still confusing what kind of special data structure the author is indicating. I recommend the author give a more intuitive or even graphic explanation in the paper. \n\n3. All the experiments are self-compared and make the result less convincing. For example, it is not clear whether different NN network would result in different performance because only NN is fixed here. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HzhpVnjqoQT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Interesting Paper; Some Confusions\n\nThis paper considers the problem of predicting the infection status of untested individuals. The authors propose a recurrent neural network-based model to impute the infection statuses over time, incorporating information on both the individuals (features of the individual) as well as a contact network that defines which individuals are related to each other.\n\nThe problem the authors describe is interesting and important. It has seen substantial attention in the ML-Healthcare community in recent years. Nevertheless there are some confusing aspects to the paper that make it hard to evaluate the efficacy of the proposed method.\n\nComments as they appear in order in the paper (comments with higher importance denoted with **):\n\nThe authors' use of the variable e for exposure state is a bit confusing and inconsistent. In S3P1 they define e as a sum of contacts' exposures and so it should be an integer. Later in S4P1, they describe a failure mode in which e_a = .9 and e_b = .1, which would suggest the authors are viewing e as on a spectrum from 0 (little exposure) to 1 (a lot of exposure). It's also not clear, throughout, how exposure (and the features x) relates to infection; are there example joint distributions or conditional distributions of y given e that could describe that relationship? Is it being treated merely as another feature (normalized/standardized for the purposes of the NN model) or does it have some special status in how it relates to x^bar and y?\n\n**In S3P2, the authors define biased testing as entailing p(Xt | Ot = 1) \\neq P(Xt | Ot = 0) \\neq p(Xt). This is rather non-intuitive and it would make more sense for the variables to be flipped: there is bias in testing if the probability of being observed (i.e. tested) is different depending on the patient features. Can the authors clarify why they've chosen to define bias like this?\n\n**In S3P2 the authors state that they assume i's outcome is independent of their contacts (entirely? just the contacts' outcomes?) given xi. Can the authors clarify why this might be a reasonable assumption? The authors, for instance, don't specify any notion of time in making this assumption and so really it seems to be saying y_i^t \\indep y_j^t | x_i^1, x_i^2, ..., x_i^t. This is potentially problematic though since it would make sense for y_i^t to depend on y_j^{t-1} (my contact's true infection status yesterday affects my infection status today) and it's obvious that y_j^t and y_j^{t-1} should be dependent. The authors should consider providing a figure (e.g. non-causal DAG) that illustrates these assumptions. As is, I'm very concerned that either the assumption is not sufficient to eliminate network dependency biases, or that the assumption isn't valid in the data the authors are envisioning their method being applied to, or both.\n\nS3P2: J_i^t \\in D_0 \\cup D_1 should be \\subset (or \\subseteq) rather than \\in.\n\nS3P3: The authors define the inverse probability of being tested as p(Ot = o)/p(Ot = o | Xt). This is also a bit weird of a way to define it: the probability of being tested is the authors' numerator P(Ot = o) and the conditional probability of being tested (or propensity for being tested) is the authors' denominator p(Ot = o | Xt) so it doesn't make sense that the inverse should be this fraction rather than just 1/(the numerator).\n\n**S3P3: Using the authors' definition of the inverse probability of being tested, it's not clear how they obtain the risk in Eq. 1. Starting with the risk in the paragraph above, Rf = E[l(f(Xt), Y^{t+1})], conditioning on O = 1 corresponds to dividing by p(O = 1). To balance this, since the LHS of Eq. 1 is the same risk Rf, it makes sense that the authors multiply the loss by p(Ot = o), the numerator of w. It is not clear how dividing by the denominator of w (thus just multiplying the whole loss by w) results in the same risk. It's unclear to me whether this is a confusion on probability algebra or these some other assumption being made here (overlap should not be sufficient).\n\nS3P4: \"However, the following reweighted empirical loss is an unbiased estimator of (...)\" -- how do we know it's unbiased? Is there a proof?\n\nS4P1: What is the reason for adding 'hats' (^) to the y's in the sentence \"This becomes obvious if we breakdown \\hat{e}_i^t draws from Q as follows (...)\". Is the idea that we're iteratively plugging in the predictions for the contacts' infection status? This seems to be the first time in the paper this sort of idea is hinted at and so it's not clear what the estimate is meant to be.\n\n**S4.1P1 \"We stress that we do not require \\hat{y} to be an accuracte estimate of the true labels, but only require that there is significant separation between the imputed values (...)\" Is this formalized anywhere? Can the authors please provide equations and math to provide intuition for why this might be the case if not a formal proof? Empirical evidence isn't exactly enough since it's feasible for there to be other explanations for \"good\" performance.\n\n**S4.1P2 The same comment as above applies to the hand wavy description of the cluster assumption at the bottom of the next paragraph. \"The clusers assumtion states that (..) exposure states tend to form near discrete clusters\" -- what's meant by 'tend to'? More intution would be nice here.\n\nS5P1 \"Let A_i^t be the set of ancestors\" -- in what sense? Is there a (potentially causal) graph underlying your model? If not, what is an ancestor?\n\nS5P1 What is \\mathcal{U}^t? This doesn't appear to be defined\n\n**General: What assumptions are being made about the missingness mechanism for the true infection states. The authors should be very clear by stating something along the lines of \"we're assuming missingness at random\". That seems to be what the authors are getting at with their conditional independence assumption but i) it's not obvious that assumption is sufficient (see above) and ii) the authors haven't done a great job of linking that assumption to their argument for the efficacy (identification) of their model.\n\nS5P2/3: The authors describe the method by which they determine convergence of model fit -- I'm curious, given that they cite Chernozukov 2017, what properties their model has wrt sample efficiency and convergence to the \"truth\" (i.e. the parameters of their model converge to the true parameters of the function that maps X to Y)? In general, I'm not aware of theory that gives convergence rates for neural networks in the way Chernozukov discusses and so I'd be concerned the authors might be picking an arbitrary convergence cutoff rather than something principled.\n\nS6.1P1: \"Our goal here is to highlight how MIINT can be used to inform testing and isolation policies that lead to reduction in infection rates\" -- the authors should be _very_ careful with using this sort of language. It implies a sort of ground truth causal efficacy to the work when this is not causal work. I agree that semi-supervised approaches are very similar to causal methods but the authors do not appear to spend sufficient time considering the assumptions necessary to ensure valid causal inferences, rather than simply fitting a black-box prediction model.\n\nSimulation study: what is the mechanism for deciding whether somewhat gets tested? It isn't obvious in the description how this selection works. Additionally, from the description, it's not exactly clear what the probability of actually being infected is, given the features (from MNIST) and the contacts (indirectly from MNIST?)\n\nSensitivity to the potency property: \"This confirms out conjecture (...)\" The authors should be careful with such strong language based on empirical, simulated evidence.\n\nMinor:\nThroughout the paper there are several instances of incorrect grammar or other writing issues that make it hard to determine the meaning of the sentence in question. For instance in the first paragraph of S2, the authors say \"In the machine learning literature, previous work has relied on proxies for exposure, e.g., the prevalence of a disease in a community (citation), and implicitly assume that conditioning on individual characteristics.\" The authors should carefully proofread to remove these artifacts as there are some instances where they pose a more substantive impact on the readability of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HuUyuvFG5y-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on contagious disease prediction with the consideration of observed data bias and patient exposure. The authors present a Model for Infections under Incomplete Testing (MIINT) and the experimental results show the proposed model outperforms baselines on *some* metrics.\n\nProblems:\n1. The paper is not well-written and pretty hard to follow.  \n- It is unclear why the authors use w^1(X) in eq (1). Why should t be set as 1?\n- It would be better if the authors gave the definition of \\hat{w}^t_i in eq (2).\n- Q(D^t_1) is a set of imputed distribution. It is unclear how to generate Q.\n- It would be better if the authors use consistent notations in different places. such as f(X,Y) and f(X^t, Y^{t+1})\n\n2. The authors want to consider the exposure relationships between patients. An intuitive solution is graph-based models. It would be better if the authors compared the proposed model with graph-based models, e.g. [1]. Moreover, exposure states are not fully considered. The authors just simply use the count of exposure of observed true infection patients at the last time point. The exposure before two time-points is ignored.\n\n[1] Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. \"Semi-supervised learning using gaussian fields and harmonic functions.\" Proceedings of the 20th International Conference on Machine learning (ICML-03). 2003.\n\n3. In Table 1, the two simple baselines (OM, NEM) and POM outperform the proposed model a lot on AUPRC. It is worth explaining why. \n\n4. Due to missing technical details (especially how to generate Q), it is hard to re-implement the proposed models. It is necessary to provide code and data as supplementary materials.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ek5fEEhz1qo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose an approach, MIINT, for identifying infected individuals using a network-based approach. They also suggest two key properties, potency and similarity among groups, which impact the efficacy of MIINT and similar approaches. A detailed simulation framework is used to compare MIINT to relatively weak baselines. The simulation results show that the MIINT modestly outperforms the baselines; the results also confirm that all approaches degrade as expected as potency decreases and the similarity among groups increases. Experimental results on a (private) real-world dataset are somewhat mixed, but show that MIINT achieves a better true positive rate at an acceptable false positive rate.\n\nThe proposed approach formalizes an intuitive understanding of the spread of an infectious disease in (as far as I am aware) a novel way. The authors also identify and empirically evaluate the conditions under which the model breaks down. I also appreciate the effort of creating a simulation model to reflect how the spread may develop over time; I believe the simulation framework could also be useful for other researchers in this area. The model may also be relevant for other network spreader domains, such as “influencers in social media” studies.\n\nMy main concerns about the work are about the experiments. First, some sort of graph-based model should be included, such as label propagation or locally-linear embedding (followed by some classifier); of course, standard graph neural network approaches could also be included in a different, more empirical evaluation-oriented study. With the current setup, it is not clear if MIINT in particular, or any approach which accounts for the graph, would outperform the baselines.\n\nSecond, it is not clear to me of the practical significance of the “isolation policy” results. Even using an oracle, Figure 1 seems to suggest that between 30-40% of the population would need to be tested to see a reduction in the infection rate. Is that actually reasonable, even in the “closed” clinical environment recommended by the authors in the conclusion?\n\nMinor comments\n----------------\n\nThe references are not consistently formatted.\n\nNeither the paper nor the supplement describe the features available in the real data.\n\nIt seems a bit strange that the AUROC and AUPRC in Table 1 are almost perfectly negatively correlated. Is there any explanation for that?\n\nWhile I found the text descriptions and intuitions very helpful, all of the in-line equations made parsing many of the paragraphs in Sections 3, 4, and 5 difficult.\n\n“outcomes, we refer” -> “outcomes; we refer”\n“If exposed individuals” -> “If exposed, individuals”", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies a timely problem and consider an interesting approach, but overall there were many concerns about technical details and the validity of the framework. The positive reviewer also mentioned concerns about the experiments, which others also found to be an insular comparison with weak baselines. Following the response period, in discussion there are additional concerns arising related to the lack of details, for instance related to possible unidentifiability of the model. As one reviewer discusses,  the authors are attempting to use RNNs to impute missing infection status labels when the missingness mechanism is assumed to be (i) not at random, (ii) playing out over time (as it is unclear whether Y^t is assumed (conditionally) independent of Y^t' with t' << t), and (iii) subject to interference (whether someone is tested is the 'treatment' here since it's a missingess problem and one person's propensity to be tested could causally affect another person's downstream infection status since apparently no Markov independence is assumed. There is also consensus that the writing quality can be greatly improved. Overall this work contains some ideas with potential in a thorough revision 
Decision: Reject