Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper addresses the problem of predicting long-term Alzheimer’s disease (AD) progression. It proposes a model that combines domain knowledge on the evolving relationships between factors affecting the brain together with a data-driven approach to learning the missing relationships. The learning objective used to learn the data-driven components satisfy a general criteria about the working of the brain, which maximizing cognition while minimizing the cost of supporting cognition. The authors propose using differential equations (DEs) to capture domain knowledge and reinforcement learning (RL) to learn the data-driven component. The model was validated on the ADNI data set and was found to outperform SOTA baselines.
main_review: 
This paper addresses a very important disease that presents one of the biggest emotional and financial burdens to patient care takers and healthcare systems, and presents an interesting approach for combining domain knowledge and data-driven model components. However, the model was only evaluated on a very small sample size (160 patients from the ADNI data set) and did not consider a lot of previous work that developed disease progression models for the same disease and the same data set. Because of these reasons, I think the paper belongs in the **borderline acceptance** category.

Few comments and questions that I'd like the author to reply to:

- It is unclear to me what is the reasoning behind using RL to solve the optimization problem for $I(t)$. Since we collect all the data retrospectively and run the model on an offline, doesn't it make more sense to solve the optimization problem for all $t$ using an offline approach?

- What is the clinical significance of the results in Table 1? The MSE and MAE by themselves do not reflect accuracy of predicting clinically-useful phenotypes of dementia. Did you consider predicting discrete 10-year outcomes (e.g. conversion of mild cognitive impairment to AD) and evaluate the model accuracy instead? 

limitations_and_societal_impact: Limitations were addressed. No negative social impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors proposed a method that combines differential equation modeling with reinforcement learning to predict the progression of Alzheimer’s disease. 

In terms of contributions, the authors claimed that this study is the first to use reinforcement learning for disease progression. During my non-exhaustive literature review there does not seem to be clear objections against this statement. The authors also claimed that their model showed a 11% lower prediction error compared to a state-of-the-art. Moreover, the authors mentioned the so-called “recovery/compensatory processes” that is present in Alzheimer’s disease could be observed in their proposed model, showcasing the interpretability of the model.
main_review: Strengths:
1.	Personally, I am in favor of the approach the authors take, where they use explicit equations instead of an arbitrary machine learning model to represent processes and behaviors that are relatively well-studied. In this way they utilized domain knowledge in an intuitive and explainable manner.
2.	The authors tend to think proactively about interpretability of their methods, and they related their observations with the “recovery/compensatory” process known in neurobiology.

Weaknesses:
1.	In introduction, the authors mentioned that existing mathematical models cannot faithfully represent all factors relevant to Alzheimer’s disease. Based on that, it would be reasonable to either refine the equations or use a different model to take care of the gap. However, it is not very intuitive why reinforcement learning is a natural choice. It would be better if some insights and rationale is given.

Additional feedback:
1.	In Figure 4B, it might be helpful to indicate the correlation coefficients, even though it is mentioned in the text. Also, it would make more sense if the predicted vs. ground truth size of PFC is included – it looks weird when both HC and PFC are described in the text but only HC is shown in the corresponding figure.
2.	Captions of Figure 5 needs a bit improvement. It is not clear which model variant Figure 5B and 5C correspond to.
It would be interesting to see whether the model can perform consistently when choosing different follow-up times (year-1, year-2, etc.) as the initial point. For example, the prediction of N years after year-0 shall be reasonably similar to that of N-1 years after year-1. At the same time, I am aware that some data might only be available at year-0 and this might not be an easy investigation.
limitations_and_societal_impact: Nothing seems to be concerning.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work presents a method for modeling dynamic systems that combines mechanistically motivated differential equations with data driven models learned by RL.  The method is applied to AD progression, measured by MMSE (a cognitive function test) scores.  The key idea is that a mechanistic understanding of the system under study may be incomplete -- e.g., in a system of coupled differential equations, a quantity and its time evolution may be unspecified -- but that it may be possible to fill this gap by learning a reasonable model for the missing function.  This method is applied to AD progression in a real world dataset with mixed results - although adding the learned dynamics model improves accuracy (measured by MSE) a great deal relative to the DE baseline, it does not do as well as a pure ML based baseline (SVR).  
main_review: This work presents a fascinating approach to modeling dynamical systems in which there is a partial mechanistic understanding of the dynamics, enabling the specification (and fitting from data) of a detailed but incomplete system of differential equations.  Specifically, let $v$ index brain regions.  They hypothesize a mechanistic model of AD progression involving brain region size $X_v$, regional activity $Y_v$ (information processing per unit size), and a $I_v$ representing total information processing in each region.  Note that $Y_v X_v = \gamma I_v$ - $X_v$ can be directly observed from brain scans, as can $Y_v$ (via functional MRI). The hypothesis further specifies - and this is the crux - the time evolution of $X_t$: it is thought that degeneration of a region $v$ (i.e., changes in size) is governed by amyloid deposition and activity $Y_v$, with patient specific modifiers.  This is formulated as  a set of differential equations.  But these differential equations are incomplete - they specify the time derivatives of $X_v(t)$, but not $Y_v(t)$, or equivalently, $I_v(t)$.  This is analogous to trying to solve a linear system in 3 unknowns from only 2 equations.  So if we want to, e.g., run the system forward from a set of initial conditions, we are stuck.  

How can we fill in the missing piece, I(t)? Rather than hypothesize a differential equation for $I_v(t)$, this work proposes using RL to learn a policy that optimizes an intuitively reasonable objective - you want to increase $\sum_v I_v(t)$ to meet cognitive demands but also want to decrease metabolic demands. The action space consists of setting of I_v(t) to meet cognitive demand (minimized $C - \sum_v I_v(t)$) while decreasing metabolic demand $\sum_v Y_v(t) = \sum \frac{I_v(t)}{X_v(t)}$.  The dynamics of the RL system are determined by the system of equations, with parameters fit to training data (with some caveats due to the fact some patients do not have complete measurements for $Y_v(t)$) - the key link is that increased activity can lead to degeneration (negative changes in $X_v$).  Now we can start from initial conditions, say year 0 of AD, and then run the system forward in time using the learned policy in place of a discretized differential equation for $I$ to forecast progression, and compare to the observed trajectories and other forecasting methods.  In this paper, they do this for a 10 year window, with one year time steps.  There is an evaluation on synthetic data, but I will focus on the evaluation on the real data (ADNI).  I am not familiar with the dataset used. 

How did they do?  There are two comparisons. First, is the usual bake off comparison of their method vs some solid baselines.The new method seems to do very well wrt MSE compared to one baseline, and about the same as another baseline, SVR, in terms of agreement with the observed progression of test patients.  Plotting the data reveals a nuance, however - the SVR baseline fits the observed data shape, which shows a weird increase in mean cognitive function in year 9 followed by a precipitous decline in year 10 - better than the new method, which forecasts (on average) a smoother, monotonic decrease in cognitive function.  The authors explain this away, asserting that the smoother forecast is more realistic.  I agree, and am willing to view this as a case where noisy data is mitigated by a (semi-)mechanistic formulation of the dynamic model. In another context, the authors also raise the issue of missing measurements in the (real) dataset - since it is so sparse (31% observed) it may be that this is a result of an unfortunate pattern of missingness towards the end of the observation period.  This could perhaps be checked (are the patients who are contributing to the population mean at 9 years the same patients contributing to the population mean at 8 years, etc).  

The second comparison is a bit more puzzling - it shows that the forecasts with and without RL are remarkably similar in shape, with a scaling or constant offset from each other over time.  This suggests that the RL component is mostly setting an offset/constant - this is perhaps not surprising given the form of the differential equations, etc.  But it does beg the question of whether there might have been a simpler way to arrive at a dynamical system that fits the data very well (e.g., fitting a constant relating the initial year 0 prediction to observed on the training data, i.o.w., fitting an intercept), and whether the RL component is learning a trivial policy.  

There is an auxiliary result - their learned dynamical system also demonstrates compensation, a well known phenomenon in which cognitive demands can be shunted around to different regions to compensate for degradation of some areas.  This seems to me to be an unsurprising result of the RL objective, so is not that noteworthy.  

Overall I liked this work very much as a demonstration of a cool idea I haven't seen elsewhere.  There are elements of the presentation that should be clarified, along with some additional questions that it would be nice to see addressed.  

First, it is not clear to me how the baseline without RL is accomplished.  The whole point was, I thought, that we couldn't run the dynamical system forward from initial conditions because we are missing a key piece we will fill in with a learned model.  So how was the baseline run? 

Second, how did the authors handle the amyloid deposition as a function of time, $D_v(t)$, introduced in equation 6, line 142?  It also seems unspecified in the dynamical system - don't you have to model that somehow too? 

Third, in figure 5A, it would good to know why, at t = 0 years, most of the estimates are offset from the ground truth mean.  I would have thought t = 0 corresponds to the initial conditions, so what does a forecast from a dynamical system mean at that time? 

Fourth, I am at a bit of a loss as to how this could be done, but I do wonder whether the RL policy governing $I_v(t)$ is accurate and "non-trivial".  This is not possible with the real dataset, and I doubt there are any datasets of fMRI over 10 years of AD progression, but this does seem to me to be an important point about whether this approach is really adding a lot.  Perhaps in the synthetic data the authors could directly evaluate the RL policy governing the time evolution of $I_v(t)$, both when the data generating process is a good fit for the hypothesis, and when it is not?  

Finally, if possible, it would be interesting to see a sensitivity analysis to see how robust results are to misspecification of both the differential equations and the learned model for $I_v(t)$.  In general, I suspect that this could be a big limiter to this general approach - we may have only a very vague sense of the mechanisms (leading to misspecification of the dynamical system) and how to formulate a good RL objective.   

Overall, I really enjoyed this work - it was quite different from the usual ML conference submission and I think it could spark a lot of cool work and discussion even if it doesn't convincingly, in my opinion, beat one of the baselines or work super well.  It's hard to say how generally applicable this approach could be since it does require a lot of (reliable) domain knowledge, and it's not clear to me how well it worked in this case, but I found the approach intriguing and worth introducing to the NeuRIPS community.  I am curious whether this sort of approach has been taken in, e.g., applications of ML to physics.  


limitations_and_societal_impact: The authors point out one limitation - only 2 brain regions were modeled.  I don't think this is a very big deal.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a disease progression model for Alzheimer's disease (AD) that combines data-driven approaches capture trends in data over the course of the disease, which have proved successful over recent times, with more mechanistic biophysics-based models of how pathology spreads over the brain.  The authors introduce the use of reinforcement learning to learn update rules for regional brain quantities as a function of time thereby informing interactions between processes and measurements of neurodegeneration. Experiments with simulations confirm basic efficacy of the estimation procedure; experiments with ADNI data show some promising predictive power and demonstrate potential insights into disease biology.
main_review: This is a timely topic and extends the state of the art by bringing together data-driven models and mechanistic models.  The use of reinforcement learning is a nice idea, highly novel, and experiments show it makes a genuine impact on predictive ability.  The model overall ends up, as any model of such a complex process, making a lot of simplifying assumptions and so arguments in the paper about "explainability" in comparison to black-box fully data-driven techniques should be tempered by the fact that explainable observations are only meaningful if/when those assumptions hold. Nevertheless, the essence of the idea here extends as our understanding of AD improves and biophysical models become more accurate.  Overall this is a nice idea, although should be considered as a demonstrator of what's possible by fusing data-driven and mechanistic modelling approaches and avoid strong claims on what the model outputs actually say about AD; some of the results are anomalous to say the least, but the basic principles of the learning algorithms that enable such approaches are sound in my opinion.

Specific comments for the authors to consider:

1. Line 184 - I don't understand how the model can be estimated without functional MRI data informing on brain activation. It is unclear from the section that describes the data set, but this line suggests the paper uses no functional MRI at all; is that true and if so perhaps the authors could add some intuition to the formulation in Eq 9 explaining the consequences of not having this data available and how the I parameters, which depend on activation, can be defined without it.

2. Figure 4 desperately needs units to make any sense.  Particularly in panel A where the errors are all of order 10^-30 - I assume they are basically zero - but it depends on the scale of the measurements.  Does this suggest that the simulations use no noise perturbations of any kind so really that "validation" is simply a sanity check that the code works in estimating true parameters from data directly sampled from the model?

3. Section 4.2. can the authors confirm that the brain region sizes used to evaluate the model are not part of the training set to which the model is fit?  What is the ratio of hold-out data and how was the full data set divided into training/validation/test sets?

4. Figure 4C. The result that normal subjects appear to decline at the same rate as MCIs and much more quickly than SMCs is indeed highly anomalous - likely wrong.  This highlights some limitations in the model as it stands and I think it's important to get to the bottom of this kind of anomaly, although it doesn't detract from the potential of the learning algorithms that are the centrepiece of the paper.

5. The Introduction states  'we observed "recovery/compensatory" processes that mitigate the effect of the AD on cognition [11], which could not be observed with state-of-the-art models', which the qualitative discussion in section 4.4 reiterates.  It's unclear to me exactly what the authors mean by this - is it the opposing trends in hippocampal and frontal activity/processing-load that Figure 4B, 4C show?  The arguments that this genuinely reflects true trends in patients brains are very hand wavy and over-played in my opinion.  This either needs clarification or toning down substantially.

6. Line 295 - ref [28] doesn't appear to relate to event-based models - do you mean Fonteijn Neuroimage 2021?


limitations_and_societal_impact: Not really discussed, although I don't see this as a concern for this work - it is well motivated.
review_title: null

Meta-review: In the discussion with reviewers, the reviewers have highlighted the merits of this paper. One particular reviewer was unable to update their score of 5 to an accept, however, in private conversation with the reviewers, they all agreed that this paper warrants acceptance. This paper addresses a very important clinical problem, even though the technical contribution and the size of the validation data are limited. The proposed approach for combining mechanistic and RL-based data driven modelling approaches is interesting and worthy of publication.
Decision: Accept (Poster)