Reinforcement Learning based Disease Progression Model for Alzheimer’s Disease
Authors: Krishnakant V. Saboo
Anirudh Choudhary
Yurui Cao
Gregory Worrell
David T Jones
Ravi Iyer
Venue: NeurIPS 2021 Poster (2021)
Keywords: disease progression model
Alzheimer's disease
cognition trajectory prediction
reinforcement learning
differential equations

Abstract: We model Alzheimer’s disease (AD) progression by combining differential equations (DEs) and reinforcement learning (RL) with domain knowledge. DEs  provide relationships between some, but not all, factors relevant to AD. We assume that the missing relationships must satisfy general criteria about the working of the brain, for e.g., maximizing cognition while minimizing the cost of supporting cognition. This allows us to extract the missing relationships by using RL to optimize an objective (reward) function that captures the above criteria. We use our model consisting of DEs (as a simulator) and the trained RL agent to predict individualized 10-year AD progression using baseline (year 0) features on synthetic and real data. The model was comparable or better at predicting 10-year cognition trajectories than state-of-the-art learning-based models. Our interpretable model demonstrated, and provided insights into, "recovery/compensatory" processes that mitigate the effect of AD, even though those processes were not explicitly encoded in the model. Our framework combines DEs with RL for modelling AD progression and has broad applicability for understanding other neurological disorders.

Reinforcement Learning based Disease Progression Model for Alzheimer's Disease
Krishnakant V. Saboo
Anirudh Choudhary UIUC ac670illinois.co edu
UICC ksaboo2@illinois.. edu
Yurui Cao UIUC yuruic2@ilinois.eede
Gregory A. Worrell
David T. Jones
Mayo Clinic
Mayo Clinic
Worrell. Worrell.Gregoryyeeeeeelel......e www.eerrrrrooooeeeeoooooo
Jones Jones.Davidoee David@mayo.ed edu
Ravishankar K. Iyer
UICC rkiyer@illinois.cm edu
Abstract
We model Alzheimer's disease (AD) progression by combining differential equa- tions (DEs) and reinforcement learning (RL) with domain knowledge. DEs provide relationships between some, but not all, factors relevant AD. We assume that the missing relationships must satisfy general criteria about the working of the brain, for e.g., maximizing cognition while minimizing the cost of supporting cognition. This allows us to extract the missing relationships by using RL to optimize an objective (reward) function that captures the above criteria. We use our model consisting of DEs (as simulator) and the trained RL agent predict individualized 10-year AD progression using baseline (year 0) features on synthetic and real data. The model comparable better at predicting 10-year cognition trajectories than state-of-the art dearning-based models. Our interpretable model demonstrated, and provided insights into, "recovery/compensatorr"" processes that mitigate the effect of AD, even though those processes were not explicitly encoded in the model. Our framework combines DEs with RI. for modelling AD progression and has broad applicability for understanding other neurological disorders.
Introduction
Models that describe Alzheimer's disease (AD) progression through time, i.e., the evolution of factors involved in the disease such as brain size, brain activity, pathology, and cognition (Fig. [T. are crucial for mitigating this highly prevalent discase [ 2] Such models can enhance our understanding the discase processes and enable crucial applications like prediction of long-term cognition trajectories for early detection. Our goal is to develop a model to predict an individual's future AD progression at 1 l-year intervals based their baseline (year 0) data. We address this goal by combining differential equations that capture the relationships between some factors, and leveraging reinforcement learning to extract the missing relationships by optimizing domain knowledge- based reward function.
Differential lqquttonn (DE) based models can describe disease progression by expressing domain knowledge as mathematical relationships between different factors 3 4 DE-based models have several advantages which make them attractive for disease progression modelling. The interpretability
For simplicity, we use the term "differential equation" to denote algebraic and differential equations.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of these models can enhance our understanding of disease processes 5 6 These models require limited data because the data only used for parameter estimation. DE- based models' mechanistic nature allows for intervention exploration [6]. However, these models provide an incomplete view of the disease because DEs describing relationships among some of the factors are unavailable [6].
We propose framework for modeling AD progression that combines differential equations with rein- forcement learning (RL) overcome the above limitation. We assume that the missing relationships are the solution of an optimization problem that can be formulated based on domain knowledge. In our model, this optimization problem's 's objective function serves as the reward function for reinforcement learning. Therefore, by optimizing the reward, RL extracts the relationships among factors for which explicit DEs are unavailable. Thus, RL combined with the DEs describes the evolution of factors pertinent for modeling AD progression.
In our model, the available DEs define the simulator and the RL agent optimizes the domain-based reward function the simulator. The parameters of the DEs are based on the available data. We overcome three main challenges for developing the proposed model. The first challenge identify the factors involved in each DE. We use domain knowledge to find existing causal relationships between different factors and differential equations relating them (when available).
Second, the DEs relating cognition, brain regions' sizes, and brain activity are unknown (Fig. [ Since proposing novel DEs relating these factors requires significant scientific knowledge that is still in development, we address this challenge as follows. Multiple brain regions work together produce cognition 7 8. We assume that the working of multiple brain regions is governed by an optimization problem that maximizes cognition while minimizing the cost of cognition [9]. We represent this optimization problem's 's objective function in terms of cognition, brain size, and brain activity.
Finally, the above optimization problem needs to be solved for multiple time points for modeling discase progression with the solution at time t affecting the solution at time t' t. Therefore, RL to optimize the above objective (reward) function.
Our contributions are:
We developed an Alzheimer's disease progression model based on domain knowledge that combines DEs and RL. To the best of our knowledge, this is the first attempt at using RL to develop discase progression model. Our framework generic and can used with other DEs, or to provide basis for modelling other neurological disorders.
(t),, (t)) Amyloid Pathology
Florbetapir-PET
Czask Cognitive demand
x Brain region size
MRI
2. We applied the model for predicting individualized long-term (10-year) future cognition trajectories using baseline (0th year) features on synthetic and real data. On real data, our model reduced the prediction error by ~ 10% than a state-of-the-art deep learning-based model 100 and produced more realistic trajectories than other benchmark models. Cognition trajectory prediction models are useful in clinical setting to identify individuals at future risk of cognitive decline.
Y(t) Brain activity
fMRI
M(C) Energetic cost
C(t) Cognition
MMSE
Figure 1: Causal relationships between variables based on domain knowledge.
Our interpretable model provided insight into how multiple brain regions together produce cogni- tion during AD. Specifically, we observed "recov- ery/compensatory" processes that mitigate the effect of AD on cognition [II), which could not be observed with state-of-the-an artt models. Recovery processes were not explicitly encoded into the model and were an outcome of the reward function's form. Further investigation into the basis of recovery processes could guide the development of interventions.

---

State (((t),(t-1)
(((
(1)
((((t)
(r)(t)
RL agent
(c)
(c)
Reward C(t) M(+)
Action
-12(Crask
Al(t)
Environment DEs for ( Y()) C(1).M(1
c(t) Cognition
Figure 2: Framework for modeling AD progression. (A) Relationship between brain size, brain activ- ity, information processing, and cognition (represented by solid and dashed edges). (B) Framework for AD progression that combines differential equations (DEs) with reinforcement learning (RL).
1.1 Background
Alzheimer's disease results from the interaction of pathology (such amyloid [12] and recovery (compensation [13 (II) processes. These processes jointly affect brain structure (the size of different
regions), brain function (activity in different regions), and cognition [14
[5]. Pathology
leads to neurodegeneration i.e., reduction in brain size 12, 16], and consequently leads to cognitive decline. Recovery processes compensate for the effect of neurodegeneration on cognition through modification of brain activity by employing other regions of the brain [33 for cognition. Ironically, compensatory processes can lead to neurodegeneration and accelerated cognitive decline long term [15 An individual's demographics such gender, education, genetic risk for dísease, etc., also play role in determining disease progression
2
Model
First, we use domain knowledge to find existing causal relationships between different factors in Sec 2.1 In Sec 2.2 we define the notation for the factors and substitute their causal relationships with appropriate DES. In 2.c we formulate an optimization problem to find missing relationships and solve it using RL. Thus, the model predicts disease progression via the interaction between the DEs (which constitute a simulator) and the action of the RL agent. Finally, we describe the training of the model Sec. 2.4
2.1 Identification of causal interactions
We simplify the interactions between different factors by using the causal relationships between them proposed in literature (Fig. [].. Amyloid beta (Ag) (measured using Morbetapir-PET is primary pathological factor in AD. propagates from one brain region to another through tracts leading to its deposition in neighboring brain regions 3. Accumulation of amyloid directly affects brain structure (measured using MRI), i.e., brain regions sizes, and leads to neurodegeneration [12]. Brain structure predicts brain activity (measured using fM 19], and brain activity results in cognition (measured using cognitive tests for e.g.. MMSE). Since brain activity also depends on cognitive task difficulty (20), we define hypothetical variable, Crask- which represents the cognitive demand on the brain and can thought of as maximum score on a cognitive test. Ctask directly influences brain activity. Brain activity has energetic cost, and itself can lead to neurodegeneration II Developing model for AD progression equivalent to describing the differential equation corresponding to each edge in Fig. I
2.2 Specification differential equations
Brain structure: We represent the brain as the graph Gs (V. E), where node E V represents a brain region, and an edge e E represents a tract. Let (1) denote the size of brain region E V at time f, and (t) [t),t,,
(t)].
3

---

Pathology propagation: To incorporate the propagation and accumulation of pathological A, in our framework, we adapt the network diffusion based model proposed by Raj et al. 3 because captures the propagation of A, through tracts. Let (t) be the instantaneous amyloid accumulation in region
V at time Then propagation of amyloid is given by:
dD(t) 3HH dt
where, (tt) D2(t), (yyyt)....,,,(((t).)) H is the Laplacian of the adjacency matrix of the graph Gs, and / is constant. The total amount of amyloid in a region, (t), is:
(t) S. (s(ss
Brain activity and cognition: Multiple brain regions work synchrony produce cognition Let (t) denote the activity in region E V in support of cognition Ctt) at time t. Although cogmition, brain size (X,,), and activity (Y,) are related, the exact relationship among them unknown and cannot be easily learned from limited data. On the other hand, we can intuitively relate a region's 's size and activity to "contribution" to cognition [21 Therefore, we introduce hypothetical variable termed as information processing, [t) E Rgo. to represent region v's contribution to cognition (Fig. 2h. Therefore, the cognition (tt) supported by the brain at time can be described as:
C(1)
Σ Tt) VEV
The activity in a given region depends on the amount of information is processing and its size (3]. For region with size (t), the activity Yt(t) increases as [t( increases [21] A healthier region (i.e.. greater region size) can be considered more efficient and will require lesser activation than an inefficient one for comparable levels of information processing (20). Therefore, propose the following relationship between (I), (4,,), and (1) with y as a constant:
(t)
(t) ? (t)
VVEV
T, (t)s are determined through reinforcement learning described in Sec. 2.3
Energetic cost: The brain consumes energy for supporting cognition. This energy consumption can be thought of as cost to the brain. The energy consumption in region is proportional to its activity [22]. Therefore, the energetic cost to the brain is the total brain activity across all the regions. We compute the energetic cost (t) as follows:
M(t) Σ Yt)
(5)
DEV
Degeneration of brain regions: Neurodegeneration influenced by two factors: amyloid deposition and brain activity. Previous studies that proposed linear relationship between the rate of brain region degeneration and A, deposition replicated the macroscopic neurodegeneration patterns seen in AD
4. Moreover, brain activity supporting cognition can further accelerate neurodegeneration [[ Therefore, we adopted the relationship proposed in [5] and modified it to incorporate the effect of brain activity follows:
(t. - dt
o, D.()) -oy,(0)
Vo EV
(6)
where a1, a2 are constants. The first term on the RHS in the Eq. 6 the degeneration due to A,8 pathology and the second term is the deterioration due to brain activity (see Appendix C.2 for an alternate formulation of this equation).
Parameter constants of equations: An individual's 's demographics, such as gender, genetic risk of
AD, education, etc., influence disease progression [7]. In our model, the influence demographics
is mediated through the parameter constants (n1, 022 3,y introduced in Eqs 4 and [5). For demographic factors Zo at baseline, let
(a1, 02 3,7) f(Zo).

---

2.3 Reinforcement learning to determine [(t)
Specifying the model requires determining the information processed by each region, i.e., com- puting [t) = (t)........ I (t)]- Multiple brain regions jointly produce cognition. Those joint relationships influence (t). We assume that those joint relationships satisfy general criteria about the working of the brain and obtain (t) by solving an optimization problem capturing those criteria. Specifically, we assume that (t) is chosen to distribute the workload related to cognition "optimally" across multiple brain regions. The objective function of that optimization problem is based on two competing criteria: (i) minimizing the deficit between the cognitive demand of a task and the cognition (t)) provided by the brain, and (ii) minimizing the energetic cost M (t) of supporting cognition (9) The optimization is performed at every time point t with the solution at t affecting the
optimization problem at 1' > t. Therefore, we use RL to determine the optimal T(t) by balancing the two competing criteria. RL a desirable method in our case since can train with limited data and also makes the framework generic and flexible changes/refinements in the reward function and the underlying DEs. Below we describe how RL is adopted in our model (Fig. 2). We implement the model for discrete time steps& & E 21, KJ.
State and action: information processed by each region at the previous time point [[t 1). The latter is provided in the state for ease of training the RL model. The action A(t) A specifies the change in information processed by each region from the previous time point, AT, (t) E R Vo V.
The state S(t) consists of the current size of the brain regions X(t) and the
S(t) (X(t), 1)).
A [ANO) |A
(000),...,,,A(000)) AM T.(1) = To -1-1-1 + (T)) ΣT( S Crask DEV
Environment: The environment is a simulator consisting of the equations relating D(t), X(t), (t), [(t,, C(t). and (t) (Eqs. 2 5 and 67. Based on the action A(t). the environment updates the state and provides reward to the RL agent.
The policy agent must balance the trade-off between the competing criteria of (i) reducing mismatch between Ctask and (tt) and (ii) reducing the cost M(t) of supporting cognition. Hence, we define the reward as follows:
Reward:
R(()- - [A(Grask C(L) + M(t)
(8)
where À is a parameter controlling the trade-off between the mismatch and the cost. The goal of the policy agent is to maximize the reward.
2.4 Training
Training the model consists of (i) estimating the parameters and finding f in Eq. 7 and (ii) training the RL model (Fig 3). The model is trained using multimodal longitudinal data from individuals in the training set. First, we estimate the parameters a1, 0:2, B, - for people in the training set and relate the estimates to baseline demographic features of the individuals Zo to find f. Second, baseline measurements of the X X0), (0)) are used with the estimated parameters to train the RL model.
Parameter estimation: We assume that for person i (1,... N). data at discrete time points / E (1...., K) is available for the variables (1),,, D' (1), Y*(),, (1). We derive parameter estimators by minimizing the L2 norm of the difference between the LHS and RHS of discretized versions of Eqs. for B, Eq. 6 for a1,, 022 and Eqs. 3 and 4 for y (see Appendix A for the derivations).
Due to unavailability of functional MRI data in our experiments, we also derived parameter estimators for the case in which (t) is unavailable for estimation. Minimizing the L2 norm obtained by results in the following estimators (see
eliminating (t( from Eqs. 6 and combining it with Eq. 3
Appendix A for the derivation).
Σ. E,0' ADiC
KiKs - KgK4,
aáT = K3 - K₁ K₂
E. E,E,("() AF
á
K2Kg, KyKs'
K₂K1 K3Kg
(9)
where, (1) (X(())
as(() ( = (X'(1) are both scalars and K1
Σ (a)(t))), = =2 E, K, = E, (t)a Kg = E. (tt)
(1)))
5

---

5-Fold Cross Validation
Training (longitudinal data for group)
Baseline year)
Braiin size, amylold pathology, cognition X, tb C
Train RL agent
Parameter estimation
optimize reward
Train (za)
Demographics
Z
Testing (baseline data of an individual)
Parameters
Ground truth
Zo
f(zn)
Trained RL agent
AI(t)
Σ
Cognition
MSE
MAE
Evaluate
( (00)
1(t - 1)
error
Figure 3: Workflow for training the proposed AD progression model and applying it for long-tern cognition trajectory prediction.
Note that in Eq. 9 the product ('22) canmot resolved further due to unavailability of Y Therefore, assumed that 7 is the same for all the individuals. Moreover, y affects the model either through o:2y in Eq. 6 or essentially through A/7 in the reward function Eq. S (see Appendix C.I) Therefore, set 7 for all our experiments.
Preliminary analysis showed that only discrete demographic features are associated with the parameter estimates (see Appendix (..)). Therefore, was the same a table lookup based on Zo. In general setting, can be approximated with a linear model or a neural network.
The agent is trained on the simulator with model-free on-policy learning using TRPO 23 (see Appendix (.4). All the agent models are trained for million episodes with batch size of 1000. We clip the reward within a range of [-2000, 2000] with KL--divegeecce of 0.01 and generalized advantage estimate discount factor of 0.97. We do not tweak any other hyperparameters TRPO. The policy network is parameterized by two-layer feedforward neural network with 32 hidden units each. We implement the simulator using OpenAl's Gym framework (24). performed a grid-search for \ and (0) and chose the value that minimized the validation set error (see Appendix C.5
RL agent:
Experimental Setup
3.1 Adopting AD progression model for 10-year cognition trajectory prediction
During testing, model can be used for predicting the long-term future cognition trajectory of an individual based on their baseline (0th year) data (Fig. 3). A new person's 's baseline demographics (Z)) used to find their parameters a1, 0:2y, using Eq. Those parameters, along with baseline value of brain regions sizes (X))) and amyloid (o(0),, are provided as initialization to the trained RL agent for predicting disease progression. We set cognitive demand Ctask = 10 for all the experiments. We predicted cognitive score at 1-year intervals for 10 years after baseline.
3.2 Dataset
We validated the model on synthetic data and real-world data derived from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni. www..onii.uscduu (25)
Synthetic data: We validated our model on synthetic data of 200 samples. Fach sample consisted of longitudinal trajectories of the involved variables (see Appendix B for details).

---

Pretici IP (Propored RRL
10
Squared Errer
Cognition Score
10-29
9.0
Diegnottic Group
EMCI
10"
8.5
CN
UMC1
SMC
01
Y Parameter
Truth
from
Figure 4: Validation of the proposed model. (A) Error in parameter estimation for synthetic data. Parameters were of the order of 10 2 to 10". (B) Predicted vs. ground truth size of hippocampus for ADNI. (C) Average predicted cognition trajectories over years for groups based on diagnosis at baseline in ADNI.
In this study, we included data from 160 individuals, used V 2 nodes representing the hippocampus (HC) and the prefrontal cortex (PFC), and measured cognition using Mini Mental State Examination (MMSE) (scaled by 3). We excluded IMRI scans from our analysis. See Appendix B for details of data preprocessing.
ADNI:
3.3 Model evaluation
Model evaluation consisted of (i) evaluating parameter estimation, (ii) validating group level AD progression trends, (iii) evaluating long-term cogmition trajectory prediction, and (iv) providing insights into AD progression. We used 5-fold cross-validation for evaluation with a 64:16:20 split into training, validation, and testing sets. In each fold, each individual's data was only part of one of the three sets. Parameter estimators were evaluated only on the synthetic data using mean squared error, since ground truth values of parameters are known. For evaluating the model's predicted cognition trajectories, we compared them with ground truth trajectories using mean squared error (MSE) and mean absolute error (MAE). Since there are a large number of missing values in ADNI data, we only considered time points for which cognitive scores were available for individual computing prediction error. We compared the prediction performance with other state-of the art methods (minimalRNN, SVR [10); see Appendix D for implementation details) for cognition trajectory prediction. To demonstrate the value of using RL in our framework, we compared the prediction with the case when I(t) is determined by optimizing the reward R(t) using grir-search separately at each time instead of RL (w/o RL) (see Appendix D.3).
Results
4.1 Parameter estimation
We estimated the parameters 027, 3 for the synthetic data using the estimators in Eq. Estimators achieved low errors (Fig. 4. Since real data has a large number of missing values, we also evaluated the effect of missing values parameter estimation. Although missing values increased the estimation error, the estimated parameters were still comparable to their ground truth values (see Appendix A.5) Zo consisted of two discrete variables for the synthetic data. Therefore, f(Zo) (from Eq. 7 was a table lookup for the synthetic data.
For ADNI data, we performed preliminary analysis with andvidual-specifie parameter estimates and chose gender and genetic risk (presence/abseece of eP genotype) as the demographic variables corresponding to Zu (see Appendix (A6)). We also observed those parameter estimates were highly variable across subjects, which could due the small number of data points per individual, missing data, or noise in the data. Therefore, we estimated parameters for groups of subjects based on their gender and genetic risk. The estimated parameters are shown the Appendix A.6 (Z)) was table lookup with Zp (gender, genetic risk).
7

---

Cognition Scote
10.0
9.5
Information. Processed
Frontal
Hippocampus
Brain Train&ctiion Activity
2,00
2:25
Frontal
Hoppocampus
3.75
Method
Prppseed
3.50
8.5
Papeood two
Ground Trute
1:00
10
10
2
10
Years from baseline
Years from baseline
Years from baseline
Figure 5: (A) Predicted 10-year future cognition trajectories on the ADNI averaged across individuals. Trajectories are shown for the ground truth, benchmark models, and proposed model. (B) Information processing averaged across individuals for HC (hippocampus) and PFC (frontal) during disease progression for the proposed model. (C) Brain activity averaged across individuals for HC and PFC during disease progression for the proposed model.
4.2 Model validation
We validated the proposed model by comparing group-level trends for individuals in ADNI data. First, we compared the size of the brain regions obtained from model with their size in the ground truth data. The values of regions size obtained from the model were highly correlated with those in the ground truth data (correlation of 0.8 for HC and 0.62 for PFC) (Fig. Fig. 9p. We also compared the average cognition trajectories for each diagnostic group based on a person's ' diagnosis at baseline. As expected, the cognition of early and late mild cognitive impairment (MCT) individuals declined faster than individuals with significant memory concern (SMC) for whom cognitive impairment is less severe in reality (Fig. 4 Counter-intuitively. cognitively normal (CN) individuals showed rate of decline higher than SMC group and similar to MCI groups. To understand this result, we compared the mean baseline region sizes and amyloid values in different groups. Mean baseline brain region sizes for the CN group were lower than SMC and more similar to MCI groups (mean for HC- CN: 3.81, EMCI: 3.77, LMCI: 3.60, SMC: 3.94; PFC- CN: 3.71, EMCI: 3.81, LMCI: 3.74, SMC: 3.75). Similarly, mean baseline amyloid for CN participants was higher than SMC for HC (CN: 1.32. EMCI: 1.29, LMCI: 1.29, SMC:1.28) although for the frontal areas was smaller than SMC (CN: 1.26.
EMCI: 1 1.30, LMCI: 1.32, SMC: 1.33). Since the disease progression prediction largely depends on baseline value the features, these observations taken together suggest that the similarity of baseline features of CN to MCI groups than SMC leads the predicted cognitive trajectories for CN being similar to MCI groups and worse than SMC.
4.3 Cognition trajectory prediction
We evaluated the ability of the model predict individualized 10-year future cognition trajectories based on baseline data (see Table In general, data-driven models tend to be more accurate than mechanistic models and thus provide an upper limit for prediction performance (see Sec. 5). The proposed framework achieved comparable or better performance than state-of sitee-of-theeeee ttthhe-rt art methods on synthetic data well as real data. The proposed model outperformed the DE- DE-based model without RL, which demonstrates the value of using RL for combining DEs. At an individual level, predicted cognition from the model without RL was either or whereas predictions from the model with RL better resembled ground truth cognition (see Appendix D.3). The improved performance of the proposed model than the sfate-of-the-ar deep learning-base to the small sample size. Although our model had higher error compared to SVR on ADNI, the average cognition trajectories from our model were more realistic 266 (Fig. 5) Average trajectories for SVR on ADNI showed an increase in cognition year 9 followed by steep decline year 10. This trend could be attributed to the disparity in the number of participants who had data at year (n=60), year 9 (n=17), and year 10 (n=4) with few individuals having cognitive data at year 9 or 10. These observations highlight the value of using DEs and domain knowledge in our model.
(minimalRNN) model could be due
We also evaluated the consistency of the trajectories predicted by the proposed model when data from different follow-up times was used as "baseline". 134 participants had all the relevant data available at year 2. For those participants, we predicted trajectories using year 2 as "baseline" and compared
8

---

them with the trajectories predicted using year O as baseline. The two trajectories for each participant
were highly correlated (average correlation across participants=0.93).
Method
Synthetic Data
ADNI
MAB
MSE
MAE
MSE
Proposed (with RL)
0.641 (0.090)
0.910 (0.229)
0.537 (0.127)
0.761 (0.370)
Proposed (w/o RI)
1.009 (1.670)
3.806 (8.073)
0.595 (0.137)
1.112 (0.406)
minimalRNN
1.395 (0. 149)
6.971 (0.753)
0.599 (0.137)
0,984 (0.659)
SVR
0.658 (0.050)
0.997 (0. (1.1
0.495 (0.067)
0.574 (0.230)
Table I: Long-term cognition trajectory prediction performance. Standard deviations are provided in
parentheses. Abbreviations: MAE, mean absolute error; MSE, mean squared error; and SVR, support vector regression.
4.4 Demonstrating recovery
We observed that our model exhibited recovery/compensatoryy processes during disease progression even though those were not explicitly encoded into the model. On averaging the trajectories of (t) for HC and PFC across individuals, we observed that as the contribution of HC to cogmition reduced, the contribution of the PFC increased, resulting maintaining the total cognition (Fig. [5). As disease progressed further, the information processing in PFC also decreased while the information processing in HC continued to decrease, resulting in cognitive decline. This observation is consistent with recovery/compensatoory behaviour that has been hypothesized to explain the response of the brain to injury [3] In control experiments, we observed that the proposed model trained with modified reward function that only included the cognition mismatch term or the energetic cost term (in Eq. () did not show recovery processes (see Appendix D.4). Our observation of recovery processes emerging from the model formulation, specifically the reward function, is crucial because could provide further understanding of the basis of those processes. Recovery- llike behavior has also been seen in other neurological disorders [II), and understanding the mechanisms driving them could be useful for exploring interventions for AD. Our model also showed an initial phase of increased brain activity (hypermetabolism) in the PFC which is consistent with previous observations in AD and aging studies [33 (27] The interpretable nature of our model enables these observations, which can help better understand AD. Note that the interpretation of these results meaningful within the context of the assumptions of the model and further research is needed to validate them.
Related Work
Studies on modelling Alzheimer's disease progression can be broadly categorized as follows:
models on biomarker data to relate disease pathology, region size, activity, cognition, and/or demographics. Examples of these models include event-based models (2]) mixed-effects models 29]. Bayesian models B0, and machine learning-bass based approaches [00]
Data-driven models fit aarametric/non-paramaa
32 33 These methods have low dependency on domain knowledge and work well for short-term prediction, but their long-term prediction performance is limited [34] due to lack of longitudinal multimodal data spanning the entire disease course. AD progresses over decades [35] (36,, so accurate modeling requires long- term longitudinal data from multiple modalities (MRI, PET, cognitive assessments) [35]. which is difficult to acquire. Further, interpretability of these models may be a challenge depending on their complexity which limits their applicability in understanding disease processes.
31
2. Mechanistic models use domain knowledge to represent the relationship among variables using algebraic and/or differential equations 4 B. Popular approaches include graph- based models [37] 38) and dynamic causal modeling [39]. These models are interpretable and
have low dependence on data. Although studies have modelled the relationships among a subset of the variables relevant for AD progression, to the best of our knowledge, a model that brings together pathology, brain structure, function, cognition, and demographics yet to be proposed.

---

Our model enjoys the advantages offered by mechanistic models through the use of several DEs in its formulation. It differs from the existing mechanistic models by incorporating the relationship between brain size, activity, and cognition. Moreover, the use of an RI. model trained in DE-based simulator using a domain-guidee reward function allows leveraging recent advances in machine learning without increasing the model's dependence on data.
Reinforcement learning has been used previously for developing treatment paradigms for neurological
disorders such as epilepsy [40] and Parkinson's 's disease A. Recently, Krylov et al. [ showcased the efficacy of policy-based basee RL framework in suppressing neuronal synchrony in Parkinson's
disease. They incorporated DE-based neuronal models into a simulator and trained multiple PPO
agents for oscillatory neuronal models. We are unaware of previous work that has used RL for AD disease progression modelling.
6 Conclusion and Future Work
We proposed a framework for modeling AD progression by combining differential equations and reinforcement learning and evaluated the validity of its predictions on real and synthetic data. Our model successfully predicted individualized 10-year-long future cogmition trajectories, which can be useful in clinical setting for identifying individuals at a future risk of decline. Our model demonstrated, and provided insights into the potential basis of, recovery processes during AD progression which emerged from the model formulation, specifically the reward function. Thus, the model can help in further understanding disease progression. The mechanistic nature of our model allows the exploration
interventions strategies through perturbation analysis of the variables [6]. We contend that our framework is generic and can be modified to support other DEs describing AD, or provide a basis for modelling other neurological disorders because the pervasiveness recovery processes [ and the prevalence of using differentiall equations for modelling discases 42 43
There are some limitations of the current work. We demonstrated the model with a 2-node graph, which is too coarse to capture the multimodal changes happening at finer spatial scales in the brain's structure and activity during AD progression. The model can scale to larger graphs with physiologically relevant nodes using data from the relevant brain regions. For relating brain activity to size, we proposed an inverse linear model (Eq. 4 but due lack of sufficient fMRI data in our experiments, we could not validate that equation. Further experiments with fMRI data may lead to better model for relating brain size (X (L)), activity (!),, and information processing (T()). AD also results in tract degradation which we have not modeled. Related to this is the sensitivity of the model to misspecification of the dynamical system. We observed that the form of the DEs affects discase progression prediction (see Appendix C.3) but the extent of the effect and its dependence on different factors needs further clarification. Finally, the model was only validated on ADNI data and, therefore, suffers from the same biases inherent in the data 225) Further experiments with larger and diverse data is needed to validate the results of this study. We plan to address these limitations in future work.
Acknowledgments and Disclosure of Funding
This work was supported by the Mayo Clinic and Illinois Alliance Fellowship for Technology-based Healthcare Research and in part by NSF grants CNS-1337732, CNS- 162476,,799, and CCF-2029049 and the Jump ARCHES endowment fund. We thank Saurabh Jha, Subho Banerjee, Frances Rigberg, and Prakruthi Burra for their valuable feedback.
References
[1] Nicole L. Batsch and Mary S Mittelman. World alzheimer report 2012. Overcoming the Stigma of Dementia. Alzheimer's ''s Disease International (ADI), London; 2012. Accessed May, 5, 2015.
[2] Zeinab Breijych and Rafik Karaman. Comprehensive review on alzheimer's discase: Causes and treatment. Molecules, 27(24):5789, 2020.
[3] Ashish Raj, Amy Kuceyeski, and Michael Weiner. network diffusion model of disease progression in dementia. Neuron, 73(6): 2012.
10

---

[4] Johannes Weickenmeier, Ellen Kuhl, and Alain Goriely. Multiphysics of prionlike diseases: Progression and atrophy. Physical review letters, 121(15)158101, 2018.
[5] Ashish Raj. Eve LoCastro, Amy Kuceyeski, Duygu Tosun, Norman Relkin, Michael Weiner, Alzheimer's Disease Neuroimaging Initiative (ADNI, et al. Network diffusion model of progression predicts longitudinal patterns of atrophy and metabolism in alzheimer's disease. Cell reports, 10(3):359-369, 2015.
[6] Daniela Conrado, Sridhar Duvvuri, Hugo Geerts, Jackson Burton, Carla Biesdorf, Malidi Ahamadi, Sreeraj Macha, Gregory Hather, Juan Francisco Morales, Jagdeep Podichetty, et al. Challenges alzheimer's disease drug discovery and development: The role of modeling, simulation, and open data. Clinical Pharmacology & Therapeutics, 107(4):796-80) 805, 2020.
[7] Danielle S Bassett, Edward T Bullmore, Andreas Meyyer-Lindenbrr., José A Apud, Daniel R Weinberger, and Richard Coppola. Cognitive fitness of cost-efficient brain functional networks. Proceedings of the National Academy of Sciences,
2009.
[8] Takuya Ito, Kaustubh R Kulkarni, Douglas H Schultz, Ravi D Mill, Richard H Chen, Levi I Solomyak, and Michael W Cole. Cognitive task information is transferred between brain regions via resting-state network topology. Nature communications, 8(1):1-14, 2017.
[9] Scott Thomas Christie and Paul Schrater. Cognitive cost as dynamic allocation of energetic resources. Frontiers in neuroscience, 9:289, 2015.
[10] Minh Nguyen, Tong He, Lijun An, Daniel C. Alexander, Jiashi Feng, and B.T. Thomas Yeo. Predicting alzheimer's s disease progression using deep recurrent neural networks. Neurolmage, 222: 17203, 2020.
[11] Frank G Hillary and Jordan H Grafman. Injured brains and adaptive networks: the benefits and costs of hyperconnectivity. Trends in cognitive sciences, 2 2165)338--401, 2017.
[12] Clifford R Jack Jr, David S Knopman, William Jagust, Leslie M Shaw, Paul S Aisen, Michael W Weiner, Ronald C Petersen, and John Q Trojanowski. Hypothetical model of dynamic biomarkers of the alzheimer's pathological cascade. The Lancet Neurology, 911::19-1228 2010.
[13] Simon W Davis, Nancy A Dennis, Sander M Daselaar, Mathias S Fleck, and Roberto Cabeza. Que pasa? the posterior-anterior shift in aging. Cerebral cortex, 18(5): 1201-1299,,, 2008.
[14] Emily L Dennis and Paul M Thompson. Functional brain connectivity using fini in aging and alzheimer's disease. Neuropsychology review, 24( 24(1):49-66 62, 2014.
[15] David T Jones, Jonathan Fraff-Radford, Val J Lowe, Heather J Wiste, Jeffrey Gunter, Matthew L Senjem, Hugo Botha, Kejal Kantarci, Bradley F Boeve, David S Knopman, et al. Tau, amyloid, and cascading network failure across the alzheimer's disease spectrum. Cortex, 97:143-159, 2017.
[16] Heiko Braak, Kelly del Tredici, Christian Schultz, and Eva Braak. Vulnerability of select neuronal types to alzheimer's 's disease. Annals of the New York Academy of Sciences, 924(1):53~ 61, 2000.
[17] Rebecca K MacAulay, Amy Halpin, Alex S Cohen, Matthew Calamia, Angelica Boeve, Le Zhang, Robert M Brouillette, Heather C Foil, Annadora Bruce-Kelledt and Jeffrey N Keller. Predictors of heterogeneity cognitive function: Apoe-e4, sex, education, depression, and vascular risk. Archives of Clinical Neuropsychology, 35(6):660-670) 2020.
[18] Christopher Honey, Rolf Kötter, Michael Breakspear, and Olaf Sporns Network structure of cerebral cortex shapes functional connectivity on multiple time scales. Proceedings of the National Academy of Sciences, 10240-11040 2007.
[19] Christopher Honey, Olaf Sporns, Leila Cammoun, Xavier Gigandet, Jean- Jean-Philippe Thiran, Reto Meuli, and Patric Hagmann. Predicting human resting-state functional connectivity from structural connectivity. Proceedings of the National Academy of Sciences, 10666)20055-2000, 2009.
11

---

[20] Yaakov Stern, Brian C Rakitin, Christian Habeck, Yunglin Gazes, Jason Steffener, Arjun Kumar, and Aaron Reuben. Task difficulty modulates young-old old differences network expression. Brain research, 1435:130 2012.
[21] Jeremy E Niven, John C Anderson, and Simon B Laughlin. Fly photoreceptors demonstrate energy-informatte trade-offs in neural coding. PLoS Biol, 5(4):e1 2007.
[22] David Attwell and Simon B Lanhhlin. An energy budget for signaling in the grey matter of the brain. Journal of Cerebral Blood Flow Metabolism, 2
2001.
[23] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pages 1889-1897, 2015.
[24] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: (okii:/6060 /606000500 2016.
[25] Michael W Weiner, Dallas P Veitch, Paul S Aisen, Laurel A Beckett, Nigel J Caimss, Jesse Cedarbaum, Michael C Donohue, Robert C Green, Danielle Harvey, Clifford R Jack Jr. et al. Impact of the alzheimer's disease neuroimaging initiative, 2004 to 2014. Alzheimer's 's & Dementia, 11(7):865-88) 884, 2015.
[26] Lars Lau Raket, Alzheimer's Disease Neuroimaging Initiative, et al. Statistical disease progres- sion modeling in alzheimer discase. Frontiers in Big Data, 3, 2020.
[27] A Ashraf, Z Fan, DJ Brooks, and P Edison. Cortical hypermetabolism in mci subjects: compensatory mechanism? European journal of nuclear medicine and molecular imaging. 42(3):447-458, 2015.
[28] Hubert M Fonteijn, Mare Modat, Matthew J Clarkson, Josephine Barnes, Manja Lehmann, Nicola Z Hobbs, Rachael Scahill, Sarah J Tabrizi, Sebastien Ourselin, Nick C Fox, et al. An event-based model for disease progression and its application in familial alzheimer's disease and huntington's 's discase. Neurolmage, 60(3): 1880-18888889 2012.
[29] Neil P Oxtoby, Alexandra L Young, David M Cash, Tammie LS Benzinger, Anne M Fagan, John C Morris, Randall J Bateman, Nick C Fox, Jonathan M Schott, and Daniel C Alexan- der. Data-driven models of dominantly-inherieed alzheimer's disease progression. Brain, 141(5):159--144, 2018.
[30] Wolfgang Fruehwirt, Adam D Cobb, Martin Mairhofer, Leonard Weydemann, Heinrich Garn, Reinhold Schmidt, Thomas Benke, Peter Dal-Bianco, Gerhard Ransmayr, Markus Waser, et al. Bayesian deep neural networks for low-cost neurophysiological markers of alzheimer's ' disease severity. arXiv preprint 1rXiv:1812.049941 2018.
[31] Weiming Lin, Tong Tong, Qinquan Gao, Di Guo, Xiaofeng Du Yonggui Yang, Gang Guo, Min Xiao, Min Du, Xiaobo Qu, et al. Convolutional neural networks-based mri image analysis for the alzheimer's s disease prediction from mild cognitive impairment. Frontiers in neuroscience, 12:777, 2018.
[32] Solale Tabarestani, Maryamossadat Aghili, Mehdi Shojaie, Christian Freytes, and Malek Ad- jouadi. Profile-specific regression model for progression prediction of alzheimer's disease using longitudinal data. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMIA), pages 1353-1357. IEFE, 2018.
[33] Krishnakant Saboo, Chang Hu, Yogatheesan Varatharajah, Prashanthi Vemuri, and Ravishankar Iyer. Predicting longitudinal cognitive scores using baseline imaging and clinical variables. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 1326-1330. IEEE, 2020.
[34] Räzvan V Marinescu, Neil P Oxtoby, Alexandra L Young, Esther E Bron, Arthur W Toga, Michael W Weiner, Frederik Barkhof, Nick Fox, Polína Golland, Stefan Klein, et al. Tadpole challenge: Accurate alzheimer's s disease prediction through crowdsourced forecasting of future data. In International Workshop on PRedictive Intelligence In MEdicine, pages 1-10. Springer, 2019.
12

---

[35] Clifford R Jack, Prashanthi Vemuri, Heather J Wiste, Stephen D Weigand, Paul S Aisen, John Q Trojanowski, Leslie M Shaw, Matthew A Bernstein, Ronald C Petersen, Michael W Weiner, et al. Evidence for ordering of alzheimer disease biomarkers. Archives of neurology, 15612):152-11535, 2011.
[36] Roy M Anderson, Christoforos Hadjichrysanthou, Stephanie Evans, and Mei Mei Wong. Why do many clinical trials of therapies for alzheimer's 's discase fail? The Lancet, 390(10100) 2329, 2017.
[37] Wei Li, Miao Wang, Wenzhen Zhu, Yuanyuan Qin, Yue Huang, and Xi Chen. Simulating the evolution of functional brain networks in alzheimer's disease: exploring disease dynamies from the perspective of global activity. Scientific reports, 6:34156, 2016.
[38] Petra E Vértes, Aaron F Alexander- Bexxnn,er-Bloch,, Nitin Gogtay, Jay N Giedd, Judith L Rapoport, and Edward T Bullmore. Simple models of human brain functional networks. Proceedings of the National Academy of Sciences, 109(15):5868-555 5873, 2012.
[39] Stefan Frässle, Ekaterina I Lomakina, Lars Kasper, Zina M Manjaly, Alex Leff, Klaas P Pruessmann, Joachim M Buhmann, and Klaas E Stephan. A generative model of whole-brain effective connectivity. Neuroimage, 179:505-529, 2018.
[40] Joelle Pineau, Arthur Guez, Robert Vincent, Gabriella Panuecio, and Massimo Avoli. Treating epilepsy via adaptive neurostimulation: reinforcement learning approach. International journal of neural systems, 19(04):227-26)) 2009.
[41] Dmitrii Krylov, Remi Tachet, Romain Laroche, Michael Rosenblum, and Dmitry V Dylov, Reinforcement learning framework for deep brain stimulation study. arXiv preprint arXiv:2002.1094 2020.
[42] Charles Venuto, Nicholas B Potter, Ray Dorsey, and Karl Kieburtz. review of disease pro- gression models of parkinson's 's disease and applications in clinical trials. Movement Disorders, 31(7):947-956, 2016.
[43] Piotr Orlowski, David O' O'Neill, Vicente Grau, Yiannis Ventikos, and Stephen Payne. Modelling of the physiological response of the brain to ischaemic stroke. Interface focus, 3(2):20120079, 2013.
[44] John R Cirrito, Kelvin A Yamada, Mary Beth Finn, Robert S Sloviter, Kelly R Bales, Patrick C May, Darryle D Schoepp. Steven M Paul, Steven Mennerick, and David M Holtzman. Synaptic activity regulates interstitial fluid amyloid-8 levels in vivo. Neuron, 48(6):913-99) 922292,22222-221 2005.
[45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,,000 2017.
[46] The garage contributors. Garage: A toolkit for reproducible reinforcement learning research.
wtt.://wwwwwwwooooo//////// / 2019.
13


Reviews:
[{"review_id": "T7i_Zz5j9l4", "reviewer": "Reviewer_zqhh", "summary": "The authors proposed a method that combines differential equation modeling with reinforcement learning to predict the progression of Alzheimer’s disease. \n\nIn terms of contributions, the authors claimed that this study is the first to use reinforcement learning for disease progression. During my non-exhaustive literature review there does not seem to be clear objections against this statement. The authors also claimed that their model showed a 11% lower prediction error compared to a state-of-the-art. Moreover, the authors mentioned the so-called “recovery/compensatory processes” that is present in Alzheimer’s disease could be observed in their proposed model, showcasing the interpretability of the model.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tPersonally, I am in favor of the approach the authors take, where they use explicit equations instead of an arbitrary machine learning model to represent processes and behaviors that are relatively well-studied. In this way they utilized domain knowledge in an intuitive and explainable manner.\n2.\tThe authors tend to think proactively about interpretability of their methods, and they related their observations with the “recovery/compensatory” process known in neurobiology.\n\nWeaknesses:\n1.\tIn introduction, the authors mentioned that existing mathematical models cannot faithfully represent all factors relevant to Alzheimer’s disease. Based on that, it would be reasonable to either refine the equations or use a different model to take care of the gap. However, it is not very intuitive why reinforcement learning is a natural choice. It would be better if some insights and rationale is given.\n\nAdditional feedback:\n1.\tIn Figure 4B, it might be helpful to indicate the correlation coefficients, even though it is mentioned in the text. Also, it would make more sense if the predicted vs. ground truth size of PFC is included – it looks weird when both HC and PFC are described in the text but only HC is shown in the corresponding figure.\n2.\tCaptions of Figure 5 needs a bit improvement. It is not clear which model variant Figure 5B and 5C correspond to.\nIt would be interesting to see whether the model can perform consistently when choosing different follow-up times (year-1, year-2, etc.) as the initial point. For example, the prediction of N years after year-0 shall be reasonably similar to that of N-1 years after year-1. At the same time, I am aware that some data might only be available at year-0 and this might not be an easy investigation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing seems to be concerning.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Qcna_fMSTwg", "reviewer": "Reviewer_2T27", "summary": "\nThis paper addresses the problem of predicting long-term Alzheimer’s disease (AD) progression. It proposes a model that combines domain knowledge on the evolving relationships between factors affecting the brain together with a data-driven approach to learning the missing relationships. The learning objective used to learn the data-driven components satisfy a general criteria about the working of the brain, which maximizing cognition while minimizing the cost of supporting cognition. The authors propose using differential equations (DEs) to capture domain knowledge and reinforcement learning (RL) to learn the data-driven component. The model was validated on the ADNI data set and was found to outperform SOTA baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper addresses a very important disease that presents one of the biggest emotional and financial burdens to patient care takers and healthcare systems, and presents an interesting approach for combining domain knowledge and data-driven model components. However, the model was only evaluated on a very small sample size (160 patients from the ADNI data set) and did not consider a lot of previous work that developed disease progression models for the same disease and the same data set. Because of these reasons, I think the paper belongs in the **borderline acceptance** category.\n\nFew comments and questions that I'd like the author to reply to:\n\n- It is unclear to me what is the reasoning behind using RL to solve the optimization problem for $I(t)$. Since we collect all the data retrospectively and run the model on an offline, doesn't it make more sense to solve the optimization problem for all $t$ using an offline approach?\n\n- What is the clinical significance of the results in Table 1? The MSE and MAE by themselves do not reflect accuracy of predicting clinically-useful phenotypes of dementia. Did you consider predicting discrete 10-year outcomes (e.g. conversion of mild cognitive impairment to AD) and evaluate the model accuracy instead? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations were addressed. No negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5+ hours"}, "extra_text": {}}, {"review_id": "FizA_4U2dqx", "reviewer": "Reviewer_VwXC", "summary": "The paper proposes a disease progression model for Alzheimer's disease (AD) that combines data-driven approaches capture trends in data over the course of the disease, which have proved successful over recent times, with more mechanistic biophysics-based models of how pathology spreads over the brain.  The authors introduce the use of reinforcement learning to learn update rules for regional brain quantities as a function of time thereby informing interactions between processes and measurements of neurodegeneration. Experiments with simulations confirm basic efficacy of the estimation procedure; experiments with ADNI data show some promising predictive power and demonstrate potential insights into disease biology.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a timely topic and extends the state of the art by bringing together data-driven models and mechanistic models.  The use of reinforcement learning is a nice idea, highly novel, and experiments show it makes a genuine impact on predictive ability.  The model overall ends up, as any model of such a complex process, making a lot of simplifying assumptions and so arguments in the paper about \"explainability\" in comparison to black-box fully data-driven techniques should be tempered by the fact that explainable observations are only meaningful if/when those assumptions hold. Nevertheless, the essence of the idea here extends as our understanding of AD improves and biophysical models become more accurate.  Overall this is a nice idea, although should be considered as a demonstrator of what's possible by fusing data-driven and mechanistic modelling approaches and avoid strong claims on what the model outputs actually say about AD; some of the results are anomalous to say the least, but the basic principles of the learning algorithms that enable such approaches are sound in my opinion.\n\nSpecific comments for the authors to consider:\n\n1. Line 184 - I don't understand how the model can be estimated without functional MRI data informing on brain activation. It is unclear from the section that describes the data set, but this line suggests the paper uses no functional MRI at all; is that true and if so perhaps the authors could add some intuition to the formulation in Eq 9 explaining the consequences of not having this data available and how the I parameters, which depend on activation, can be defined without it.\n\n2. Figure 4 desperately needs units to make any sense.  Particularly in panel A where the errors are all of order 10^-30 - I assume they are basically zero - but it depends on the scale of the measurements.  Does this suggest that the simulations use no noise perturbations of any kind so really that \"validation\" is simply a sanity check that the code works in estimating true parameters from data directly sampled from the model?\n\n3. Section 4.2. can the authors confirm that the brain region sizes used to evaluate the model are not part of the training set to which the model is fit?  What is the ratio of hold-out data and how was the full data set divided into training/validation/test sets?\n\n4. Figure 4C. The result that normal subjects appear to decline at the same rate as MCIs and much more quickly than SMCs is indeed highly anomalous - likely wrong.  This highlights some limitations in the model as it stands and I think it's important to get to the bottom of this kind of anomaly, although it doesn't detract from the potential of the learning algorithms that are the centrepiece of the paper.\n\n5. The Introduction states  'we observed \"recovery/compensatory\" processes that mitigate the effect of the AD on cognition [11], which could not be observed with state-of-the-art models', which the qualitative discussion in section 4.4 reiterates.  It's unclear to me exactly what the authors mean by this - is it the opposing trends in hippocampal and frontal activity/processing-load that Figure 4B, 4C show?  The arguments that this genuinely reflects true trends in patients brains are very hand wavy and over-played in my opinion.  This either needs clarification or toning down substantially.\n\n6. Line 295 - ref [28] doesn't appear to relate to event-based models - do you mean Fonteijn Neuroimage 2021?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not really discussed, although I don't see this as a concern for this work - it is well motivated.", "review_text": "", "extra_scores": {"ethical_concerns": "Also not discussed, but again personally I don't see it as a problem.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Eo87YXj-Uiv", "reviewer": "Reviewer_31zC", "summary": "This work presents a method for modeling dynamic systems that combines mechanistically motivated differential equations with data driven models learned by RL.  The method is applied to AD progression, measured by MMSE (a cognitive function test) scores.  The key idea is that a mechanistic understanding of the system under study may be incomplete -- e.g., in a system of coupled differential equations, a quantity and its time evolution may be unspecified -- but that it may be possible to fill this gap by learning a reasonable model for the missing function.  This method is applied to AD progression in a real world dataset with mixed results - although adding the learned dynamics model improves accuracy (measured by MSE) a great deal relative to the DE baseline, it does not do as well as a pure ML based baseline (SVR).  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work presents a fascinating approach to modeling dynamical systems in which there is a partial mechanistic understanding of the dynamics, enabling the specification (and fitting from data) of a detailed but incomplete system of differential equations.  Specifically, let $v$ index brain regions.  They hypothesize a mechanistic model of AD progression involving brain region size $X_v$, regional activity $Y_v$ (information processing per unit size), and a $I_v$ representing total information processing in each region.  Note that $Y_v X_v = \\gamma I_v$ - $X_v$ can be directly observed from brain scans, as can $Y_v$ (via functional MRI). The hypothesis further specifies - and this is the crux - the time evolution of $X_t$: it is thought that degeneration of a region $v$ (i.e., changes in size) is governed by amyloid deposition and activity $Y_v$, with patient specific modifiers.  This is formulated as  a set of differential equations.  But these differential equations are incomplete - they specify the time derivatives of $X_v(t)$, but not $Y_v(t)$, or equivalently, $I_v(t)$.  This is analogous to trying to solve a linear system in 3 unknowns from only 2 equations.  So if we want to, e.g., run the system forward from a set of initial conditions, we are stuck.  \n\nHow can we fill in the missing piece, I(t)? Rather than hypothesize a differential equation for $I_v(t)$, this work proposes using RL to learn a policy that optimizes an intuitively reasonable objective - you want to increase $\\sum_v I_v(t)$ to meet cognitive demands but also want to decrease metabolic demands. The action space consists of setting of I_v(t) to meet cognitive demand (minimized $C - \\sum_v I_v(t)$) while decreasing metabolic demand $\\sum_v Y_v(t) = \\sum \\frac{I_v(t)}{X_v(t)}$.  The dynamics of the RL system are determined by the system of equations, with parameters fit to training data (with some caveats due to the fact some patients do not have complete measurements for $Y_v(t)$) - the key link is that increased activity can lead to degeneration (negative changes in $X_v$).  Now we can start from initial conditions, say year 0 of AD, and then run the system forward in time using the learned policy in place of a discretized differential equation for $I$ to forecast progression, and compare to the observed trajectories and other forecasting methods.  In this paper, they do this for a 10 year window, with one year time steps.  There is an evaluation on synthetic data, but I will focus on the evaluation on the real data (ADNI).  I am not familiar with the dataset used. \n\nHow did they do?  There are two comparisons. First, is the usual bake off comparison of their method vs some solid baselines.The new method seems to do very well wrt MSE compared to one baseline, and about the same as another baseline, SVR, in terms of agreement with the observed progression of test patients.  Plotting the data reveals a nuance, however - the SVR baseline fits the observed data shape, which shows a weird increase in mean cognitive function in year 9 followed by a precipitous decline in year 10 - better than the new method, which forecasts (on average) a smoother, monotonic decrease in cognitive function.  The authors explain this away, asserting that the smoother forecast is more realistic.  I agree, and am willing to view this as a case where noisy data is mitigated by a (semi-)mechanistic formulation of the dynamic model. In another context, the authors also raise the issue of missing measurements in the (real) dataset - since it is so sparse (31% observed) it may be that this is a result of an unfortunate pattern of missingness towards the end of the observation period.  This could perhaps be checked (are the patients who are contributing to the population mean at 9 years the same patients contributing to the population mean at 8 years, etc).  \n\nThe second comparison is a bit more puzzling - it shows that the forecasts with and without RL are remarkably similar in shape, with a scaling or constant offset from each other over time.  This suggests that the RL component is mostly setting an offset/constant - this is perhaps not surprising given the form of the differential equations, etc.  But it does beg the question of whether there might have been a simpler way to arrive at a dynamical system that fits the data very well (e.g., fitting a constant relating the initial year 0 prediction to observed on the training data, i.o.w., fitting an intercept), and whether the RL component is learning a trivial policy.  \n\nThere is an auxiliary result - their learned dynamical system also demonstrates compensation, a well known phenomenon in which cognitive demands can be shunted around to different regions to compensate for degradation of some areas.  This seems to me to be an unsurprising result of the RL objective, so is not that noteworthy.  \n\nOverall I liked this work very much as a demonstration of a cool idea I haven't seen elsewhere.  There are elements of the presentation that should be clarified, along with some additional questions that it would be nice to see addressed.  \n\nFirst, it is not clear to me how the baseline without RL is accomplished.  The whole point was, I thought, that we couldn't run the dynamical system forward from initial conditions because we are missing a key piece we will fill in with a learned model.  So how was the baseline run? \n\nSecond, how did the authors handle the amyloid deposition as a function of time, $D_v(t)$, introduced in equation 6, line 142?  It also seems unspecified in the dynamical system - don't you have to model that somehow too? \n\nThird, in figure 5A, it would good to know why, at t = 0 years, most of the estimates are offset from the ground truth mean.  I would have thought t = 0 corresponds to the initial conditions, so what does a forecast from a dynamical system mean at that time? \n\nFourth, I am at a bit of a loss as to how this could be done, but I do wonder whether the RL policy governing $I_v(t)$ is accurate and \"non-trivial\".  This is not possible with the real dataset, and I doubt there are any datasets of fMRI over 10 years of AD progression, but this does seem to me to be an important point about whether this approach is really adding a lot.  Perhaps in the synthetic data the authors could directly evaluate the RL policy governing the time evolution of $I_v(t)$, both when the data generating process is a good fit for the hypothesis, and when it is not?  \n\nFinally, if possible, it would be interesting to see a sensitivity analysis to see how robust results are to misspecification of both the differential equations and the learned model for $I_v(t)$.  In general, I suspect that this could be a big limiter to this general approach - we may have only a very vague sense of the mechanisms (leading to misspecification of the dynamical system) and how to formulate a good RL objective.   \n\nOverall, I really enjoyed this work - it was quite different from the usual ML conference submission and I think it could spark a lot of cool work and discussion even if it doesn't convincingly, in my opinion, beat one of the baselines or work super well.  It's hard to say how generally applicable this approach could be since it does require a lot of (reliable) domain knowledge, and it's not clear to me how well it worked in this case, but I found the approach intriguing and worth introducing to the NeuRIPS community.  I am curious whether this sort of approach has been taken in, e.g., applications of ML to physics.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors point out one limitation - only 2 brain regions were modeled.  I don't think this is a very big deal.  \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the discussion with reviewers, the reviewers have highlighted the merits of this paper. One particular reviewer was unable to update their score of 5 to an accept, however, in private conversation with the reviewers, they all agreed that this paper warrants acceptance. This paper addresses a very important clinical problem, even though the technical contribution and the size of the validation data are limited. The proposed approach for combining mechanistic and RL-based data driven modelling approaches is interesting and worthy of publication.
Decision: Accept (Poster)