Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The contribution of the paper is twofold. On one hand a hybrid model to incorporate background knowledge in a form of ODEs to a Neural ODE model is presented. On the other hand a causal inference tool was provided for time series data. The work focuses on interpretability and clinical decision support.
main_review: On integrating expert knowledge:
The hybrid model is theoretically solid and the assumptions are well presented. However, due to the problem of confounding in real world setting (discussed below) the assumption of autonomous evolution of the expert model is likely violated even if there is no direct feedback from $z^m$ to $z^e$, as there will be a feedback loop through $a$, as the clinician likely use information from the measurements to select the next treatment.

There have been a work on ODE based expert knowledge integration in this year ICLR: Le Guen, Vincent, et al. "Augmenting physical models with deep networks for complex dynamics forecasting." Ninth International Conference on Learning Representations ICLR 2021. 2021. (ICLR2021 is in the 2 month time window before submission deadline therefore this work is contemporaneous)

On causal inference:
There are major issues with the handling of causality.
The data is not sporadically observed but rather a sequence of labelled evens. It is not true that the measurements are missing at random. The fact that the physician order a measurement already indicate something on their belief (the missingness indicator causally depend on some hidden state). 
 
The fact that the model estimate individual treatment effects (conditioning on individual initial state $z_i(0)$) places the model to rank 3 on Pearl's ladder of causation, and require operations on counterfactuals.

The model assumes no unobserved confounders, what is a strong assumption in this particular case, when COVID-19 treatments were followed up in a non-controlled experiment.
All actions are considered exogenous (there is no backdoor path) as shown on Figure 1. C). This is even stronger assumption than "no unobserved confounders" and definitely violated by any dataset recorded in clinical practice. This assumption would mean the physician do not use any information in the previously measured variables when deciding on the next treatments what we passively observe.
Even more complicated dependence structures can exist in real life sequential treatment situations, like the kind addressed by [ J. Pearl and J.M. Robins. Probabilistic evaluation of sequential plans from
causal models with hidden variables.  Uncertainty in Artificial Intelligence 11, 1995, p 444–453. ]
Please define all the assumptions necessary for your method to estimate unbiased treatment effects, and provide a sufficient argument that the dataset in the Experimental section indeed satisfy these conditions.

Real-data experiments: The illustration of clinical decision support is not possible to verify. The paper contains counterfactual statements based on the prediction but there is no evidence these statements are indeed valid. Only reference we have is a possibly heavily confounded observational data. To make claims on value for estimating individual treatment effects like "LHM predicts that immune activity $z^e_1$ would decrease afterwards even without treatment. " we need some experimental data, or a detailed simulation.


limitations_and_societal_impact: The work clearly have societal impact as it  addresses clinical decision support, this is described in the paper. As the work also describes, the expert ODE integration is limited for self contained expert odes .
However, the causal assumptions are not well described. The following questions need to be addressed: Does the method assume "no unobserved confounders"? Does it assumes (strong)ignoraebility?  Does it assume missing at random observation? Are these assumptions at least approximately valid in the application domain?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a latent hybridization model that incorporates pharmacological knowledge into ML models. In particular, it utilizes expert pharmacological variables and regular clinical measurements to improve the optimization process of neural ODEs. The model was evaluated on a synthetic dataset and a real-world COVID-19 dataset.
main_review: Pros:
1.	This idea of leveraging pharmacological models in ML modeling makes sense. The clinical analysis for the model is also interesting
2.	The performance on COVID-19 dataset looks good, especially under small dataset size.

Con:
1.	There are some unclear assumptions. For example, Eq. 6 assumes the independency between two measurements Y(t0:T) and Y(0:t0) for patient i. Does this discard the temporal dependencies in clinical measurements?  This is important since the paper claims to model a system's temporal behavior.

2.	The paper makes a broader claim but only demonstrates a very narrow application. For example, it claims that the model integrates pharmacology model into neural ODEs for "predicting the progression of disease under medications". Usually when we claim a model is designed for predicting disease progression, we need to at least evaluate its utility against chronic diseases or neurodegenerative diseases that actually have longer term progression. However in this paper, the model is only evaluated on two COVID-19 dataset (synthetic and real-world).  It is unclear whether the model would outperform existing models on real disease progression tasks against state-of-the-art baselines. Therefore, without major improvement to demonstrate the general utility, the paper is just a narrow-focused application paper, rather than what it claims.

3. The experiments are weak due to missing many important baselines.  The authors only compare with Time LSTM as the non-ODE model, which is not persuasive considering there are lots of more recent DL-based disease progression models, as well as DL-based disease progression/prediction models that incorporate expert rules/knowledge.



limitations_and_societal_impact: No.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a new model called latent hybridisation model (LHM) to integrate a system of expert-designed Ordinary Differential Equations (ODEs) -- from domain knowledge, with Neural ODEs -- learnt from data. The aim is to enable learning in small sample regimes and have interpretable latent variables. Empirical results are shown on a clinical task of predicting temporal physiological variables after treatment, using a pharmacological (expert) model of the treatment effect.
main_review: My comments along the 4 dimensions:

1. Originality

The task/problem addressed is well known. It is clinically important and challenging.

The model appears, to me, as a new combination of older known techniques. It is well developed for this application. Previous related work has been adequately cited but the explanation of how the proposed model differs from previous models could be improved further.

2. Quality

The model and its inference appears to be sound. However, the experimental results could be further improved to better support the claims and provide more insights into the strengths and weaknesses of the model.

- For example, in both the simulation and real settings, only one-time treatment has been considered. In dexamethasone treatment the dose can be once daily and multiple time points could be considered along with the experimental evaluation. In other applications this may be even more important to consider. 

- Simulation settings could be further expanded, e.g., by varying values of T and characteristics of individual time series (to model clinical data).

- I believe the results show the RMSE aggregated for all the physiological variables (both simulations and real data). Are some variable's predictions worse than others? In real clinical data, the variables differ in data type and frequency of measurement. How do the predictions differ with respect to these characteristics? Model performance could be analyzed along these dimensions.

- Similarly the impact of expert knowledge could also be quantified better. As a practitioner, one would want to know when to use a particular expert model, and if there are multiple models, which ones to choose? 

- Is the improved model performance due to the expert model added or is it just because LHM has more latent variables/capacity compared to other models compared? An experiment could be suitably designed to answer this, e.g., by comparison with another hybrid model where an alternative "false" model of similar capacity has been added. I think this is important to convince the reader of the utility of the expert model.

- Real data results are shown on only 1 dataset. More datasets (that are available, for other treatments/diseases) could be added.

- Is there some way to evaluate the correctness of the results in fig. 4, rightmost column? Perhaps through some proxy clinical variable. Similarly, to convince the reader of the interpretability/accuracy of all the expert latent variables, suitable comparisons could be presented.

- What are the disadvantages of LHM, compared to existing models? Does it require more training time? Does it get biased by the expert model - if so, how? Under what circumstances would that lead to worse results?

3. Clarity

The paper is clear, very well written.

4. Significance

The model is potentially useful in practice but its advantages and limitations need more assessment (as described above). It addresses two challenging problems (small samples, interpretability) in the clinical context and develops a good solution for them, as shown in the preliminary experimental results. In my opinion, the work is promising but incomplete.


limitations_and_societal_impact: Societal impacts have been adequately addressed in section A.2.

Limitations: briefly mentioned in section 6 but could be elaborated further. What is mentioned now is the current modeling inadequacy of the model. It's limitations with respect to its performance and other existing models needs to be described in more detail. This can also be done empirically (as discussed earlier).
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a novel hybrid modeling framework, the Latent HybridisationModel (LHM), that imbeds a given pharmacological model (a collection of expert variables and the ODEs that describe the evolution of these variables ) into a larger latent variable ML model (a system of Neural ODEs). 
Nice approach and Extendable to other fields and applications
main_review: Originality:  
Methods are not new, but there is a novel combination of previous techniques. It is clear how this approach is distinguishable from the others and most of the related works are sufficiently cited 

Quality: I
As a strength,  the theoretical grounding and the empirical evaluations are sufficient and they provide nice details about both simulations and real datasets. I would say that I found the approach very interesting and towards the appropriate control/exploitation of the expertise (pharmacological or other) by the models.
It seems a rather complete work.

As a weakness, I would mention the manuscript structure. I do not understand the reason that authors, although they have some clear ideas, they chose a rather “unorthodox” manuscript structure in my opinion. 
Briefly, 
1.	I would propose Related works before Methodology
2.	3.5 subchapter seems more like a Discussion part
3.	I would propose enrichment in the Discussion part… Why LHM works better than the others, where the others make it better and in which conditions. Mostly take some already written arguments and elements from the manuscript and create a structured discussion chapter. 

Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)
ideas are clearly presented, I have already made the comments about the structure in order to avoid repetitions.

Significance: 
The results are important and such method may help not only practitioners but also other fields in creating personalized models(from the clinicians point of view) that are related to their daily practice. 
limitations_and_societal_impact: Some Limitations are presented in the manuscript  but they have not a special chapter or paragraph. It is always the same issue of unorthodox structure that limits the reader and the  full appreciation of the ideas. I kindly recommend to revisit your manuscript's structure and create a Limitation paragraph (like the medical articles) inside the Discussion chapter, for both methodological and population/simulation/experimental limitations.
review_title: null

Meta-review: This work integrates a system of expert-designed ODEs with a machine-learned neural ODEs to improve the prediction of disease progression. While the paper generated much discussion, ultimately shortcomings related to issues around causal inference from observational data remain. While the authors claim that they are not promoting this work as a causal inference algorithm, nonetheless the text states in Section 3.5 that the approach "can provide clinicians with prediction of the disease progression given the treatments, enabling the clinicians to design the best treatment plan for the patient at hand," which sounds a lot like estimating counterfactuals. Thus significant reframing seems necessary. I encourage the authors to carefully consider reviewer feedback, in particular with respect to the framing and empirical results, when working on their revision.
Decision: Accept (Poster)