Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed MetaPhysiCa, a treatment to OOD initial conditions in PIML, via meta-learning algorithms with structural causal models.

strength_and_weaknesses: Strength:
I like the motivation of this paper, and the illustrative examples are effectively demonstrating the limitations of vanilla PIML.

Weakness:
I would prefer that the authors state clearly the current limitation of this approach. The SCM adopted in this paper would not scale to the realistic, complex ODE cases, and either some experimental investigation or discussions on the limit of this approach would be beneficial. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow, very novel, and with sufficient reproducibility. 
summary_of_the_review: In summary, this is an interesting paper. I am willing to raise my score if the authors could conduct a more comprehensive investigation on the current scalability of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on dynamical system prediction in the case where the initial conditions of test trajectories are sampled differently from that of training. Metaphysica is introduced, a model based on the identification of causal models which are then finetuned on the new test trajectories to maintain good performance on the new OOD trajectories.
strength_and_weaknesses: First, I would like to point out that the paper is, in my opinion, very poorly written. Almost all of the arguments are detailed in a convoluted way that is difficult to read, especially section 2 and 3. Figure 1 needs to be re-worked. It is difficult to separate contributions from their poor presentation, so my review may contain misinterpretations.

Many points bother me about the method used. My main concern is on the adaptation phase: it seems that you train your model on the test data. This is a very dangerous choice, which requires a lot of carefulness about how to achieve it. Here are some leads:
- The amount of data on which MetaPhysiCa is adapted in test-time is not so negligible, according to the appendix. The risk of overfitting is huge. I suggest using a validation set from the same distribution as the test set but different to adapt the model, and then test it on a completely unseen set of initial conditions.
- It would also be interesting to see how the model reacts if it is again exposed to the training trajectories after adaptation. My fear is that MetaPhysiCa performs poorly, due to overfitting on testing data.
- When MetaPhysiCa is trained on constant $W*$ tasks, the adaptation phase should be unnecessary. Its ablation in this configuration would confirm that the model is indeed able to identify the causal model.

I strongly recommend that authors demonstrate indisputably that adaptation during test-time is reasonable and relevant.

I also disagree with several claims:
- I see many similarities between MetaPhysiCa and SINDy: the two models seek to identify an analytical model describing the dynamic system. I would have appreciated a much clearer discussion of the differences between MetaPhysiCa and this baseline. I particularly disagree with the sentence "These transductive methods, however, do not transfer knowledge learned in training to predicting test examples unseen during training". SINDy (and EQL to a lesser extent) makes it possible to identify an analytical dynamic equation from the data. This equation is (theoretically) general and therefore perfectly transferable to other initial conditions. The failure of SINDy seems very surprising to me.
- Moreover, MetaPhysiCa seems to me to be able to (even forced to) identify the real dynamic equation. This is a simple check to perform. It would then interesting to verify that the causal graph is correctly identified, which would make it possible to generalize OOD. That being said, I don't see what prevents SINDy from doing the same, especially in the case where the $W*$ parameters are fixed. I would appreciate an in-depth analysis of this.
- The authors justify the failure of methods like APHINITY or NODE by arguing that the neural networks are not algorithmically aligned to the problem. This is quite reasonable, but in my opinion makes the comparison unfair, since the basis functions necessary to solve the problem are directly implemented in MetaPhysiCa by hand. What happens if the structure of NODE is adapted to suit the problem?
- The authors propose constrain their causal graph to be minimal. Although this is a common assumption for the identification of the equations of a dynamical system, a discussion on the relevance of this choice (and its interest in practice) seems to me necessary.

Finally, section 5 shows results that I don't understand:
- The total failure of SINDy and EQL seems very surprising to me. These methods identify an analytical equation which should therefore produce relatively correct results. Their failure requires further analysis.
- The figures presented in addition to the tables show predictions from baselines and MetaPhysiCa. However, I don't understand how the initial conditions (ie the starting point of the curves) can be different from one baseline to another. Could it be that the baselines are evaluated on different trajectories? Authors must justify this.
- I suspect the Deep learning baseline to overfit on the relatively simple system chosen by the authors. However, I could not find information about the size of the networks used for NODE and APHINITY. I believe that the OOD failure of this model maybe tempered with smaller models less subject to overfitting. I would appreciate if the authors could discuss this.
clarity,_quality,_novelty_and_reproducibility: The paper is very poorly written, and novelty feels limited. MetaPhysiCa is very close to SINDy but with an added adaptation phase using fine-tuning on test data which is very problematic. Reproducibility is not straightforward as many details are missing.
summary_of_the_review: The paper suffers from its poor writing, and its lack of experience and analysis to clearly justify the successes of MetaPhysiCa and the failures of other models. As it stands, I do not recommend this paper for acceptance. Nevertheless, the proposed method is interesting, as is the task, which seems to me to have potential.

## EDIT
The authors provided very convincing answers to my questions. In my opinion, there are still two weaknesses to the proposed method:
1) This is based on a dictionary of functions, and the model assumes that this contains the functions necessary to model the dynamics. This is a difficulty that is also found in SINDy, and many other methods.
2) Experiments are performed on relatively simple dynamical systems. The contribution would have been reinforced if it could have been applied to more complex systems, even real ones, although I am aware of the difficulty of bringing together a dataset for this task.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Current deep and physics-informed machine learning models struggle to correctly forecast dynamical systems in out-of-distribution (OOD) settings. This paper proposes a new approach consisting of a meta-leaning strategy combined with causal structural discovery. The method is evaluated on standard ODE benchmarks, showing better forecasting performances on OOD initial conditions.
strength_and_weaknesses: Strengths:

This paper is well-written and very enjoyable to read. It tackles a very relevant issue that undermines physics-informed machine learning in OOD settings. The method is clearly presented and well positioned with respect to the state-of-the-art. The combination of the structural causal model with meta-learning is an appealing idea. This method outperforms with a large margin other competitors on OOD initial conditions, while remaining equivalent in-distribution. 

Weaknesses:

W1: The experimental section contains only forecasting performances in and out of distribution. Without an in-depth analysis of the model, for example an ablation study, it is hard to understand what elements make this model more successful. Could you provide (possibly in appendix) a more thorough analysis? For instance, what is the impact of the SCM? Have you analyzed the discovered causal structure and what interpretation can be drawn? 


W2: Moreover, considering the robustness to ODD initial conditions is fine and interesting, but why not addressing in the paper the robustness to ODE system parameters? This is done in CoDA and DyAd and I a wondering if this method would still have such a performance gain in this context. This would be interesting for the community to analyze what source of ODD interventions have the most impact.

W3: The authors make experiments on common benchmark ODEs (damped pendulum, predator prey, epidemic). Why not evaluate your method on more complex PDE equations? Published methods evaluate on 2D reaction-diffusion and Navier Stockes (for CoDA) and turbulent flows and sea surface temperature (for DyAd). 

Questions:

Q1: The loss minimized in practice is a MSE between the derivatives along the trajectories. How is the derivative estimated for the ground-truth trajectory (finite differences)? Are these estimates noisy with noisy data and how does it affect performances? 

Q2: In Eq 3, the authors write the global optimization objective. Is this the real bi-level objective optimized in practice? An algorithm script could help to clarify the training scheme. 


clarity,_quality,_novelty_and_reproducibility: This hybrid method combining meta-learning and structural causal discovery seems quite new to me, and very promising. The causality part should be more studied and interpreted (W1). What is the link with other recent approaches leveraging causality in PIML, eg [1] ? 
The implementation details are missing (in paper or appendix). 

What are the neural network architectures used? (a schematic of the model could be fine for understanding but is not mandatory) How are chosen the hyperparameters? How are chosen the basis functions for the SCM? Are they fixed manually or can they be learned? The neural network details should also be reported for the baselines for a fair comparison. Besides, will the code be released?

Typos:

Page 2: the definition of xi_star is missing.

Page 7: “a structure that (is) minimizes”

[1] Sifan Wang, Shyam Sankaran, Paris Perdikaris, Respecting causality is all you need for training physics-informed neural networks
summary_of_the_review: In summary, this hybrid approach for OOD dynamical forecasting is appealing and shows promising results. However, in the current state of the paper, it is hard to really understand the working principle and if the method would apply to more complex physical systems.

EDIT: after discussion with other reviewers, I have still concerns on the clarity of the method. The positioning with respect to Coda and DyAd is not discussed and  the reason that this method outperform the other by an order of magnitude is still unclear for me. Besides, I do not understand in experiments why Aphynity is largely inferior to the data-driven Neural ODE for the in-distribution pendulum. 
I return to my original pre-rebuttal score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider the challenging problem of learning robust solvers for ODEs in physics-inspired machine learning, mainly when initial conditions at the test time are the out-of-distribution. First, with simple experiments on well-motivated simulated physics tasks (such as damped pendulum systems and predator-prey), the paper presents the limitations of existing PIML algorithms in dealing with OOD initial conditions. Next, the authors propose meta-learning algorithms that embed a structural causal model to identify the most suitable model for the ODE task to tackle this. Finally, evaluating the normalized RMSE, experiments across multiple domains suggest that MetaPhysiCa is competitive on in-distribution tasks but significantly outperforms the baselines in OOD settings. 
strength_and_weaknesses: ### **Strength**
+ The paper is well-written, with pedagogical examples highlighting the limits and challenges of current PIML algorithms. The authors do a great job presenting their contributions in the context of existing literature. 
+ By effectively combining the strengths of meta-learning algorithms and causal inference, the proposed algorithm significantly improves performance on OOD tasks in well-studied benchmark physics simulation tasks (such as damped pendulum and predator-prey).

### **Weakness**
- The assumptions of realizability in the family of structural causal models seem restrictive and might not hold in more complex ODE tasks. A discussion on the scope of these assumptions would significantly help understand the algorithm's limitations and strengths. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**: The paper is well-written and easy to follow. 
**Quality**: The authors compare against relevant baselines and present significant improvements in OOD tasks.
**Novelty**: The proposed algorithm is a novel contribution that builds on well-studied meta-learning and causal inference ideas. 
**Reproducibility**: The authors present information sufficient to reproduce key-experiments and results from the paper. 
summary_of_the_review: The paper studies challenges in forecasting dynamical systems with OOD initial conditions and the limits of current PIML-based approaches. Building on insights from causal inference and multi-task learning, the authors propose MetaPhysica. This hybrid algorithm performs a structural causal search on a family of SCMs (under realizability) to perform appropriate interventions. With extensive experiments and comparisons to relevant baselines, the proposed algorithm outperforms prior work on OOD tasks by a considerable margin.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method called MetaPhysiCa, which is a physics-informed machine learning method to better model dynamical ODE systems tested under OOD Initial conditions with changing or non-changing ODE parameters. The approach is making use of various meta-learning and causal structure discovery techniques in combination with a predefined set of possible basis functions to parametrize the RHS of an ODE. Authors validate their approach on 3 different ODE setups (damped pendulum, predator-prey and epidemic modelling) outperforming multiple baselines in OOD test scenarios.
strength_and_weaknesses: Strengths:
- The paper is very clearly written and the problem setup, as well as the proposed method, are presented comprehensively.
- The authors validate their method on three prominent ODE tasks and benchmark their method against a multitude of relevant baselines
- The paper touches upon an important problem setting and I appreciate the general idea of the method to obtain better OOD generalization via causality-inspired regularization and modelling assumptions.
- Nice and convincing experimental results
- Nice contextualization of the method with respect to prior works

Weaknesses:
- The method appears to make several rather strong assumptions on the underlying ODE system that make the method only apply to a smaller subset of ODE systems. (1) The underlying ODE system is assumed to be parametrized with the task-specific parameters W acting as coefficients in a linear combination of basis functions, which happens to be exactly the case for the studied toy experiments. I thus would expect the method to potentially exhibit several issues if this is not possible. The three environments are all building upon the sparsest possible causal structure in that every RHS is a single term such that the sparsity regularization will always help as long as exactly one entry per row in the structure parameter matrix survives. It would be important to understand how the method performs if there are linear combinations of more than one basis function in the RHS. Finding a good trade-off in the right regularization through $\lambda_\Phi$ and its effect on OOD generalization might quickly become very challenging. While I understand that the more general setting is much harder and I appreciate the progress over baselines in the here studied experiments I would have appreciated it if these limitations and assumptions are stated more prominently as it seems the method predominantly benefits a lot from being given optimal basis functions.
- Presentation (minor): The plots in Figures 4 and 5 should be enlarged and could be improved as the labels and legends are barely readable.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and easy to follow and the experimental evaluation seems technically rigorous and sound. Various components and assumptions of the method such as sparsity regularization or meta-learning are well-known regularization techniques but it is the combination of all these components that has some novelty to the best of my knowledge.

summary_of_the_review: I think this paper is a nice contribution by proposing a new physics-informed ML method for ODEs that exhibit more robust generalization with respect to OOD initial conditions and makes use of several important concepts and necessary assumptions to achieve this. While I appreciate the number of experiments and convincing performance over existing baselines I have mentioned a few remaining concerns as the studied setups appear to be particularly easy for the here proposed method. I am concerned that the method might not be very applicable beyond such very simple ODEs that adhere to the structure assumed by the method. I, therefore, lean towards rejecting but I am willing to change my rating if the authors can address my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper deals with the problem of robustness to initial conditions in physics-informed machine learning. The authors point out the brittleness of state-of-the-art methods with respect to changes in initial conditions' distribution, especially when they fall outside the training support. They introduces an approach for improving this robustness, based on a combination of causal structural discovery, meta-learning, and hybrid transductive-inductive test-time adaptation. Experiments are conducted on 3 datasets: Damped pendulum, predator-prey systems, and epidemic modelling. \
The paper initially received mixed reviews, with two borderline reject (5), one reject (3), and two borderline accept (6) recommendations. The main concerns pointed out by reviewers related to the clarity of the presentation, unjustified claims, and several issues in experiments, especially related to the performances of the baselines. The author's feedback was very detailed, with long answers and some new experiments and ablation provided. After rebuttal, the reviewers' grades have been increased to 6 6 6 6 8\
The AC's own reading of the paper reveal additional shortcoming in the paper: 
- The claimed contributions related to adapting the model in OOD initial condition setting is based on a combination of meta-learning, causal discovery and hybrid transductive-inductive adaptation. The related work on these topics is, however, not discussed in section 3. One would at least expect a short literature review on these field and how they should be adapted in the context of the submission. Also, there is no positioning with respect to the adaptation methods compared in the experiments, e.g. DyAd and CoDA.  
- The paper is poorly presented. Until section 4, the paper mixes up general elements of motivation or experimental results. The method description starting at the end of page 5 is too short, insufficiency illustrated and detailed, making its insights unclear and difficult to follow. 
- Experimental conclusions are unclear: the split on methods between deep learning and PIML questionable, since the classification would expected to be between adaptation and non-adaptation methods. The baseline seems not have been properly tuned or evaluated in artificial contexts where they fail. At the end, the proposed approach outperforms recent adaptations method by an order of magnitude, and the explanation of this large gain is left without a thorough analysis.

During the virtual meeting RhnK8, Rrgk4, and Rt2KB discussed about the points raised by the AC. Overall, the weaknesses highlighted by the AC echo some of their own concerns, but added some points that they overlooked. The combination of issues make the reviewers agree that the paper should not be accepted in the current form (see below). RBmNe was not present in the virtual meeting but participated to the discussion: he also hear the AC's point and finally voted for weak rejection.
After the virtual AC meeting, there was a consensus among active reviewers that the submission would highly benefit from a full re-writing and a clarification on experiments.

The AC considers that the paper addresses an interesting and important problem in physics-informed machine learning, and the general idea of combining causal structural discovery, meta-learning, and hybrid transductive-inductive test-time adaptation is promising. However, the submission cannot be accepted in the current state. The proposed method should be expanded, the insights given in a clearer way to reach a broad audience, and the positioning needs to be specified. The experiments should specify the problem statement and justify the choice of the baselines. 
The AC thus recommends rejection but highly encourages the authors to re-submit their works based on reviewers' feedback.
justification_for_why_not_higher_score: - Poor paper presentation
- Lack of positioning with respect to the most related works from the literature
- Consolidations in experiments
Decision: Reject