Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work tackles the task of forecasting dynamics in different domains simultaneously. Using an encoder which is trained to determine the task, the inferred latent vector is then used to adapt a forecasting network to the task at hand. Experiments on three datasets linked to fluid dynamics are then conducted to assess the proposed model.
main_review: Pros :

- This is an interesting problem which is quite timely given the development of the field of forecasting physical dynamics using neural networks. 

- The proposed solution seems sound and principled. Moreover, it is well motivated and the writing was quite clear.

- The different additions made to the forecaster network are also quite interesting, I especially liked the AdaPad solution to deal with boundary conditions. Conducting an ablation study also considerably strengthens the paper. 

Cons :

- All experiments are conducted on somewhat similar datasets, which are based on fluid dynamics PDEs. It would be nice to see how the model deals with other families of dynamics. Especially given the fact that the contributions of this work seem geared towards practical considerations.

- The setting of the experiments should be more precise and additional details should be given: how are the different datasets constructed, what supervision is there exactly regarding the different tasks, how many domains are there in each dataset and what are the differences, how is the balance between the different domains ect.
summary_of_the_review: This is a good work on a timely subject. The contribution is not groundbreaking but should be significant enough to warrant acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of learning a deep learning model for dynamics forecasting which generalizes to changes in dynamics. These changes can be induced by different parameters, boundary conditions or external forces. The proposed model takes a meta-learning approach and proposes to partition data into different heterogeneous domains. It consists of two components: an encoder which infers time-invariant features given observed domain data and a forecaster which predicts the dynamics given these features. The paper evaluates the proposed approach on several datasets and provides some theoretical insights.
main_review: +

* This paper addresses a new and interesting generalization problem for dynamics forecasting 
* It proposes a model to address different changes in the dynamics. 
* Evaluation is done on relevant datasets with several baselines and some ablation studies.

-

* The applicability of the proposed approach is restricted to problems where relevant weak supervision from task parameters is available. This seems like an important limitation in real-world applications. How valid is this scenario? The question of choosing relevant parameters for weak supervision is important for applying this model to other datasets, yet the definition of these parameters is unclear; how robust is the model when chosen parameters are not useful ? The performance of Wrong_enc (Table 2) tends to say that this model will then fail. 
* It is unclear why the model can adapt to changing boundary conditions with AdaPad as it generates them from features $\hat{z}_c$ extracted from data inside the domain and weakly supervised by quantities unrelated to the boundary condition (e.g. mean vorticity or season). 
* The theoretical analysis, inspired by existing work in multi-task learning / domain adaptation, has some limitations and does not add much value to the paper. I have some concerns with the domain adaptation upper-bound to the target error in Theorem 3.4 and Proposition 3.5. This upper-bound is not minimized thus the target risk can be high i.e. the model is not guaranteed to adapt well. Moreover, the validity of the theoretical analysis is unclear as several assumptions may not be verified e.g. bounded loss in Theorem 3.1, Proposition 3.3; lipschitz continuity in Proposition 3.5. Theorem 3.4 requires that the assumptions in Theorem 2 in Redko et al 2017 are verified, yet these assumptions are not mentioned in the paper.
* Some ablation studies are missing: 1) the contribution of each term in equation (2) and 2) the dimensionality of $\hat{z}_c$ which is fixed arbitrarily.

Other questions: 
* It would be good to better explain how the experiments include changing boundary conditions between domains. The testing scenarios only mention different initial conditions or external forces.
* Why do the baselines ResNet-c and Unet-c not adapt well despite having access to relevant weak supervision (p8)? This is the same information used by the proposed model to adapt.
* How redundant is the time invariance term (3rd term in equation (2)) with the invariances enforced in the architecture of the encoder?

summary_of_the_review: This paper tackles a new generalization problem for dynamics forecasting and proposes a model supported by experimental results. However, this model can only be applied to problems with relevant weak supervision which may not always be available in practise. Moreover, the definition of relevant parameters is unclear and the robustness of the model to the choice of these parameters is not measured which may restrict its application to other datasets. There are also unclarities on the ability of the model to adapt to changing boundary conditions with AdaPad, some ablation studies are missing and I have concerns on the theoretical analysis which brings limited value to the paper. For this reason, I am giving this paper a weak reject.

--- Post-Rebuttal comments ---
I thank the authors for their response. After studying it, the theoretical results still have some major issues and feel disconnected from the model. In particular, key assumptions are not enforced in the model (e.g. lipschitz continuity) and the generalization error of the model in Th3.3 is uncontrolled as the upper-bound is not minimized by the model (the Wasserstein distance between domains is fixed and is high in all generality). Its use for the model is thus not very convincing. On practical aspects, the capability of handling boundary conditions should be better justified and evaluated. For this reason, I keep my score unchanged and recommend rejecting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper suggest a remediation for a common problem for dynamics forecasting which is the lack of generalization to other domains/tasks. The author suggest to tackle this with via a 2 component architecture, one for learning the task and one for forecasting. In empiricial experiments the authors show the practical feasibility of their approach.
main_review: As a caveat: I'm not an expert in the area, so my review remains on a superficial level consequently for which I apologize. I overall liked the paper quite a bit, the question discussed is relevant, the empirical evaluation is very good, the theoretical results seem as relevant as they would get and the related work discussed is crisply presented and relevant. 

One question I would have is that results in Table 1 are overwhelmingly good with only UNET-c coming close. Do we know for these tasks what the "theoretical" upper bound (e.g. by the right PDE system) would be? Is it computationally even possible to compute this upper bound? I'm wondering how much of a gap there still is too close. 

In a similar vein, what is the intuition behind DyAD + ResNet being better than DyAD + UNET mostly? Are there some complementary strengths between DyAD and ResNet that this combination can exploit better than DyAD + UNET?
summary_of_the_review: This is a good paper that I'd like to see accepted for its combination of theoretical results, empirical results and methodological novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is interested in learning general forecasting models for physical dynamical processes. The paper proposes a decomposition of such a model into an encoder that captures the innate properties of the system, and a forecaster that autoregressively makes predictions conditioned on the encoded properties. This is framed as a meta-learning approach, and is shown to substantially outperform single-task approaches and off-the-shell meta-learning approaches across multiple datasets. The paper provides some theoretical analysis, and qualitative analysis of what is learned. Overall, the paper shows that learning shared models across domains is an important and fruitful way forward for modeling physical processes with machine learning.
main_review: Strengths:
- The problem statement is well-motivated. Learning generalizable deep learning models across diverse settings is an important open problem.
- Experiments use interesting and real-world problems.
- Results are strong and appear reliable.
- AdaPad is an interesting idea specialized to the case of physical complex systems, since it is designed to address boundary condition issues.
- Visualizations show the model is behaving essentially as expected.
- Although there are many design choices that go in to the model, each such design choice is well-motivated.
- Aside from some aspects of the theory section, the exposition is generally quite clear and well-organized.
- Assumptions are made clear.
- The fact that the encoder can be trained first and independently of the forecaster should be very useful for further rapid developments.
- Great to see ESE metric used as a complement to raw error.
- Table in Appendix showing alternatives to AdaIn is very useful in increasing confidence in AdaIn for this application.


Weaknesses:
- The biggest concern is the theory section. The multi-task learning and domain adaptation results are general results that are not adequately connected back to the specific model and problem the paper is considering. Yes, it is widely accepted that multi-task learning and domain adaptation can work well, especially when tasks are related in some measurable way, and it can be a useful exercise to restate existing theory in the language of your framework, but what (if any) novel claims is the theory implying? Are there any predictions the theory makes about the particular approach which can be validated in experiments?
- The theoretical bound on error that decomposes the error of the encoder and forecaster is similarly lacking in its interpretation. Yes, it can be a useful exercise to show that the error can be decomposed along the lines of the model, but does this bound somehow suggest that the decomposition results in lower error than a monolithic model? Or is it showing that you can work independently on improving either part of the model and improve the overall error? Where is there potential for practical value in this theorem?
- For example, one place there could be potential to validate the theory is to check in experiments that task pairs with lower Wasserstein distance actually support better domain adaptation. However, in the Introduction of the paper it acknowledges that “Even the slightest change in these features may lead to vastly different phenomena”, but doesn’t that suggest that Wasserstein distance may not be a useful metric here for measuring task similarity? Couldn't turbulence limit the usefulness of such a metric?
- Proposition 3.3 says the bound is “strictly looser” than the bound in Theorem 3.1. For clarity, it would be very helpful to combine the bounds into an inequality showing this strictly-looser property. It is not immediately apparent from the statement of the theorems since the inequalities contain different terms.
- As is, the theory doesn’t really hurt the paper, but, for the amount of space dedicated to it, it doesn’t add much. The paper could be substantially improved by either (1) adding interpretation/predictions/validation of the theory that connect it back to the approach in the paper, or (2) removing some of the less useful parts of the theory from the main paper to free up space for more of the interesting analysis of what the model actually learns.
- Also, it is interesting but a bit counter-intuitive that the theory section relies on results in multi-task learning and domain adaptation, instead of theoretical results from the meta-learning literature. As is, since the paper relies on multi-task learning so much, it is missing references to related work in multi-task learning (i.e., related work outside of modeling physical dynamical systems).
- Similarly, it would be helpful to mention why there are no comparisons to multi-task learning or domain adaptation methods in the experiments. Why do they not apply here?
- The three terms in the loss function of the encoder are well-motivated, but it is not clear how important each term is. Ablations on these terms would be very informative for the reader to understand what’s generally required to train an encoder.
- In Section 5 it says “VarSepNet employs separation of variables through different loss terms”. What are these loss terms and how are they different from the ones in the paper?
- In the ablations with no encoder, how do AdaIn and AdaPad work? Don’t they require some z? Where does this come from if not from the encoder?
- U-Net does seem it could be at a qualitative disadvantage compared to DyAd in terms on number of parameters, especially since U-Net c is one of the more competitive baselines. It would be useful to see results for a larger U-Net c, or at least some evidence that the U-Net is not underfitting the training data.


Additional question of interest:

Overall, this is a very important a potentially deep line of research. The most exciting promise of such work is the potential of revealing shared regularities across vastly disparate dynamic systems, that is, across complex physical processes. And it seems the approach in the paper could be particularly well-suited to such research. For example, the authors could train a single encoder+forecaster model across all the datasets in the paper, and analyze relationships in the learned encodings across datasets. Training models across highly diverse domains have been tried in multi-task learning (e.g., "Pretrained Transformers as Universal Computation Engines" arxiv 2021, "The Traveling Observer Model" ICLR 2021, "Modular Universal Reparameterization" NeurIPS 2019, "One Model to Learn Them All" arxiv 2017). Is such a generalization part of the longer term vision for this line of work?


Minor comments:
- In Section 2.4, some references would be useful in the sentence ending with “…the combined force equation.”
- There are several inconsistencies in the use of parentheses in citations throughout the paper. Correcting these would improve readability.
- In last sentence of first paragraph of Section 4, the word “task” could be changed to something like “problem”, since “task” has another meaning in the paper.
- Should the 7.26 for U-Net-c on Ocean Currents future be bolded?
- In the last paragraph of Section 5.1: “We tried to vary…” -> “We tried varying…” or “We varied…”.
- Appendix A.2.1: footnote for PhiFlow is on the wrong page.
- Appendix A.2.1: The last paragraph seems like it should be the first paragraph of A.2.2.
- In proof of Proposition B.5, there is an extra or missing set of norm bars in the first inequality.
summary_of_the_review: Overall, this is very interesting and useful work. The problem is well-motivated, and the approach and experiments are carefully designed and generally convincing. If the concerns about the theory are addressed, I would be happy to increase my score. Adding the additional info and experiments requested could increase it further, and make this a particularly strong paper.
review_title: null

Meta-review: The paper addresses the problem of domain generalization for learning spatio-temporal dynamics. It proposes a solution where an encoder captures some characteristics of a given environment, and a forecaster autoregressively predicts future dynamics conditioned on the characteristics learned by the encoder. Said otherwise, the forecaster learns the general form of dynamics parameterized by an environment representation extracted by the encoder. The conditioning is implemented via an adaptive instance normalization mechanism. A form of padding is also introduced in order to take into account boundary conditions. The two components encoder and forecaster are trained sequentially. This approach is casted in a meta-learning framework. Theoretical results inspired by multi-task learning and domain adaptation are also demonstrated. The model is evaluated and compared to different baselines on three problems, and for two different settings: varying initial conditions with a given dynamics, and dynamics with varying parameters.

This is a borderline paper. It targets a timely and important problem of domain generalization for dynamic environments. The proposed solution is original and compares well experimentally to several baselines. It allows for better generalization performance for the two test settings considered. In the current version, the paper however suffers from different weaknesses. First there is the imprecision of the arguments and the description of the experiments. Some of the arguments and claims are vague and sometimes abusive, not backed up by evidence. For example, a central claim is that the encoder learns time invariant quantities characterizing the environment when the learned representations indeed change with a time shift in the input for any environment. The same goes for the argument developed for the padding construction. It is claimed to model boundary conditions, but this is not supported by any theoretical or empirical evidence.
As noted by the reviewers, the theoretical analysis is disconnected from the algorithmic and experimental developments and does not bring much additional value to the paper. What is more embarrassing is that some of the claims in this section are overstated and induce incorrect conclusions.  From Theorem 3.1 and proposition 3.3, the authors suggest that multitask learning leads to better generalization than learning independently, while this is not formally guaranteed by the results (this is acknowledged by the authors in a later comment). Besides, the conditions of validity are not discussed while they seem to only cover situations for which the train and the test distributions are the same. The same holds for the second theoretical results (theorem 3.4). It is claimed that this result supports the authors’ idea of training encoder and forecaster sequentially, while it does not. Besides, the bounds in this result cannot be controlled as noted by the reviewers and are not useful in practice.

Overall, the paper addresses an important topic and proposes new solutions. The results are promising and it is indeed an interesting contribution. However, inaccuracies and incorrect or exaggerated claims make it difficult to accept the current version of the article. The article would make a strong and innovative contribution if it were written as a purely experimental article with a detailed description of the experiments and comparisons.
Decision: Reject