Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to regard the system identification problem as a reinforcement learning problem. Compared with traditional supervised solutions, the proposed reinforcement method has advantages over delayed effects, high non-linearity, non-stationarity, partial observability, and error accumulation when using bootstrapped predictions.
strength_and_weaknesses: Strengths:
1. The research problem is important and the recursive reformulation of the forward model as the summation of the previous state and a derivate approximation is novel. System identification is one of the most important topics in the modern automatic control area. Vanilla supervised methods are intuitive and straightforward, yet suffer from several challenges, e.g., rollout testing and loss accumulation. The proposed forward model has advantages in solving the aforementioned challenges.
2. The justification of the motivation to learn forward models with RL and the design of function approximation is reasonable.
3. The experiments are supportive.

Weaknesses:
1. The major difference is to reformulate the system identification problem into a recursive relation, and take accumulation error into account. In my opinion, the proposed method still uses a supervised learning strategy. Reinforcement learning here only plays the role of an optimization tool, which optimizes the supervised loss.
2. The paper is not well-polished. Subfigures in Figure 3 is massive and some figures overlap with each other.
clarity,_quality,_novelty_and_reproducibility: The logic is clear, but the presentation is not well-polished. The ideas are novel. There seems no big issue with reproducibility.
summary_of_the_review: This paper proposed a wrapper framework to solve system identification problems as reinforcement learning tasks. The idea of recursive relation between states is beneficial to dealing with accumulative errors in rollouts. The logic is clear, but the presentation needs polish.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors propose to learn forward models using RL methods on a new MDP who's states correspond to state-action tuples and who's actions correspond to additive changes in state. From this idea, they propose to use soft actor-critic to learn a gaussian transition model using negative prediction errors as reward with an additional reward injected at the end of rollouts related to the some negative error defined on the whole trajectory. This approach is then applied to 3 MuJoCo domains and compared to a supervised learning baseline.
strength_and_weaknesses: The basic idea is original and could be interesting. However, there are are several limitations in it's current iteration.

It's not quite clear what inherent advantage there is to reformulate the model-learning problem as an RL problem. What property of this newly defined RL problem makes it better behaved than an equivalently expressive supervised learning approach? The authors seem to imply that one-step prediction errors are all that supervised learning could on this task, but there are a plethora of ways to formulate this learning problem. I don't see what would be the fundamental difference between "rollout learning" and a supervised learning approach which optimizes error defined on entire trajectories. The authors should take more time expanding their arguments and accompany them with explicitly written out equations.

One major limitation of the proposed method is that it doesn't address the inherent difficulties of learning stochastic transitions and instead limits itself to additive gaussian transitions. Could the main idea not be applied to discrete MDPs? Exploring the subject in illustrative discrete MDPs would help provide insight while also showing that the method depend on having limited stochastic transition models.

Similarly, the assumption that states support addition or subtraction greatly limits it's applicability to domains that have discrete or hybrid state-space.

Minor comments
=============

I could not find the tips cited as Fleming 2018.

clarity,_quality,_novelty_and_reproducibility: The paper is hard to follow and the author's meaning often unclear. It seems like many terms are used very loosely vaguely or even incorrectly (e.g., bootstrapping). At times, it feels like the authors have something specific in mind but use general terms to reference them, making it difficult to understand the argument being discussed (e.g., "bootstrapped rollouts" in the context of learning forward models). Overall, the authors would benefit from making sure to rigorously define terms before using them when motivating and explaining the work.

As for novelty, I'm not aware of any work proposing to learn models by formulating an MDP in this way. Some concepts of duality of automata theory [1], MDPs and POMDPs [2] might be of interest to the authors since the consider transformations with a similar flavor. Although not immediately relevant to this work as it is, it might inspire future work on this idea of formulating model-learning as a different RL problem.

The authors should consider looking into [3] and citing what is relevant since it covers some impactful work on learning models, by say, solving fixed-points [4]. 

[1] Brzozowski JA. Canonical regular expressions and minimal state graphs for definite events. InProc. Symposium of Mathematical Theory of Automata 1962 (pp. 529-561).
[2] Hundt C, Panangaden P, Pineau J, Precup D. Representing Systems with Hidden State. InAAAI 2006 Jan 1 (pp. 368-374).
[3] Rust J. Structural estimation of Markov decision processes. Handbook of econometrics. 1994 Jan 1;4:3081-143.
[4] John Rust. Maximum likelihood estimation of discrete control processes. SIAM journal on control and optimization, 26(5):1006–1024, 1988.

Minor comments
============

Eqs. (1-4) are very confusing. The order of operations/relations seems ambiguous. Are those even equations?

Eq. (11), how is $Z_E$ an open choice if it respects equality with equation above it?

p. 5, " A specific Gym environment MF has been developed", what does that mean?

p. 5, showing what the update rules of SAC on $M_F$ in terms of states/actions of $M$ would be helpful.

Eq. (15), what is the motivation for this? What is the derivative taken with respect to?

p. 6, last para, what is the "true dataset"?

p. 7, how is the training data generated?

Figure 4 and 5, what the rollout step value (50 and 500) represent here? Why does the shown trajectory both seem of the same length?


summary_of_the_review: The authors proposes an original idea but the clarity of the paper prevents me from appreciating its possible benefits.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an approach based on using reinforcement learning for learning dynamics models in simulated robotics benchmark. A method based on SAC for learning multi-step models is implemented and compared to traditional supervised learning in terms of long-term accuracy in the predictions.
strength_and_weaknesses: Strength:
- The problem of learning multi-step models is important, especially compared to the amount of work that explicitly addresses it;
- The motivation section, while being a bit overly schematic, explicitly states the goal of the work;
- The approach seem to have an advantage compared to supervised learning, in the limited evaluation it was subjected to.

Weaknesses:
- The relationship with existing work is not clearly presented. Essentially presenting a method for learning a multi-step model, the paper should position itself, and compare experimentally, to other work in this space. A few works that come to my mind are the investigation of Erin Talvitie (e.g., https://arxiv.org/abs/1612.06018) and Kavosh Asadi (e.g., https://arxiv.org/abs/1905.13320) and their collaborators, as well as work more based on the option perspective (e.g., https://arxiv.org/abs/2108.03213). Without a comparison with at least some of these works, it is not easy to interpret the actual contribution of the paper;
- In particular, there is a tight relationship between teacher/student forcing in RNN training and multi-step models. To put things into perspective, using reinforcement learning to train a model of the dynamics resembles using reinforcement learning for training RNNs. In the lack of a direct comparison or a justification for this, one might wonder if this is an unnecessary overhead over more traditional methods for learning multi-step models;
- More broadly, showing that one can obtain more accurate models without showing how useful are they for reinforcement learning can generally be misleading: as it has been shown several times in recent work, an accurate model of the dynamics does not necessarily mean better resulting policies after policy optimization (see workshop on the topic: https://icml.cc/virtual/2022/workshop/13463).
clarity,_quality,_novelty_and_reproducibility: The clarity is generally acceptable, albeit there is an overly aggressive use of bullet points which makes at times hard to follow the discussion.

The quality of the work could be improved. since there is no comparison with existing relevant work on the topic.

I am not aware of any work explicitly using reinforcement learning for learning models of the dynamics.

There seem to be no clear reproducibility issues.
summary_of_the_review: Overall, the approach is a potentially interesting direction for effective multi-step model learning. However, there is a lack of comparison with related algorithms, both in the text and in the experiments, as well as an insufficient investigation of the implication for actual policy optimization. Thus, I for now recommend rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a reinforcement learning (RL) based formulation for learning forward models, which is often referred as learning the dynamics of an environment in a control setting. The authors first motivate the desire for computing forward models and then highlight some of the issues with current approaches to forward model computations that heavily on supervised learning setups. The authors the describe some of the potential advantages of computing forward models with RL and introduce a formulation to achieve that. Next, the authors introduce relevant definitions and design choices for their method, such as the choice of the policy (SAC-based Gaussian policy), the rollout loss based on trajectory similarity functions and a motivation for using Actor-Critic methods by leveraging ACs ability to resolve temporal credit assignment.

In their experiment, the authors study three different D4RL environments (Hopper-v2, Walker2d-v2, Halfcheetah-v2) and their specific forward model definition based on calculation of deltas of positions between different states. The first set of experiments outlines how well a RL trained model performs (measured in rmse loss) on three environments. The authors then compare their RL-based formulation to a supervised learning formulation for the same three environment and study the performance of each method. The results show that RL method generally takes longer to train and does not perform as well in RMSE as the supervised learning. The RL method does appear to have a lower error with longer rollout length compared to supervised learning.
strength_and_weaknesses: **Strengths**

* The paper provides a novel formulation to train forward models in RL settings. The authors outline their method, its formulation, and relevant parts in great detail for the reader to follow.

**Weaknesses**

* The experimental data and the authors description of it does not provide a set of clear conclusions about the method. It would be good if the authors could provide greater clarity into the main conclusions from their experiments.
* It's unclear how the authors proposed method would perform (RL calculated forward models) would perform in relevant application settings, such as model-based RL. The authors allude to this point in the conclusion and future work, but the paper would be greatly strengthened if indications of performance would be included.
* Following on the above two points, the significance of the overall results is unclear. It would be good to add further clarity to this, ideally by additional experiments in an application setting or an indication of the advantages this method could provide.

**Additional Questions**

* What is dt in Table 1? I do not see it defined in the text near it.
* Where is the reward in Figure 2? Is it the same as the RMSE, which would be confusing since it should be the signal loss, correct?
* Could you clarify test rollout setting - is the model just predicting further into the future?
* It's unclear what Figure 4 and Figure 5 are trying to show. Could you provide further description?
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The method sections of the paper and descriptions of the experiments are generally well-written. The experiment sections starting at Section 4.1 has some weaknesses related to clarity of the meaning of the results. 

**Quality**

In its current form, it is difficult to determine the quality and significance of the results and potential contributions of the method.

**Novelty**

The novelty of the calculating forward models with RL appears sound, but is limited without showing how those kind of models would perform in applications.

**Reproducibility**

The paper appendix provides detail on experiment setups, but the paper does not include a reproducibility statement or code in the supplementary material.
summary_of_the_review: In its current form, the reasons to reject outweigh the reasons to accept the paper. I think that the paper could be significantly strengthened by providing greater clarity as to the significance of the results and its broader implications.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed to formulate and solve system Identification as an RL problem. The idea is interesting but the paper is premature for publication. In particular, several drawbacks are raised by reviewers, e.g. insufficient discussion of related works, poor writing, and weak experiments. Therefore, I vote for rejection.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject