System Identification as a Reinforcement Learning Problem
Authors: Jose Antonio Martin H.
Oscar Fernández Vicente
Sergio Perez
Anas Belfadil
Cristina Ibanez-Llano
Freddy Perozo
Jose Javier Valle
Javier Arechalde Pelaz
Venue: Submitted to ICLR 2023 (2023)
Keywords: System Identification
Reinforcement Learning
Offline Reinforcement Learning
Forward Models

Abstract: System identification, also known as learning forward models, transfer functions, system dynamics, etc., has a long tradition both in science and engineering in different fields. Particularly, it is a recurring theme in Reinforcement Learning research, where forward models approximate the state transition function of a Markov Decision Process by learning a mapping function from current state and action to the next state. This problem is commonly defined as a Supervised Learning problem in a direct way. This common approach faces several difficulties due to the inherent complexities of the dynamics to learn, for example, delayed effects, high non-linearity, non-stationarity, partial observability and, more important, error accumulation when using bootstrapped predictions (predictions based on past predictions), over large time horizons. Here we explore the use of Reinforcement Learning in this problem. We elaborate on why and how this problem fits naturally and sound as a Reinforcement Learning problem, and present some experimental results that demonstrate RL is a promising technique to solve these kind of problems.

Under review as a conference paper at ICLR 2023
SYSTEM IDENTIFICATION AS A REINFORCEMENT LEARNING PROBLEM
Anonymous authors Paper under double-blind review
ABSTRACT
System identification, also known as learning forward models, has a long tradition both in science and engineering in different fields. Particularly, is a recurring theme in Reinforcement Learning research, where forward models approximate the state transition function of Markov Decision Process by learning a mapping function from current state and action to the next state. This problem is com- monly defined and solved as a Supervised Learning problem. However, several difficulties appear due inherent complexities of the dynamics be be learn, for example, delayed effects, high non- non-linearity, non-stationarity, partial observabil- ity and error accumulation when using compounded predictions (i.e., predictions based on past predictions) over large time horizons. In this paper, we elaborate on why and how this problem fits naturally and sound Reinforcement Learning problem, and present some experimental results that demonstrate RL is promis- ing technique learn forward models.
INTRODUCTION
One the most distinguishing features of Reinforcement Learning TRL.,SSttton & Barto 1998) is the direct trial and error interactions between an agent (learner) and its environment (real world, plant, game, etc.), so that the agent can learn the consequences of its actions and thus adapt its behavior to optimize a goal in the long run. However, in real life applications and specifically in industrial settings, there are many mission critical assets where RL cannot be applied in its canonical form, "Online" RI., due to security or operational risks, i.e., the risk of unsafe exploratory actions or the interruption of asset operations. Among recent lines of research in this field, "batch" and "Offline" RL (Lange et al.) 2012 Fujimoto et al. 2019 Levine et al., 2020) formulations break down many of these barriers leading to its successful application to these problems.
Our goal is to perform offline RL for these critical real life situations (see dreaming or imagination, Ha & Schmidhuber 2018 Hafner et al.) 2020), however, our method is to apply online RL two (data): first, for learning good enough forward models, which is the focus of the present paper, and second, for learning a policy using that forward model as the environment. We seek avoid the well known negative results reported by Fujimoto al. 2019). while leveraging the exploration ability of RL to learn good policies.
times (RL
Learning forward models has been active
of research, with abundant contributions the application of Machine Learning (ML) (see for instance Werbos) 1989 Fu & Li 2013 Zhang 2014
Abdufattokhov & Muhiddinov] 2019 Roehrl et al. 2020). Particularly, is recurring topic of
research within RL (Sutton) 1991 1 Sutton & Barto 1998; Polydoros & Nalpantidis) 2017 Moerland
al. 2020). where forward models usually represent the transition function S1+1 (st. ae) of some Markov Decision Process (MDP). We denote an MDP as a tuple M = (S,ATT, T, where the state space, A is the action space, T the transition function and the reward function, thus, St+1 = T(s,, ad)) represents the immediate state after the evolution of the system, starting at time with state 8t and conditioned by an action at, and T is defined by mapping function s++1 = (st, Learning a forward model is a task commonly defined as Supervised Learning problem in a direct way (Jordan & Rumelhart 1992 Moerland et al. 2020). having the set of observations X ... labels y { ..., and loss, e.g., L [[f a.) still, however, this approach faces several challenges. Here, we propose that this problem has a more complete and natural definition RL problem, and show, experimentally, positive results.

---

Under review as a conference paper at ICLR 2023
(3. a)
agent
agent
As t f()
0++1 c [8)
C
As
environment
environment S1+1 t T(ss, a)
5c+1 = + As
a++1 + D
Figure The typical RL setting (left). RL flow for learning a forward model (right).
2 MOTIVATION AND PROBLEM DEFINITION
Why learning forward models with RL? The domains, tasks, and problems to which forward modeling being applied are of increasing complexity, including time delayed dynamical effects, high degree of non-linearity, partial observability (POMDPs) and the compounding enror issue: compounded rollout (i.e., generating samples based on past generated samples itcratively) induces compounding error "when imperfect model is used to generate sample rollouts, its errors tend to compound a flawed sample is given as input to the model, which causes more errors, and so on" (Talvitie) 2014). This situation raised the need for additional techniques vised Learning framework to deal with this increasing complexity, for instance: rollout testing for beyond single step learning robustness, loss accumulation over rollouts for large horizon prediction, recurrent networks or stame-stacking or neural Turing Machines for partial observability, curriculum learning over increasing horizons, data augmentation to aid learning symmetries in data, ensembles of stochastic neural networks to increase the prediction accuracy and reduce bias, and set of works focused on studying the compounding error (c.f. Oh et al. 2015 Talvitie 2017 Silver et al. 2017
adapt the Super-
Fleming 2018 Asadi et al. 2018 2019 Xiao al. 2019 Lambert et al. 2021 2022).
On the other hand, RL has intrinsic features that provides a natural way to deal with many those complexities: rollout learning by working with episodic tasks, minimization of the compounding error by optimizing for the long-run by solving Bellman's 's equation, continuous learning from new experience without requiring a full retraining, and it is by design well suited for stochastic scenarios and works in situations of partial observability.
Does RL come with an unreasonable extra search cost? Solving regression problem by RL has extra cost due the required exploration, i.e., searching for point ys y (label) that is indeed already known. Thus, why solving this problem with RL? Is it an unreasonable cost? May have
problems instead of one?
Let's 's suppose that we run compounded rollouts of certain length h (time horizon), while minimizing the errors between each predicted point 2; f(s,, a.) and its corresponding true target (label) E y. Since the predictions in the compounded rollouts trajectories are sequentially dependent (by composition). then face temporal credit assignment problem, q.e., the root cause of the compounding eттоr. The RL framework deals with this problem naturally through the Bellman's optimality criteria (Bellman 1957). Thus, in this problem setting, we shall not assume that we are given "true targets". Moreover, since we are relying composition, then, in the Supervised Learning setting, we need the corresponding observation points x, for the predicted labels ýe = (*,,, however, these are not in the dataset, since I2 are predictions as well. Finally, this extra search cost allows to learn a Q-function that learns explicitly the expected prediction error that the model will commit.
Forward model learning as an RL problem: Learning forward models with RL can be directly achieved by just translating a regression problem to an RL problem, where the observation (state) is formed by the current state of the system and the previous observed action, the actions of the agent represent the predictions of the next state, and the reward signal is just the negated total/cumulative prediction error. More formally, given an MDP, M A, R), the forward learning problem defined as the MDP My = (Sf: Ar, (DIO)=, F)), where D O refers to a time series of tran- sitions stored in dataset (D) or observed (O) from real world process, Lx ||8141 8 is
2

---

Under review as a conference paper at ICLR 2023
loss function of the true observed next state (st+1 D)O) and the predicted next state ($++1), Sy denotes the state space formed by tuples of (s, a) E S x A. and Ay is the action space de- S. Figure shows the diagrams for the MDPs M
fined by elements As $1+1 8t, where and M,. respectively. Hence, to apply RL to the forward model learning problem, we just need a "transposition" of the problem and its observations:
DIO (5t, Te, 8t41),, common format of offline RL datasets
(DIO)* (e)), As,r 0, (82+1, (c-11)
where the final objective is to obtain, by RL, policy π on the approximated dynamics 7 = MMs < MMr such that " A is as close as possible to the optimal policy лM in the original MDP,
as shown in Equations 1 aadd]
TM M (8, (8,,,T
(1)
TM n JMll x IT -1
(2)
3 METHOD AND DERIVATION
Starting from the one-step definition a forward model (Eq.
$441 f(s, az).
(3)
where s, is the state of the system at time step and a, being the control actions applied to the system at 1, we obtain a one-step (forward) prediction of the next state $,+1 of the system. Learning such forward model f(s,a) a) implies learning the approximator function f TT with any statistical or ML method (e.g.. by fitting the parameters 0 of a parametric estimator fo). For this, a loss function defined as L (Eq.
L = =|s44111 - $141ll-
(4)
where L is minimized during training, for instance, using gradient descent techniques by computing Thus, the gradient VoL. Thus, the problem setting naturally appears as a Supervised Learning problem
(regression) 0N = (xx;), where the collection of inputs X is formed by tuples æ; = (St, a.) and the targets collection (labels) y are commonly defined by the next observed states y/1 8t+1- With this definition, a Supervised Learning task is completely defined and can be then easily achieved through a "vanilla" ML pipeline. Now, let the predictions of our model be the deltas (A5) w.r.t. the last state (s), instead of predicting the new state, thus, Eq. 3 becomes of the following form (Eq. 5):
814 = 8, + f(r, oz), and hence, AS (s,
(5)
(6)
which describes a well-known recursive relation in the field of Dynamical Systems, by expressing the evolution the system in terms of its past state plus its derivative (residual).
Stochastic policies: Focusing on the most recent works on model-based RL (Janner et al. 2019 Yu et al. 2020), stochastic Gaussian networks are used as the function approximator to learn the policy ~ such that:
(s. = As ~ N (Ha,a) P.a))
This allows to approximate the dynamics through stochastic model. There is, perhaps, good margin for improvements over Gaussian networks as pointed out by by/Chou et al. (2017) i.e., beta distribution), however, the original SAC algorithm, which we use for our experiments, uses squashed Gaussian Normal Policy for the policy network (see, HHarnoja et 2018).
Compounded rollouts and large horizon loss: An effective forward model should predict not only the next state, based on the true past state and true action accurately, but should allow as well to simulate the system by rumning rollouts over its own predictions, implying thus composition process, that is, predicting the next state based on a previous predicted state (or even a history of predicted states). This condition appears the most difficult part from the learning perspective, and
3

---

Under review as a conference paper at ICLR 2023
it is the cause of the compounding error well. However, think that it should be considered as a requirement for a sound and robust definition of forward model learning problem. Also, some benefits can be obtained which are inherent to compounded rollout training process, for instance, it creates, trough exploration, an implicit data-augmentation (van Dyk & Meng. [001]] Garran & König 2018 Iwana & Uchida 2021) over the source data (D[O), which is common practice to improve ceneralization and reduce over-fitting, but requires an extra work in the supervised ML setting. This implicit data augmentation, instead of enlarging training dataset from existing data using various translations, acts as a kind of "spatio-temporc" transformation, as in the case of auto-encoders, generating new training data "on-the-fly" (Tu al.] 2018). contributing to obtain robust policies to noisy inputs, aid in learning problem symmetries and generalization, and thus helps to avoid overfitting to the fixed dataset or observations.
Thus, by applying Eq. 8 for running compounded rollout E(st, (a))+h, .h) of length h simulation steps, over fixed sequence of actions (a)! = (0t....(t+bb) and from an initial state 8,, we obtain a compounded prediction & & of length h and a predicted trajectory y = (St, St+1, 8tth).
8ith St +
(Beti, acts)
(8)
8n=
Also, equations 8 define well known recurrent relations that can be seen as the equivalent of recurrent connections, computed with a loop over a sequence input batch where the loss is calculated the end, as well as dynamical system modeling through Neural ODEs (ODENet, Chen al. (0188), its augmented version (Teh et al. (219)) and ResNets (He et al. 2016).
Now, a rollout loss (s(st, (a)t .,, 9. h) can then be defined as follows:
h Cx(SE, (a)) U. h)- = Σ
18845 - Setell, Se44 E 8844 E E or
(9)
1m1
Cg(# (a) y, 1. h) = Z (v)
(10)
as more general definition, being Zz(y, 9) a signal (trajectory) similarity function. Thus, in rollout learning, the network is trained with the rollout loss (F4. [T]. instead of the common supervised loss (c.f. Eq. 4). The function (g D) is an open choice, however, it changes the problem to be solved, as it is the objective function and the goal of the RL problem. In particular, here the problem is sequential and thus the objective is to evaluate how "close" (in terms of shape/pattern recognition) is the predicted trajectory vs. the observed trajectory (ú y). We must note that (r)mse for this problem is just a weak approximation, and that doing signal/pattern recognition using (r)mse alone measure signal/shape similarity is definitely Schuller (201)) Also, there are many measures for signal similarity (e.g.. KL--divergence, statistical (invariant) moments, signal correlations, etc. ), and usually a combination works better depending on the type of the signals and which signal's features are more important for the problem.
good general measure, see for instance Pandit &
Policy learning with the Actor-Critic architecture Finally, natural way to implement rollout learning is episodic learning, where sequences of transitions are divided in episodes, and rollouts are run over such episodes using loss accumulation for computing the gradient. This approach naturally conducts to thinking on the temporal credit assignment problem (see, Minsky 1961 Sutton 1984 Sutton& & Barto| 1998). which can be solved effectively through TD-Learning (Sutton 1988). That is, to solve a sequential decision problem optimizing the cumulative sum of scalar signal over an episodic task. For this, it looks natural to use the Actor-Critic method (Sutton & Barto 1998 Degris et al.) (012) to train policy network and, in particular, stochastic to predict the deltas (residuals) of the next system state. This can be achieved trough methods like the SAC algorithm (Haarnoja al. 2018
EXPERIMENTAL EVALUATION
We tested the proposed approach three different MuloCo environments from OpenAl Gym con- trol suite: Hopper-v2, Walker2d-v2, and Halfcheetah-v2. D4RL (Fu et al., 2020) datasets of the MuloCo environments have been used to train the forward models. A specific Gym environment
4

---

Under review as a conference paper at ICLR 2023
has been developed for training over the trajectories (time series of episodes) contained in the D4RL datasets. The designed Gym environment resets the state and stacked states at each rollout-length step, that is, if we train with rollout length of h 50 then every 50 steps the state and the whole stack is reinitialized and filled with the ground truth data of the episode contained in the dataset. All MuloCo environments used here rely continuous state and action spaces. Further details on the data preparation can be found in Appendix [.4] In MuJoCo, the state variables are divided in two vectors: positions (qpos) and velocities (gve(). Since positions can be derived from velocities and vice-verss then we opted to predict only the deltas of the positions Aqpos, and then infer the
original velocities of the simulation. The predicted qpos and its corresponding quel are calculated according to:
gpos qpos; + Aqpos
(11)
queh+1 Aqpos dt
(12)
The observations (inputs) for the models are windows (stacks) of states of length w 20 (pasttcurrent), that is, the well-known frame-stacking method. The observations contains well the respective actions taken at each time step as stored in the episodes of the D4RL datasets. Also, for adding more variability the training trajectories data, each episode starts randomly from time step in the range 30), so that the model will not see the same sequences all the time but random shifts of the episodes.
We use the Soft Actor-Critic (SAC) algorithm (Haarnoja et al. 2018) as the RL algorithm, thus the actor networks policies represent the learned models. For all the experiments, the same set of hy- perparameters for the Actor and Critic are used: encoder architecture is an MLP of 6 hidden layers of 512 elements each with Mish (Misra) 2019) as activation function. Quantile regression (Dab- ney et al.) (017)) is used for the Q-function networks (with n-quantiles
64). For each network, and learning step, the batch-size used is 1024. Networks inputs are min-max scaled based on the datasets. Other information about specific training parameters like the number of episodes run per experiment, or gradient steps can be found at Table Refer to Appendix A.2 for additional details about hardware and software used.
Table Experiment's 's training parameters and main test metrics results.
Environment
Training Parameters
Testing
last values (average)
Episodes
#Params
Grad. Steps
Cr. loss
Act. loss
rmse
Hopper-v2
12K
1425K
121K
0.49
2.05
0.09
Walker2d-v2
12K
1488K
122K
5.76
10.63
0.19
Halfcheetah-v2
12K
1488K
121K
69.06
33.00
0.28
For these experiments, we used pseudo-sparse reward signal as follows:
as) 1 (y (signal similarity function) at rollout ends, otherwise,
(13)
||st SellL,
being the signal similarity function (y-()0)
ZE(u. 0) = (1 11 (يااا) (1 + IVy Villcorr) (1 y1 1 Ú(KL),
(14)
where, |1y Úا the sum of the squared eтror between true and the predicted values, I|Vy Villoory is the correlation distance between the derivatives of both predicted and true values and bly all K is the Kullback-Leible divergence between both true and predicted values, over all the the current rollout steps.
For Hopper, Figure 2 shows the overall RMSE of predicted variables for the forward models. After running total of 12K episodes and 121k gradient steps, the overall RMSE converged close to 0. Additionally. both actor and critic loss curves show that the SAC agent learning good pol- icy from episodes 1000 to 4000 (see additional experimental results). After these episodes, both curves converge asymptotically, suggesting a pseudo optimal policy has been achieved. Walker2d has similar number of gradient steps and episodes Hopper, it is noticeable how emror higher
5

---

Under review as a conference paper at ICLR 2023
Hopper v2
Walker2d v2
Halfcheetah, v2
Remde loss
0:35
losseloss
Rusia loos
0.8
0.130
0.25
0.4
0.15
Episode
Episode
Episade
2.2
msse loss
0.4
0.8
0.3
0.4
rollout length
rollout length
rollout length
Figure 2: Hopper, Walker2D and HalfCheetah: RSME per episode and rollout test performance
in terms of RMSE (Fig. 2 as well as actor and critic losses (see Appendix 2 than Hopper. This is basically due the higher number of predicted variables (almost double), and the higher complex- ity of the environment. We can also notice an error increase when the model is tested with higher rollout-steps. However, the asymptotic behavior of these curves is still maintained, suggesting the finding of a stable sub-optimal policy. Finally, HalfCheetah is the hardest environment tested. De- spite having the same number of target variables than Walker2d, these are noisier as the dynamics of the robot is more complex and the results seems worse than in the other two.
4.1 COMPARISON OF REINFORCEMENT LEARNING VS. "VANILLA" SUPERVISED LEARNING
To compare both Supervised Learning and Reinforcement Learning approaches selected the Hopper environment and designed a specific setup.
Reinforcement Learning setup: Two SAC based RL. forward models have been trained (RL-v1 and RL-v2) with different stack windows. RL-v1 uses stack of w 10, while RL-v2 uses = 30. The reward function used for this experiments is fully-sparse reward, that is, r = for every time step, and r signal similarity measure (between both true and predicted trajectories) every rollout end. This has been designed in this way to test the RL approach in it's extreme version, despite this sparse reward function will hurt convergence speed. The signal similarity measure (RL goal) used in this experiments is a simplified but effective similarity measure. It incorporates explicit measures from time-domain, frequency-done domain and power-domain measures (c.f. Ivan Brian 2022). The rollout length for training was h = 50 episode steps.
Supervised Learning setup: Two Supervised Learning models (SL-vl and SL-v2) have been trained and tested with different stack windows as well. SL-v1 has a stack of w 10, while SL-v2 uses w = 30, same as RL setup. The Supervised Learning experiment follows the same structure of steps/rounds of the RL training setup. particular, we use the same SAC Actor Networks for the Supervised Learning policy, however, samples are taken from the ground truth DRL4 datasets instead of a replay buffer as RL does. Each sample is of batch - size = 1024 and, at every step/round, 100 mini-batches are randomly sampled to train the network using the MSE Loss.
Thus, both approaches use the same Policy Networks, but trained using two different methods. Special attention have to be paid the fact that these two approaches solve different problems, since they have different optimization objectives: the RL training uses a signal similarity function over entire rollouts as its goal, while the SL. training optimizes the MSE over random mini-batches.
Experimental results: A total of 25.000 steps/rounds have been performed per experiment. Fig- ure 3 shows four different plots: Critic Loss (upper-left) of the RL approach per evaluation-round Critic Loss for SL is since no critic is trained for the SI. experiments. Supervised loss MSE

---

Under review as a conference paper at ICLR 2023
(upper-right) per evaluation-roumt for all policies (RL+SL) since it is possible evaluate RL- actor policy network over the ground truth DRL4 datasets in the same way as the SL policy. RMSE rollout-metrie (lower-left) which is the RMSE evaluated the current evaluation-round over all the steps of a random episode é, from the dataset, comparing the predicted trajectory of the entire episode vs. the trajectory of the entire episode. Mean rollout reward (lower-right) shows the mean of the rewards obtained by the evaluation of the signal similarity measure at each rollout end of the random episode.
Critic-loss (TD-error + Entropy)
One-step SL-loss (MSE) over dataset
Experiment:
S1.v1
0.003
1500
SI-22
RI-11
0.002
1000
RI-v2
500
0.001
0.000
Rmse rollout metric
Mean rollout reward (similarity measure)
12.5
200
10.0
7.5
400
5.0
-800
0
5000
10000
15000
20000
25000
5000
10000
15000
20000 25000
Evaluation-round
Evaluation-round
Figure 3: Supervised Learning vs. Reinforcement Learning training and evaluation metrics.
Some conclusions can be elaborated from observing the results in Figure 3 SL trained policies converge very fast to very low values of the Supervised loss MSE and low values of the RMSE rollout-metric (SL-vl and SL-v2 lines almost converge in the firsts training rounds), however, it is the same for the mean rollout reward (signal similarity measure) and the SL-v2 version (w = 30) achieves better optimality than SL-v1 (w 10). The RL trained policies convergence speed is no- tably slower (orders of magnitude), however both versions RL policies achieves better optimality than SL. in RMSE rollout-metric and Mean rollout reward. Although RL-v2 policy of (w 30) shows a better optimality in the RMSE rollout-metris than the RL-vI (w 20), not clear that this is true for mean rollout reward metric. It is somehow natural that the trained SL policies does not perform well for the mean rollout reward metric, since they are not trained with this learning objective, however, both RL policies are also converging low enough values of the Supervised loss MSE metric, even when they are not trained with this objective.
Finally, from the whole figure, it can be observed that beyond some "rounds" RL starts to gain advantage over the vanilla SL approach, not only achieving better values in the RMSE rollout-metric and mean rollout reward metrics, but also with significantly less standard deviations, as shown in the shaded regions enclosing all curves.
Also, with the corresponding trained policies, we tested each one by predicting trajectories on random episodes for different rollout lengths:
Copertment R,12
h {50, 100, 200, 300, 1000}.
Rmse
By observing the results in Figure ticeable how the SL policies (vl and v2) per- form better than RI. on shorter rollouts (h S 300), however, as we increase the length of the rollouts, the SL trained policies suffer an expo- nentially increasing compounding error. Nev- ertheless, the RL approach seems to be more robust against rollout lengths, suggesting, per- haps, that RL polices may be able to better re- sist strong exploration and planning over them without diverging.
no-
10-1
10-2
400 700 800
Rollout Length
Figure 4: Increasing rollout size testing for the learned policies (log scale).

---

Under review as a conference paper at ICLR 2023
(SL)
Testing hopper sode: 13074, rollout longth: 500, start-stop: m.se: 1. 1.9320
pootx_pos,ae rmss: 0.6610, nomsiiidd-mis: O. 0.0327
thigh joint angle, mise: 2.9959, normalized-mms: 2.654]
true
predktted
10
rootz_pos, rmse: 1,7308, normalized rmse: 2 2481
leg joins angle, mes: 1,590,, rormalized-mae:: 2. 1889
Compounding erer broakss simlation
mm
1.5
-05
1.0
rosty, angle, mese: 1.697, normalized-mmse 6.6201
foot_ Loo_jjoin___nee angle, mor:: 2.1490, normalized rose: 1.1972
0.0
-05
02
(RL)
Testing hopper sode: 13074, rallout length: 500, start-step: O, mmse: 0.1163
roobx
2224, normaizzed 0,0110
thigh
0.0747. normalized 0.062
truw
10
1o0tz_oo% normalized-
leg. angle 0.0550, nomalized
1.75
0.00 pu
1.30
0025
rocty 0.0342, nomallzaeddmmse 1348
Joint angle, 0.1451, nomallized mes. 0808
0.0
0.5
-02
-0.
-02
400
100
Figure 5: Above: Random episode of rollout length 500 steps for the Supervised Learning policy. Below: Random episode of rollout length 500 steps for the Reinforcement Learning policy.
Finally, F random episode comparing the true vs. predicted trajectories using rollouts of h 500 steps, for both SI-v2 and RI-v2 policies. It can be seen how the compounding error increases for the SL. policy up to the point that it completely becomes useless. On the contrary, the RL policy manages to control the compounding error and keep predictions in range. More importantly, in the RL policy now we can't 't talk about a systematic compounding error, since it is possible for the RL policy have error E in some steps and have E in subsequent steps.
4.2 INITIAL RESULTS ON TRAINING RL AGENTS ON FORWARD MODELS
To investigate the behavior of the learned forward models (SL, RL) as training environments, a stan- dard RL training procedure using Gym performed. These experiments were run using the well known tianshou (Weng et al., 2022) library examples. The mujoco Hopper-2 SAC algorithm exam- ple (examples/mujjcc/nmjjoo sac.py) was used with additional parameter specify different environment for testing the agent, in a way that the evaluator performs the tests on the real Gym Hopper-v2 environment instead of the forward model environment.
From the results of testing the SL-v2 forward model (Figure 5) we have seen that it is clearly affected by the compounding error, and also, by observing Figure training process producing very short episodes and small training rewards. A hypothesis to explain this is that the SL-v2 model protects itself from data it has not seen in its training dataset, cutting the episodes earlier due to an exploding compounding error (Hopper-v2 explicitly cut episodes when variables are out of band), and thus admitting only inputs that are very close to the data in its training dataset, hence, data close to the real environment. However, for environments that do not penalize out-of-band variables this can be a strong problem. Based on the previous hypothesis, the forward model RL-v2, observed in Figure 5 does not suffer severe compounding error, and thus, it should be able to admit more diverse inputs and longer episodes without an exploding compounding error, as it effectively occurs in the experiments (Figure 6). However, this robustness to "bad" inputs may come with the possible drawback not cutting episodes which are clearly artificial and impossible exist in the real environment, affecting severely the learning process. This is known
how this affects the
8

---

Under review as a conference paper at ICLR 2023
Training Forward models
Testing real Hop
800
RLVZ: 0.1c
SLV2:
reward
600
400
400
200
200
1000
800
Episode (cb
600
800
600
400
Aml
400
M
200
200
Forward model env
1e6
Hopper-v2
1e6
Figure 6: Results of training an SAC agent in an environment whose dynamics are based on the forward models SL-v2 and RL-v2. At each epoch, test runs are performed over the "real" gym Hopper-v2 environment.
common concern: an agent trained over a learned forward model may exploit weaknesses of the model gain unreal advantages in optimizing the reward function (the cheating effect).
Hence for the SAC agent trained on the RL-v2 forward model, we opted for penalizing the reward (r) with model uncertainty measure. In particular, the reward is penalized by 0.le, where e e= Of(((, (x, a), As) is the expected error of the model prediction. This penalization an attempt to aid the learning agent to focus on optimizing transitions for which the forward model has low error, and thus, are closer to the real environment. This idea has been applied as shown also in Figure
5 CONCLUSION
This paper presents framework to learn forward models with Reinforcement Learning. We have shown significant reasons on the benefits to consider forward-mode learning as an RL problem. We have defined and described how to model this problem with RL, and we have tested the pro- posed method over three well-known environments from the MuJoCo Gym collection, compared it against commonly-use Supervised Learning setup, showing significant results in reducing the compounding error for large horizon simulations. Finally, we have shown the results of training RL agents these RL-based forward models with promising results well. We think that the main possible reasons for these positive results can be that: 1. RL controls the compounding error by optimizing for the long-run solving Bellman's equation; 2. The exploration of RL during training on compounded rollouts enriches the robustness of the policy to noise and diverse inputs, and thus, it reduces the compounding error as well; 3. Using similarity function whole rollout trajectories is a preferred objective over the widely adopted one-step l. norms as a loss function.
The guiding objective the use of these forward models to train RL control agents leverage ex- ploration in offline RL, with the aim to obtain results which are more comparable against agents trained "real" environments. Also, we are doing research on how to exploit the Q-function learn- ing by RL during the policy training. This Q-function learns explicitly to estimate the expected cumulative error the model will commit in its predictions, an information which not commonly present in many of the current available methods. That information gives explicitly learnt uncer- tainty measure on the model's predictions, and thus can be very helpful when exploited by model predictive control methods or Reinforcement learning algorithms, e.g., by weighting rewards any other exploratory or penalizing scheme. We presented as well initial experimental results on this line with promising results.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shokhjakhon Abdufattokhov and Behzod Muhiddinov. Stochastic approach for system identification using machine learning. In 20/9 Dynamics of Systems, Mechanisms and Machines (Dynamics), pp. 1-4, 2019. doi: 10.1 109/Dynamics471 13. 55,2019.8944455
Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity model-based rein- forcement learning. In International Conference on Machine Learning, pp. 264-273 PMLR, 2018.
Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L. Littman. Combating the compounding-error problem with multi-step model, 2019. URL https://arxiv.org/ abs/1905. 13320
R.E. Bellman. Dynamic Programming. Dover Books on Computer Science Series, Dover Pub- lications, 1957. ISBN 9780486428093, URL https://bookss wwwggle.es/boosss..........///..
EyVEp3EMxasC
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Po-Wei Chou, Daniel Maturana, and Sehastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learn- ing, volume of Proceedings of Machine Learning Research, pp. 834-843. PMLR, 06-11 Aug 2017. URL https: https::/proceedl.nneg htt.://eeed..noa.......eeee/e//a https///woooooooaaaaoaatocm
mlr
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression, 2017. URL,
hbtgg//bs/111 10044
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. In International Con- ference on Machine Learning, 2012.
Jim (Fomoro) Fleming. Forward models tutorial, 2018. URL https://gitthit com/
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Li Fu and Pengfei Li. The research survey of system identification method. 2013 5th International Conference on Intelligent Stuman-Machine Systems and Cybernetics, volume 2, pp. 397-401, 2013. doi: 10. 109/THMMS.22113.222..
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
David Ha and Jürgen Schmidhuber.
Recurrent world models facilitate policy evo- lution. In Advances in Neural Information Processing Systems 31, pp. 2451- cc cctpaper/
2463. Curran Associates, Inc., 2018. URL. https:://papess. ni wttps//sssippscccppppeeeeep
751 2- www.rrrcuueee-tee -o -- fa-aaailtttteepllccyoooooo wor 1 www.ddddde..giiiiee
on https://
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul / http::///pooeeeeig.s..me wan.r.presse www.speesssvvv... /tt.://pee///////aaaoooaaaaaaaa
2018. URL https:::.....
html
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In 8th International Conference Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https::/ net wtt./oormmiddsi 10TCEED
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Alex Hernández-Garee and Peter König. Data augmentation instead of explicit reguuarizzaiiiinnnnon arXiv preprint arXiv:1806.03300 03852, 2018
Ivan and Brian. In "similarity between two signals: looking for simple measure". stackoverflow.com, ow. com/ a/ 6821 9235
2022. URL htt.tttss//ssacceeee
Brian Kenji Iwana and Seiichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. PL08 ONE, 16(7):1-32, 07 2021. doi: 10.1371/jonl 1 0254841. URL https: //doi. org. / 1371 journal pone. 0254841
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model- based policy optimization. In Proceedings of the 33rd International Conference on Neural Infor- mation Processing Systems, volume 32, Red Hook, NY, USA, 2019. Curran Associates Inc.
Michael Jordan and David E Rumelhart. Forward models: Supervised learning with a distal teacher. Cognitive science, 16(3):307-354, 1992.
Nathan Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, and Roberto Calandra. Learn- ing accurate long-term dynamics for model-based reinforcement learning. 2021 60th IEEE Conference on Decision and Control (CDC), pp. 2880-2887. IEEE, 2021.
Nathan Lambert, Kristofer Pister, and Roberto Calandra. Investigating compounding prediction errors in learned dynamics models. arXiv preprint arXiv:2203.096377 2022.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning, pp. 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 97833-67733- doi: 10. 118-3-642-276455 2. URL http:///00......11.0000007333323
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644; 2020.
Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1)=8-30, 1961.
Diganta Misra. Mish: A self regularized non non-monotonic neural activation function. arXiv preprint arXiv: 2019.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Model- reinorremeet leamn ing: A survey. CoRR, abs/2006.167111 16712, 2020. URL http:s//axx....orrraaa///0000
16712
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances neural information processing systems, 28, 2015.
Vedhas Pandit and Björn Schuller. The Many-to-Many Mapping Between the Concordance Corre- lation Coefficient and the Mean Square Error. arXiv e-prints, art. 1rX.v:1902.051188 February 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bi Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Athanasios Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement learning: Applications robotics. Journal of Intelligent & Robotic Systems, 86(2):153-177 2017.

---

Under review as a conference paper at ICLR 2023
Manuel A. Roehrl, Thomas A. Runkler, Veronika Brandtstetter, Michel Tokic, and Stefan Ober- mayer. Modeling system dynamics with physics-informed neural networks based on lagrangian mechanics. PFACCPPpersOOLLie 93(2):9195-1)) 9200, 2020. ISSN 2405-8963. doi: https://doi.org URL https: www.///w..........oww..../e.///t sc encedirect. www.soocmcs.e.....oo. sc ctom/ccienee// 21st IFAC World Congress.
10.10166/j 2222 12.2182.
atticcle//s11/2240595522222
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. In International Conference on Machine Learning, pp. 3191-3199. PMLR, 2017.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 264):160-163, 1991.
Richard Sutton and Andrew G Barto. Introduction to reinforcement learning. MIT press Cam- bridge, 1998.
Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. University of Mas- sachusetts Amherst, 1984.
Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Erik Talvitie. Model regularization for stable sample rollouts. In UAI, pp. 780-789, 2014.
Erik Talvitie. Self-correcting models for model- based reinforcement learning. In Thirty- AAAI Conference on Artificial Intelligence, 2017.
Y Teh, E Dupont, and A Doucet. Augmented neural odes. Advances in Neural Information Pro- cessing Systems 32 (NIPS 2019), 32((019), 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-50333 IEEE, 2012.
Juanhui Tu, Hong Liu, Fanyang Meng, Mengyuan Liu, and Runwei Ding. Spatial-temporal data augmentation based on Istm autoencoder network for skeleton- based human action recognition. In 2018 25th IEEE International Conference Image Processing (ICIP), pp. 3478-3482, IEEE, 2018.
David A van Dyk and Xiao-Li Meng. The of data augmentation. Journal of Computational and Graphical Statistics, 10(1):1-50, 2001. doi: 10. 1998/ 1011160011112444
Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1-6 2022. URL http:: /etr.//mrrrr///////eee org/ http://apprs//////
21-1127 html
P.J. Werbos. Neural networks for control and system identification. In Proceedings of the 28th IEEE Conference on Decision and Control, pp. 260-265 vol.1, 1989. doi: 10. 109/CDC.1989
Chenjun Xiao, Yifan Wu, Chen Ma, Dale Schuurmans, and Martin Müller. Learning combat compounding-error in model-based reinforcement learning. arXiv preprint 2019.
Hiroyuki Yamada. cpprb, 1 2019. URL https:s::///itle com/ ymd h/ cpprb.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14142. Curran Asso- neuriis..e.r...... wwwprx/pss.cccpppp//////11111
ciates, Inc., 2020. URL 03228822eecaf73224447788888 - pd
12

---

Under review as a conference paper at ICLR 2023
Wenle Zhang. Mimo system identification with extended madaline neural network trained by levenberg-man levenberg-marquarce marquardd method. In The 26th Chinese Control and Decision Conference (2014
CCDC). pp. 1191-1196, 2014. doi: 10.1 10.109/CCCDC2220446655311
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 EXTENDED DISCUSSION AND CONCLUDING REMARKS
Real Life motivation Reinforcement Learning is a very powerful technique for solving Industrial Control Problems. However, in the Industrial setting, there many mission critical assets where RL cannot be applied in its canonical form, for security or operational risks, i.e., the risk of unsafe exploratory actions the interruption of asset operation. Among the last lines of research on this matter, the "Offfine Reinforcement Learning" formulation drops many barriers in the successful application of RL to these problems. Offline RL uses observations from the true industrial asset operation trough a dataset of trajectories (including control actions). This dataset is then used to perform offline RI. to learn a control policy.
However, it is very common, by causes of operational reasons, that not all control actions are well represented in the dataset (out of distribution "OOD" actions), and thus the estimated value of such actions may be largely over/under estimated. The reason is that the returns of such OOD actions can not be learnt (penalized/rewardee) during training, since there no interaction with the true envi- ronment. Hence, OOD actions become untested actions with maximal uncertainty in its estimated values. This fact, induces poor (or completely wrong) policies withing the canonical RL setting, and the performance of the current state of the art Offline RL algorithms is being actively investigated. However, a forward model is provided then RL. algorithms can explore on it, and avoid or mitigate the problem of ODD actions.
Along this line, the natural improvement over the presented results, is being able to understand how good or bad our forward models represent the real environment dynamics (beyond the information contained in source datasets and error metrics). In order to analyse this key point, the goal is to train standard RL agents on these models and compare their performance against real environment trained ones.
Although have done initial steps on this direction, setting up, and controlling the experimental setup of such experiments must be further studied, since our initial tests did not yield the expected results. We argue that once these tests are completed, their results will give us better understanding of the robustness of our approach.
The claim First, we want to remark that the presented comparison between Supervised Learning and Reinforcement Learning not about algorithms, but problem paradigms. We do not deny that Supervised Learning, with enough extra features, can be used to solve the problem as effectively as any Reinforcement Learning approach or even better. However, what we argue is that the pro- cedure to build such specialized Supervised Learning method signals the quest to solve a problem that is more naturally stated as a Reinforcement Learning problem or a general sequential decision problem.
The proposed method There are many other aspects to consider for further optimization of the learning procedure, for instance:
Does this problem impose special constraint to the Q-function? Since we know that a se- quence of perfect actions should return 0 as rollout error, it can be argued that we know in advance the right returns of many (s, a) pairs we do not consider signal noise and a source of randomness in the collected data. We can even consider that these sources exist, but are negligible we can then explicitly impose such constraints into the learning of the Q-funtiion.
Along this line, one tested trick is to select random tuples (s, s', 0) from the datasets and up- date the critic network with such tuples with the aim improving learning convergence. However, in our initial experiments, using this trick cause the Q-function to diverge if the ratio of this updates vs. the updates from the replay buffer is not controlled.
Transferring the knowledge from the policy to the Q-function Another trick is to try training the policy network via Supervised Learning with some random (sa, s') pairs that the policy can take advantage of Supervised Learning as well. But this will require a special procedure to update the critic network specifying confidence on some (s, a, R) tuples. This looks analogous
14

---

Under review as a conference paper at ICLR 2023
to the procedures used by offline-RL algorithms to constraint the Q-values out-of-distribution (OOD) points. We have tested this trick by helping the policy learning using random (sa, s') to try to improve learning, however without clear procedure to update the critic network well, there a fierce competition between the updates to the policy to optimize the critic and the Supervised Learning ones. Besides, these results show also divergence in the Q-function and the policy learning.
Avoiding the hacker agent against a learned model A known common concern the idea that an agent trained over learned forward model may exploit weaknesses of the model to gain unfair advantage optimizing the reward function (the cheating effect). Some authors have argued that this can be alleviated due to the inherent noise contained in the learned forward-model. Another simple idea is to over constraint the reward function for the learned forward models. We think that future research direction is how to prevent this cheating effect. However, in early experiments can not conclude that the agents systematically gain unfair advantages. Instead, what we observe is just slightly different behaviors that of course yield different returns, but not clear trend to optimize over the model, degrading the performance in the original environment.
15

---

Under review as a conference paper at ICLR 2023
A.2 HARDWARE AND SOFTWARE
For all the experiments we have used a Linux virtual machine with enough system RAM and Nvidia Tesla-T4 GPUs. The experiments presented here rely on the following Deep Reinforcement Learning specific software: for the environments (problems to solve) OpenAl Gym (Brockman al. (2016)) and MuJoCo (Todorov al. (2012)) are used. For Deep RL algorithms, the SAC implementation in the d3rlpy library (Takuma Seno (20TT) is used. For implementing replay buffer we do not use the d3rlpy's replay buffer implementation, instead the cpprb library (Yamada (2019) is used, Finally, as the Deep Learning backend and framework, Pytorch (Paszke et al. (2019) is used. D4RL datasets (Fu et al. (2020)) are used as as the source of dynamics trajectories. Every library and asset mentioned in this paper and used in this research has the license to be used without restriction.
A.3 ENVIRONMENTS' SPACES
In this section, present the list of variables of the state spaces for both the original problem M and the forward model approach MM for the three MuJoCo environment used in the experimental evaluation. We recall that the case of the forward model environment we have decided to predict only the position's 's state variables (target variables) because the velocities can be derived from them in order to reduce complexity of the problem. Those variables have been marked with an (*).
Table State space variables for MuloCo and derived forward environments
Hopper
Walker2d
Halfcheetah
rootx pos*
rootx pos*
rootx pos*
rootz pos*
rootz torso*
rootz pos*
rooty angle*
rooty torso angle*
rooty pos*
thigh joint angle*
thigh joint angle*
bthigh angle*
leg joint angle*
leg joint angle*
bshin angle*
foot joint angle*
foot joint angle* thigh left joint angle* leg left joint angle* foot left joint angle*
bfoot angle*
fthigh angle*
fshin angle*
ffoot angle*
rootx vel rootz vel
rootx vel rootz vel
rootx vel
rootz vel
rooty angle vel thigh joint angle vel leg joint angle vel foot joint angle vel
rooty angle vel thigh joint angle vel leg joint angle vel foot joint angle vel thigh left joint angle vel leg left joint angle vel foot left joint angle vel
rooty angle vel
bthigh angle vel
bshin angle vel bfoot angle vel fthigh angle fshin angle vel ffoot angle vel
16

---

end for Under review as a conference paper at ICLR 2023
A.4 TRAINING DATA PREPARATION
Training data for the forward model have been extracted from D4RL repositories (Fu et al. (2020)
)
For each example, several datasets from this library have been included in order to get large and diverse collection of trajectories (see Table 3 Additionally, and for training purposes they have
been filtered only those experiences with episodes longer than minimum number of steps (500 for the main experimental results and 100 for the additional tests). This filtering aims to find longer and more stable episodes learn.
Table 3: D4RL source datasets used per experiment.
Hopper-v2
Walker2d-v2
Halfchectah-v2
hopper-random--22
walker2d-randoo-v22
halfcheetah--rnddom-->
hopper-medium-yy
walker2d-medium--2
halfcheetahh-meiim--22
hopper-expert-v2
walker2d-expert--2
hallcchettab-experr-22
hopper-medduun--eplay-2
walker2d-medduu--rrlay---
malfcheetub-mmddiu--rppp
lalker2d-mediumg 2eediuu-expert-v22
halfcheetah-mmeiim-- -xpeuutt-2
A.5
TRAINING PROCEDURE
Here, you find the main algorithms of the presented forward models (Algorithms and
Algorithm Forward model main training loop
1: Initialize replay-buffer (RB), SAC algorithm (SAC) and forward model GYM environment (FW)
2:
3: total steps = 0
4: for episode do
5: steps 0
6: total reward =
7: results collect( SSAC) {(described at algo
8: for sample in results do
9:
RB + sample {(append)}
10:
steps + = sample[steps]
11: total reward = sample[reward]
12:
end for
13: total steps + steps
14: for in range(10) do
samples RB M.B.RB.ammppe AC..ppleeSAA..atttlll
16:
loss = SSACCupdate(samples)
18: Update metrics
if episode % 100 then
20: Save SAC model and policy
end if
22: end for
17

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Collect Algorithm
Initializate fu ouffer fer. env and SAC included as argument.
2:
3: state env.reset()
4: for in range (10000) do
5: if explore then
6: action SAC.sumple(stttte)
7: else
8:
action SAC.predict(state))
9:
end if'
step env.step(a) (described algo
11: state = step[s]
fur < step {append}
13: end for
14: return bur
Algorithm Forward model environment step algorithm
1: Initializate observation as stack, predicted state as buf fer and true state as buf fer. dataset included as argument, with the real experiences from d4rl datasets. action performed by SAC agent (sac action) is passed as argument.
2:
3: step counter +
4: rollout step counter + =
5: true obs = dataset[
6: true action dataset[step counter][action]
7: predicted state observation + sac action
8: observation = predicted state
9: if rollout step counter >= rollout step then
10: observation = true obs
rollout step counter 0
12: rollout terminal = True end if
14: predicted + predicted state, action, rollout terminal [append]
true state <- true obs, true action, terminal (append)
16: if step counter > total episode length then
17: terminal True
18: else
19: terminal False
20: end if
21: reward getReward(predicted state, true state, terminal)
22: return observation, reward, terminal
18

---

Under review as a conference paper at ICLR 2023
A.6 ADDITIONAL EXPERIMENTAL RESULTS
In this section, there are some detailed additional results that may result useful in order to understand only the training stage (i.e., actor and critic losses figures, well detailed training main metrics Table I. but also the testing of our forward model on dataset actions.
Walker2d v2
Walker2d v2
10
Critic
150
actor loss
critic loss
100
-10
50
-20
Actor loss
0
2500
5000
7500
10000 12500
0
2500
5000
7500
10000 12500
Episode
Episode
Figure 7: Walker2d SAC Actor loss and Critic loss (training) per episode
Halfcheetah_ v2
Halfcheetah v2
100
Actor loss
1000
Critic loss
actor loss
80
critic loss
800
60
600
40
400
20
200
0
0
2500
5000
7500
10000
12500
0
2500
5000
7500
10000 12500
Episode
Episode
Figure 8: HalfCheetah SAC Actor loss and Critic loss (training) per episode
19

---

Under review as a conference paper at ICLR 2023
Halfcheetah v2
rockr_pos 30, RMSE: 0.4641
True
Predicted
rooz pos 30, RMSE: 0.1376
0.25
pna 30, RMSE: 0.2266
0.00
bihign angle 30, RMSE: 0.1490
0.1700
-05
angle
0.0
-0.5
angle
angie
Boot angle 30, RMSE: 0.1370
inole
-0.5
0
20
40
80
80
100
120
140
Episode
Figure 9: Halfcheetah forward model prediction of target variables, compared to real values. Roll- outs of 30 steps. Sample 150 steps (5 full rollouts 30 steps each) obtained from random episode.
20

---

Under review as a conference paper at ICLR 2023
Walker2d v2
rocor_pos 30, RMSE: 0.2382
2
True
Predicted
0
roolz torso 30, RMSE: 0.0996
1.25
1.00
rocty torso angle 30, RMSE: 0.1182
Uigh_ jon, RMSE 0100
0.0
RMSE: 0.1185
-0.5
nor_join_aagge angle 0.186
thigh angle
0.25
-025
leg, left jont angle 0.3400
fool Jell Jir ange 30, RMSE 0.1787
-12
20
40
50
80
100
120
140
Episode
Figure 10: Walker3d forward model prediction of target variables, compared to real values. Rollouts of 30 steps. Sample of 150 steps full rollouts of 30 steps each) obtained from a full random episode.
21

---

Under review as a conference paper at ICLR 2023
Hopper v2
moxx_pos RMSE:
2
Predicted
1007 pos 30,
1.00
ronyy angle 30, RMSE: 0.0396
-0.2
thigh jant angle 30, RMSE 0.0854
0.0
-055
RMSE:
0.00
joint, angle
0.0
-0.5
0
20
40
60
80
100
120
140
Episode
Figure 11: Hopper forward model prediction of target variables, compared to real values. Rollouts of 30 steps. Sample of 150 steps full rollouts of 30 steps each) obtained from a full random episode.
22


Reviews:
[{"review_id": "zY4HgG1gqd", "reviewer": "Reviewer_NoVW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a reinforcement learning (RL) based formulation for learning forward models, which is often referred as learning the dynamics of an environment in a control setting. The authors first motivate the desire for computing forward models and then highlight some of the issues with current approaches to forward model computations that heavily on supervised learning setups. The authors the describe some of the potential advantages of computing forward models with RL and introduce a formulation to achieve that. Next, the authors introduce relevant definitions and design choices for their method, such as the choice of the policy (SAC-based Gaussian policy), the rollout loss based on trajectory similarity functions and a motivation for using Actor-Critic methods by leveraging ACs ability to resolve temporal credit assignment.\n\nIn their experiment, the authors study three different D4RL environments (Hopper-v2, Walker2d-v2, Halfcheetah-v2) and their specific forward model definition based on calculation of deltas of positions between different states. The first set of experiments outlines how well a RL trained model performs (measured in rmse loss) on three environments. The authors then compare their RL-based formulation to a supervised learning formulation for the same three environment and study the performance of each method. The results show that RL method generally takes longer to train and does not perform as well in RMSE as the supervised learning. The RL method does appear to have a lower error with longer rollout length compared to supervised learning.", "summary_of_the_review": "In its current form, the reasons to reject outweigh the reasons to accept the paper. I think that the paper could be significantly strengthened by providing greater clarity as to the significance of the results and its broader implications. ", "strength_and_weaknesses": "**Strengths**\n\n* The paper provides a novel formulation to train forward models in RL settings. The authors outline their method, its formulation, and relevant parts in great detail for the reader to follow.\n\n**Weaknesses**\n\n* The experimental data and the authors description of it does not provide a set of clear conclusions about the method. It would be good if the authors could provide greater clarity into the main conclusions from their experiments.\n* It's unclear how the authors proposed method would perform (RL calculated forward models) would perform in relevant application settings, such as model-based RL. The authors allude to this point in the conclusion and future work, but the paper would be greatly strengthened if indications of performance would be included.\n* Following on the above two points, the significance of the overall results is unclear. It would be good to add further clarity to this, ideally by additional experiments in an application setting or an indication of the advantages this method could provide.\n\n**Additional Questions**\n\n* What is dt in Table 1? I do not see it defined in the text near it.\n* Where is the reward in Figure 2? Is it the same as the RMSE, which would be confusing since it should be the signal loss, correct?\n* Could you clarify test rollout setting - is the model just predicting further into the future?\n* It's unclear what Figure 4 and Figure 5 are trying to show. Could you provide further description?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "csdZzuYelU", "reviewer": "Reviewer_xo1S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach based on using reinforcement learning for learning dynamics models in simulated robotics benchmark. A method based on SAC for learning multi-step models is implemented and compared to traditional supervised learning in terms of long-term accuracy in the predictions.", "summary_of_the_review": "Overall, the approach is a potentially interesting direction for effective multi-step model learning. However, there is a lack of comparison with related algorithms, both in the text and in the experiments, as well as an insufficient investigation of the implication for actual policy optimization. Thus, I for now recommend rejection.", "strength_and_weaknesses": "Strength:\n- The problem of learning multi-step models is important, especially compared to the amount of work that explicitly addresses it;\n- The motivation section, while being a bit overly schematic, explicitly states the goal of the work;\n- The approach seem to have an advantage compared to supervised learning, in the limited evaluation it was subjected to.\n\nWeaknesses:\n- The relationship with existing work is not clearly presented. Essentially presenting a method for learning a multi-step model, the paper should position itself, and compare experimentally, to other work in this space. A few works that come to my mind are the investigation of Erin Talvitie (e.g., https://arxiv.org/abs/1612.06018) and Kavosh Asadi (e.g., https://arxiv.org/abs/1905.13320) and their collaborators, as well as work more based on the option perspective (e.g., https://arxiv.org/abs/2108.03213). Without a comparison with at least some of these works, it is not easy to interpret the actual contribution of the paper;\n- In particular, there is a tight relationship between teacher/student forcing in RNN training and multi-step models. To put things into perspective, using reinforcement learning to train a model of the dynamics resembles using reinforcement learning for training RNNs. In the lack of a direct comparison or a justification for this, one might wonder if this is an unnecessary overhead over more traditional methods for learning multi-step models;\n- More broadly, showing that one can obtain more accurate models without showing how useful are they for reinforcement learning can generally be misleading: as it has been shown several times in recent work, an accurate model of the dynamics does not necessarily mean better resulting policies after policy optimization (see workshop on the topic: https://icml.cc/virtual/2022/workshop/13463).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aNBfi8OYjeG", "reviewer": "Reviewer_C9nJ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to learn forward models using RL methods on a new MDP who's states correspond to state-action tuples and who's actions correspond to additive changes in state. From this idea, they propose to use soft actor-critic to learn a gaussian transition model using negative prediction errors as reward with an additional reward injected at the end of rollouts related to the some negative error defined on the whole trajectory. This approach is then applied to 3 MuJoCo domains and compared to a supervised learning baseline.", "summary_of_the_review": "The authors proposes an original idea but the clarity of the paper prevents me from appreciating its possible benefits.", "strength_and_weaknesses": "The basic idea is original and could be interesting. However, there are are several limitations in it's current iteration.\n\nIt's not quite clear what inherent advantage there is to reformulate the model-learning problem as an RL problem. What property of this newly defined RL problem makes it better behaved than an equivalently expressive supervised learning approach? The authors seem to imply that one-step prediction errors are all that supervised learning could on this task, but there are a plethora of ways to formulate this learning problem. I don't see what would be the fundamental difference between \"rollout learning\" and a supervised learning approach which optimizes error defined on entire trajectories. The authors should take more time expanding their arguments and accompany them with explicitly written out equations.\n\nOne major limitation of the proposed method is that it doesn't address the inherent difficulties of learning stochastic transitions and instead limits itself to additive gaussian transitions. Could the main idea not be applied to discrete MDPs? Exploring the subject in illustrative discrete MDPs would help provide insight while also showing that the method depend on having limited stochastic transition models.\n\nSimilarly, the assumption that states support addition or subtraction greatly limits it's applicability to domains that have discrete or hybrid state-space.\n\nMinor comments\n=============\n\nI could not find the tips cited as Fleming 2018.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3LMmu-Y4Wd", "reviewer": "Reviewer_wuwo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to regard the system identification problem as a reinforcement learning problem. Compared with traditional supervised solutions, the proposed reinforcement method has advantages over delayed effects, high non-linearity, non-stationarity, partial observability, and error accumulation when using bootstrapped predictions.", "summary_of_the_review": "This paper proposed a wrapper framework to solve system identification problems as reinforcement learning tasks. The idea of recursive relation between states is beneficial to dealing with accumulative errors in rollouts. The logic is clear, but the presentation needs polish.", "strength_and_weaknesses": "Strengths:\n1. The research problem is important and the recursive reformulation of the forward model as the summation of the previous state and a derivate approximation is novel. System identification is one of the most important topics in the modern automatic control area. Vanilla supervised methods are intuitive and straightforward, yet suffer from several challenges, e.g., rollout testing and loss accumulation. The proposed forward model has advantages in solving the aforementioned challenges.\n2. The justification of the motivation to learn forward models with RL and the design of function approximation is reasonable.\n3. The experiments are supportive.\n\nWeaknesses:\n1. The major difference is to reformulate the system identification problem into a recursive relation, and take accumulation error into account. In my opinion, the proposed method still uses a supervised learning strategy. Reinforcement learning here only plays the role of an optimization tool, which optimizes the supervised loss.\n2. The paper is not well-polished. Subfigures in Figure 3 is massive and some figures overlap with each other.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject