Self-Consistent Models and Values
Authors: Gregory Farquhar
Kate Baumli
Zita Marinho
Angelos Filos
Matteo Hessel
Hado van Hasselt
David Silver
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
model-based reinforcement learning
planning
value equivalence

Abstract: Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment.
Models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions.
In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly \emph{self-consistent}.
This lies in contrast to classic planning methods like Dyna, which only update the value function to be consistent with the model.
We propose a number of possible self-consistency updates, study them empirically in both the tabular and function approximation settings, and find that with appropriate choices self-consistency can be useful both for policy evaluation and control.

Self-Consistent Models and Values
Gregory Farquhar DeepMind
Kate Baumli DeepMind
Zita Marinho DeepMind
Angelos Filos University Oxford
Matteo Hessel
Hado van Hasselt
David Silver
DeepMind
DeepMind
DeepMind
Abstract
Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate way of augmenting model-based RI., by additionally encouraging a learned model and value function be jointly self-consistent. Our approach differs from classic planning methods such Dyna, which only update values be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.
Introduction
Models of the environment provide reinforcement learning (RL) agents with flcxible ways of making predictions about the environment. They have been used to great effect in planning for action selection [45, 29, 49], and for learning policies or value functions more efficiently [51]. Learning models can also assist representation learning, serving Traditionally, models are trained to be consistent with experience gathered in the environment. For instance, agent may learn maximum likelihood estimates of the reward function and state-transition probabilities, based on the observed rewards and state transitions. Alternatively, an agent may learn model that only predicts behaviourally relevvann quantities like rewards, values, and policies [47].
auxiliary task, even if not used for planning [25, 48].
In this work, we study a possible way to augment model-lcarning, by additionally encouraging learned model nn and value function ê to be jointly self-consistent, in the sense of jointly satisfying the Bellman equation with respect to in and é for the agent's policy æ. Typical methods for using models in learning, like Dyna [51], treat the model only update the value to be consistent with the model. Self-consistency, by contrast, jointly updates the model and value be consistent with each other. This may allow information flow more flexibly between the learned reward function, transition model, and approximate value. Since the true model and value self-consistent, this type of update may also serve useful regulariser.
fixed best estimate of the environment, and
We investigate self-consistency both in a tabular setting and at scale, the context of deep RL. There many ways formulate model-value update based on the principle of self-consistency. but we find that naive updates may be useless, or even detrimental. However, one variant based on semi-ge gradient temporal difference objective accelerate value learning and policy optimisation. We evaluate different search-control strategies (i.e. the choice of states and actions used in the self- consistency update), and show that ceff-consistency can improve sample efficiency in environments such as Atari, Sokoban and Go. We conclude with experiments designed to shed light on the mechanisms by which our proposed self-consistency update aids learning.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2 Background
We adopt the Markov Decision Process (MDP) formalism for an agent interacting with an environment
[7, 42]. The agent selects an action A, E A in state S; E S each time step t, then observes reward Re+1 and transitions to the successor state S,+1. The environment dynamics is given by a "true" model m² = (r, P), consisting of reward function r(s, a) and transition dynamics P(s'|s a).
The behaviour of the agent characterized by a policy π : S AA, a mapping from states to the space of probability distributions over actions. The agent's objective is to update its policy to maximise the value (s)) = Em x Thr
S = of each state. We are interested in model-based approaches to this problem, where agent uses a learned model of the environment rh (f, P). possibly together with learned value function i(s) R v", optimise π.
The simplest approach to learning such a model is to make a maximum likelihood estimate of the true reward and transition dynamies, based on the agent's 's experience [32]. The model can then be used to perform updates for arbitrary states and action pairs, entirely in imagination. Dyna [51] is one algorithm with this property, that has proven effective in improving the data efficiency of RL [22, 31].
An alternative model-learning objective value equivalence' i.e. equivalence of the true and learned models in terms of the induced values for the agent's policy. This can be formalised as requiring the induced Bellman operator of the model to match that of the environment:
T 2 T
where the Bellman operator for model m is defined by T models do not need to capture every detail of the environment, but must be consistent with it in terms of the induced Bellman operators applied to certain functions (in this case, the agent's value v²).
En,, [r(s, a) + yu(ss), Such
We take a particular interest in value equivalent models for two reasons. First, they are used in state- of-the-art agents, described in the next section and used baselines in our deep RL experiments. Second, these algorithms update the model using a value-based loss, like the self-consistency updates we will study in this work. We describe this connection further in Section 3.3.
2.1 Value equivalent models in deep RL
A specific flavour of value equivalent model is used by the MuZero [47] and Muesli [25] agents. Both use representation function (encoder) he construct a state 2; (o1.)))) from history of observations o1.t² We define a deterministic recurrent model m = ma in this latent space. On each step the model, conditioned on an action, predicts a new latent state z and reward ft; the agent also predicts from each = value ot and policy tt.. We use superscripts denote time steps in the model: i.c., 2 k is the latent state after taking k steps with the model, starting in the root state =f 24-
Muesli and MuZero unroll such a model for K steps, conditioning on a sequence of actions taken in the environment. The learned components are then trained end-to-end, using data gathered from interaction with the environment, D by minimizing the loss function
Chasc cheso (m, 0, + (D;) = ED: Σ [e k=
larget
target of) + E ( + + (1
target at)].
(2)
This loss is the sum of a reward loss E", value loss l and a policy loss ET. Note that there is no loss on target the latent states zf: e.g. these are not required to match 21+4 = ha ho(01:44)) The targets *t4 for the reward loss are the true rewards Re+ observed in the environment. The value targets are constructed sarget from sequences of rewards and r--bootstrap value estimates u = Σド= 7 r++kkt + "
The bootstrap value estimates û, as well the policy targets, are constructed using the model m. This is achieved in MuZero by applying Monte-Carlo tree search, and, in Muesli, by using one-step look-ahead to create MPO-like [1] targets. Muesli also employs policy-gradien, objective which does not update the model. We provide some further details about Muesli in Appendix B.2, as we use
'We borrow the terminology from Grimm al. [19], who provided theoretical framework to reason about such models. Related ideas were also introduced under the name 'value aware' model learning [15, 14] We denote sequences of outcomes of random variables O o1...., O: oe 01 for simplicity.

---

Algorithm 1: Model-based RL with joint grounded and self-consistency updates.
input :initial m, output :estimated value i Rs v" and/or optimal policy n repeat
Collect D from m following π
Compute grounded loss: C (rn, 0, (D+)
1/ e.g. Eq.
Generate D, from ín following 1
Compute self-consistency loss: ( pD)
// see Eqs. (4,5,6)
Update m. é, π by minimising (e.g., with SGD): L = Casse + L until convergence:
it as a baseline for our deep RL experiments. For comprchensive descriptions of MuZero and Muesli we refer to the respective publications [47, 25].
In both cases, the model is used to construct targets based on multiple imagined actions, but the latent states whose values are updated by optimising the objective (2) always correspond to real states that were actually encountered by the agent when executing the corresponding action sequence in its environment. Value equivalent models could also, in principle, update state and action pairs entirely in imagination (similarly to Dyna) but, to the best of our knowledge, this has not been investigated in the literature. Our proposed ceff-consistency updates provide one possible mechanism do so.
Self-consistent models and values
A particular model in and policy æ induce a corresponding value function uy which satisfies a Bellman equation T vx = v.g We describe model-value pairs which satisfy this condition as self-consistent. Note that the true model m and value function vx are self-consistent by definition.
If an approximate model nn and value are learned independently, they may only be asymptotically self-consistent, in that the model trained to converge to the true model, and the estimated value to the true value. Model-based RL algorithms,, such as Dyna, introduce explicit drive towards consistency throughout learning: the value is updated to consistent with the model, addition to the environment. However, the model is only updated to be consistent with transitions in the real environment D and not to be consistent with the approximate values. Instead, we propose to update both the model and values so that they are self-consistent with respect to trajectories D,, sampled by rolling out model n under action sequences sampled from some policy μ (with μ
general). We conjecture that allowing information to flow more freely between the learned rewards, transitions, and values, may make learning more efficient. Self-consistency may also implement useful form of regularisation - in the absence of sufficient data, we might prefer model and value to be self- consistent. Finally, in the context of function approximation, may help representation learning by constraining it to support self-consistent model-value pairs. In this way, self-consistency may also he valuable an auxiliary task even if the model is not used for planning.
3.1
Self-consistency updates
For the model and value be eventually useful, they must still be in some way grounded the true environment: self-consistency is not all you need. We are therefore interested in algorithms that include both grounded model-value updates, and self-consistency updates. This may be implemented by alternating the two kind of updates, or in single optimisation step (c.f. Algorithm 1) by gathering both real D and imagined Dr, experience and then jointly minimize the loss
L(m, I D',) Chase C é, क x|);) ("" i(D.)
There are many possible ways to leverage the principle of self-consistency to perform the additional update to the model and value (Lee). We will first describe several possible updates based on 1-step temporal difference errors computed from K step model rollouts following μ but we will later
3

---

consider more general forms. The most obvious update enforces self-consistency directly:
K
Cseresidual (fn,, 6) E
Σ
F(Ss; as) + +0((+++)) -((8))
(4)
When used only to learn values, updates that minimize this loss are known as "residual" updates. These have been used successfully in model-free and model-bases deep RL [57]. even without addressing the double sampling issue [4]. However, it has been observed in the model-free setting that
residual algorithms can have slower convergence rates than the TD(0) update [4], and may fail find the true value [35]. In our setting, the additional degrees of freedom, due to the loss depending learned model, could allow the system to more easily fall into degenerate self-consistent solutions (e.g. zero reward and values everywhere). To alleviate this concern, we can also design our self-consistent updates to mirror a standard TD update, by treating the entire target ((kk, ax) + (08++)) fixed:
Cr0-direct (i, 0) = Eii,,
{ (F(Ss)) (1((((S)))))) LA
Σ
K
(1
+ + -6(s))
)
(5)
where 1 indicates a suppressed dependence (a "stop gradient" in the terminology of automatic differentiation). This is sometimes referred to using the "direct" method, gradient objective [52]. Dyna minimises this objective, but only updates the parameters of the value function (Fig. la). We propose to optimise the parameters of both the model and the value (Fig. lb).
using semi-
For 0, this will not update the transition model, as v(sg) does not depend on it. This necessitates the use of multi-step model rollouts (k 2 1). Using multi-step model rollouts is also typically better in practice [27], at least for traditional model-based value learning. This form of the self-consistency update will also never update the reward model. We hypothesise this may be desirable property, since the grounded learning of the reward model is typically well-behaved. Further, if the grounded model update enforces value equivalence with respect to the current policy's 's value, the transition dynamics are policy-depnndent, but the reward model may be stationary. Consequently, we expect this form of update may have particular synergy with the value- aalue-equivalent setting.
Alternatively, we consider applying a semi-gradient in the other direction:
K
27
CSC- revereeeee (m,, 0))
Σ
F(su, ux) 70(8k+1) - (($s))
(6)
This passes information forward in model-time from the value to the reward and transition model, even if k = 0.
3.2 Sell-consistency in tabular setting
First, we instantiate these updates in a tabular setting, using the pattern from Algorithm 1, but alternating the updates to the model and value for the grounded and self-consistency losses rather than adding the losses together for a single joint update. At each iteration, we first collect a batch of transitions from the environment (s,a, s') m The reward model is updated as f(8, a) +- r(s, a) ar (r(s, a) - f(s, a)). We update the value with TD(0): 0(s) + (1 a. (x)) (r(18,))) + 70()")) Here, Or, and or, learning rates.
Next, we update the model, using either a maximum likelihood approach or the value equivalence principle. The transition model P is parameterised as a softmax of learned logits: P(s' [s, a) = softmax(p(s. [s']. To learn a maximum likelihood model we follow the gradient of log P(s a). For value equivalent model, we descend the gradient of the squared difference of next-state values under the model and environment: (E:r ((aas) P(s'|s, a))(s') iss)))
Then, update either the value only to be consistent with the model (Dyna), update both the model and value to be self-consistent according to the objectives in the previous section. For control, we can construct an action-value estimate Q(s a) = f(s, a) + + E.- P(s. a, s') s'))('') with the model m (f,P), and act e-greedily using Q.

---

22
Y02
+
LD
772
TD
CTD
01
70
0
6
711
m
172
2
772
-0
772
771
(a) Dyna value update
(b) SC-Direct model & value update
(c) VE learning
Figure I: Schematic of model and/or value updates for k = 1 of a multi-step model rollout. Model predictions are red; dashed rectangle identifies the TD targets; superscripts denote steps the model rollout. Real experience is in black and subscripted with time indices. Blocks represent functions; when color-filled they are updated by minimising a TD objective. (a.b) show planning updates that use only trajectories generated from the model: (a) Dyna updates only the value predictions to be consistent with the model; (b) our SC-Direct jointly updates both value and model be self-consistent. (c) The value loss in the MuZero [47] form of VE learning is grounded update that is similar in structure to SC, but uses real experience to compute the TD targets. The model unroll must therefore use the same actions that were actually taken in the environment m grounded update like (c) may be combined with updates in imagination like (a) or (b). Best viewed in color.
3.3 Self-consistency in deep RL
Self-consistency may be applied to deep RI. as well, by modifying a base agent that learns a differentiable model m and value using the generic augmented loss (3). We follow the design of Muesli [25] for our baseline, using the objective (2) to update the encoder, model, value, and policy in grounded manner. We then add to the loss an additional self-consistency objective that generalises to the latent state representation setting:
rs (G o(
C Ex Zg Bay a
The base objective (2) already jointly learns model and value to minimise a temporal-difference value objective 2" faraget 0). Our self-consistency update is thus closely related to the original value loss, but differs in two important ways. First, instead of using rewards and next states from the environment, the value targets G are (K -tte bootstrapped value estimates constructed with rewards and states predicted by rolling out the model m. The reliance of MuZero-style VE on the environment for bootstrap targets is illustrated schematically colf-consistency. Second, because the targets do not rely on real interactions with the environment, may any starting state and sequence of actions, like in Dyna.
Figure Ic, in contrast to
Zo denotes the distribution of starting latent states from which roll out the model for K steps, and y is the policy followed in the model rollout. When differs from we may use off-policy corrections to construct the target ŌZ in our experiments, we use V-Trace [13]. For now, we default to μ π. and to using the same distribution for Zo as for the base objective. We revisit these choices in Section 4.2. Note that by treating different parts of the loss in 7 fixed, can recover the residual, direct, and reverse forms of self-consistency.
The self consistency objective is illustrated schematically in Figure 1. Subfigures (a) and (b) contrast Dyna update to the value function with the SC-Direct update, which uses the same objective to update both value and model. Subfigure (c) shows the type of value equivalent update used in MuZero and Muesli. The objective similar SC-Direct, but the value targets come from real experience. Consequently, the model unroll must use the actions that were taken by the agent in the environment. seff-consistency instead allows update the value and model based on any trajectories rolled out in the model; we verify the utility of this flexibility in Section 4.2.
5

---

Baseline
Dyna Evaluation (VE)
SC-Residual
SC-Direct
SC-Reverse
0.5p
Evaliation (MLE)
1.00
(MLE)
1.00
Control (VE)
Normalled value erver
0.5
value
0.95
0.85
64
0.80
0.90
6.3
0.85
0.85
82
0.80
0.00
25
25
0.75
25
50
Number of
(a) Sclf-consistency in random tabular MDPs. Each experiment was run with 30 independent replicas using different random seeds. (Left) Normalised value prediction error for policy evaluation, using MLE and VE models respectively. (Right) Normalised policy values for control, using MLE and VE models respectively.
sokoban
aben
assault
asteroids
wizard
Avg. episode return
24k
24k
30k
3ick
16K
164
1.5k
6kc
84
10k
-10
100M
200M
100M
200M
1000
0
0
100M 200M
200M
Number
(b) Self-consistency on Sokoban and 5 Atari games, Each experiment was run with independent replicas using different random seeds. The self-consistency updates were applied to variants of Muesli, model-based RL agent using deep neural networks for function approximation and value equivalent models for planning.
Figure 2: An evaluation of Dyna and the self-consistency updates described in section 3.1. Each vari- ant was evaluated in both a tabular setting (a) and with function approximation (b). The experiments included both MLE and value equivalent models. Shaded regions denote 90% CI.
Experiments
4.1 Sample efficiency through self-consistency
Tabular. In first set of experiments, used random "Garnet" MDPs [2] to study different combinations of grounded and self-consistent updates for approximate models and values. We followed an alternating minimization approach described in Section 3.2. In each case, applied the Dyna, SC-Direct, SC-Reverse, SC-Residual updates at each iteration starting at every state in the MDP. See Appendix A for further details of our experimental setup.
Figure 2a (on the left) shows the relative value error, calculated as ((v(s) - ((s))))(s)//o(o))) averaged across all states in the MDP, when evaluating random policy. As expected, Dyna improved the speed of value learning over a model-free baseline. We saw considerable difference among the self-consistent updates. The "residual" and "reverse" updates were ineffective and harmful, respectively. In contrast, the "direct" self-consistency update was able to accelerate learning, even compared to Dyna, for both MLE and value-equivalent models.
Figure 2a (on the right) shows the value of the policy (normalised by the optimal value), averaged across all states, for control experiment with the same class of MDPs. At each iteration, we construct ((s,a) a) estimates using the model, and follow an é-greedy policy with 0.1. In this case, the variation between methods is smaller. The "direct" method still achieves the best final performance. However, the "residual" and "reverse" self-consistency updates competitive for tabular control.
Function approximation. We evaluated the same selection imagination- updates in a deep RL setting, using variant of the Muesli [25] agent. The only difference is in the hyperparameters for batch size and replay buffer, as documented in the Appendix (this seemed perform more stably baseline in some carly experiments). All experiments were run using the Sebulba distributed

---

architecture [26] and the joint optimization sketched in Algorithm 1. We used 5 independent replicas, with different random seeds, to assess the performance of each update.
We evaluated the performance of the updates on a selection of Atari 2600 games [6], plus the planning environment Sokoban. In Sokoban, we omit the policy-gradiens objective from the Muesli baseline because with the policy gradient, the performance ceiling for this problem is quickly reached. Using only the MPO component of the policy update relies more heavily on the model, giving us a useful testbed for our methods in this environment, but should not be regarded as state-of-the-art.
We first evaluated Dyna baseline, using experience sampled from the latent-space value-equivalent model. As in the tabular case, we did so by taking the gradient of the objective (7) with respect to only the parameters of the value predictor 8. Unlike the simple tabular case, where Dyna performed well, we found this approach to consistently degrade performance at scale (Figure 2b, "Dyna"). To the best of our knowledge, this was the first attempt to perform Dyna-style planning with value-equivalent models, and was a perhaps-Surprising negative result. Note the contrast with the success reported in previous work [23] for using Dyna-style planning with MLE models on Atari.
Next we evaluated the self-consistency updates. We found that the "residual" self-consistency update performed poorly. We conjecture that this was due to the model converging on degenerate trivially celf-consistent solutions, at the cost of optimising the grounded objectives required for effective learning. The degradation in performance was even more dramatic than in the tabular setting, perhaps due to the greater flexibility provided by the use of deep function approximation.
This hypothesis is consistent with our next finding: both semi-gradiens objectives performed much better. particular, the "direct" self-consistency objective increased sample efficiency over the baseline in all but one environment. We found the "reverse" objective to also help in multiple domains, although less consistently across the set of environments we tested. As a result, in subsequent experiments we focused on the "direct" update, and investigated whether the effectiveness of this kind of self-consistency update can be improved further.
4.2 Search control for self-consistency
Search control is the process of choosing the states and actions with which to query the model when planning [52]; we know from the literature that this can substantially influence the effectiveness of planning algorithms such as Dyna [40]. Since our self-consistency objective also allows flexibly updating the value of states that do not correspond to the states observed from real actions taken in the environment, we investigated the effect of different choices for imagination policy μ and starting states Zg.
Which policy should be followed in imagination? We now explore four choices for the policy followed in imagination to generate trajectories used by the self-consistency updates. The simplest choice is to sample actions according to the same policy use to interact with the real environment (μ = TT. A second option is to pick a random action with probability e, and otherwise follows क (we try e=0.5). The third option is to avoid the first action that was actually taken in the environment to ensure diversity from the real behaviour. After sampling a different first action, the policy æ is used to sample the rest (Avoid ag). The last option is to replay the original sequence of actions that were taken in the environment (Original actions). This corresponds to falling back to the base agent's 's value loss, but with rewards and value targets computed using the model in place of the real experience.
In Figure 3a, we compare these options. notable finding was that just replaying exactly the actions taken in the environment (in brown) performed considerably worst than re-sampling the actions from π (in blue). This confirms our intuition that the ability to make use of arbitrary trajectories is a critical aspect of the self-consistenge update. Introducing more diversity by adding noise to the policy π had little effect on most environments, although the use of e = 0.5 did provide a benefit in assault t and asteroids. Avoiding the first behaviour action, to ensure that only novel trajectories are generated in imagination, also performed reasonably well. Overall, as long as we generated a new trajectory by sampling from policy, the self-consistency update was fairly robust to all the alternatives considered.
From which states should we start imagined rollouts? A search control strategy must also pre- scribe how the starting states are selected. Since we use latent-space model, we investigated using latent-space perturbations to select starting states close (but different) from those encountered in the environment. We used a simple perturbation, multiplying the latent representation of clement-wise
7

---

Original actions
c=0.5 Avoid ac
Latent noise, μ Latent noise, 8=0.5
π
Normalised AUC
1.25
1.00
0.75
0.50
0.25
0.00
sokoban
alien
assault
asteroids
wizard
zaxxon
(a) Different search control strategies for self-consisteney. Each experiment used 5 random seeds, bars denote 90% CI.
atar157 median
Median butman- Iumnan--orrmee nont-ioommellann score
600%
Mussii
Muneli SC
0.6
100%
LASER
Winrase Pachi
0.5
0.4
300%
Mathod
200%
Sainbow
SC
MCTS
100%
0.1
True
False
0.0
75
100
125
150
175
200
0
500 M
1.5 Environment steps
1.
2G
2.5
Millions of frames
(b) Median human-normaliss normalised episode return on across 57 Atari games. Each experiment used 4 random sceds, shaded region denote 90% CI.
(c) Performance 9x9 self-play Go, with eval- uation against Pachi [5] Each experiment used random seeds, shaded region denote 90% CI.
Figure Search control for self consistency, and evaluations on Atari and Go.
by noise drawn from ((0.8,) 1.2) (Latent noise, л). The results for these experiments are also shown Figure 3a. The effect was not large, but the variant combining latent state perturbations with exploration in the imagination policy (Latent noise, 0.5) performed best amongst all variants we considered. The fact that some benefit was observed with such rudimentary data augmentation suggests that diverse starting states may be of value. Self-consistnncy could therefore be effective in semi-supervised settings where we have access to many valid states (c.g. reachable configurations of robotic system), but few ground-truth rewards or transitions.
full evaluation. In the next experiments we evaluated our seff-consistency update on the full Atari- 57 benchmark, with the search-control strategy found to be most effective in the previous section. As shown in Figure 3b, we observed a modest improvement over the Muesli baseline in the median human-normalise
score across the 57 environments. We also evaluated the same self-consistency update in a small experiment on slaf-play 9x9 Go combination with Muesli-MPO bascline, with results shown in Figure 3c. Again, the agent using self-consistency updates outperformed the baseline, both with (solid lines) and without (dashed lines) the use of MCTS at evaluation time. This confirms that the self-consistent model is sufficiently robust to be used effectively for planning online.
4.3 How does self-consistens work?
Representation learning. In our final set of experiments we investigated how self-consistency affects the learning dynamics of our agents. First, we analysed the impact of self-consistency on the learned representation. Figure we compare (1) a policy-gradient baseline "PG" that does not model (2) learning value-equivalent model purely an auxiliary task "+VE Aux", and (3)

---

VE
AuX
1.5
0.05
0.5
Normalised AUC
1.25
Nornallsed AUC
1.0
1.00
0.75
0.5
0.50
0.25
alien
assault
asteroids
wizard
20:30n
0.00
solobbr
alin
axpuul
asterods
wizurd
201005
(a) Policy gradient baseline (PG), adding VE as auxiliary task (+VE Aux) improves the representa- tion, adding SC improves even further (+SC Aux).
(b) Episode return area under curve function of the ratio between number of imagined trajectories and real tra- jectories in each update to the model and value parameters.
Figure 4: Self-consistency can help representation learning an auxiliary task.
Evaluation
Evaluatioo
Evaluation (MLB)
Evaluation
25
40-
Value error AUC
200
20
Mathod Dyns
200
115
80
100
SC-Direct
100
C0 Fimitizllsation diddev
20
60
30
60
40 50
Finttialisation stddev
Inlttallsatioo stddev
Vinitialssation stddev
Figure 5: Tabular policy evaluation error as the initialisation of reward (left) and value (right) are varied. Gaussian noise is added to the true or T to initialise f and ö. Error bars show 90%CI.
augmenting this auxiliary task with our self-consistency objective "+SC Aux". When we configure Muesli to use the model purely as an auxiliary there is no MPO-like loss term, and the value function bootstrap uses state values rather than model-based Q values. We found that self-consistency updates were helpful, even though the model was neither used for value estimation nor policy improvement; this suggests that self-consistency may aid representation learning.
Information flow. Self-consistency updates, unlike Dyna, move information between the reward model, transition model, and value function. To study this effect, we returned to tabular policy evaluation. We analysed the robustness of self consistency to initial additive Gaussian noise in the reward model. Figure 5 (left) shows the area under the value error curve as a function of the noise. With worse initialisation the efffectivnness of the self-consistency update deteriorates more rapidly than the Dyna update, especially when learning with MLE. The reward model is learned in a grounded manner, so will receive the same updates with Dyna or SC. In the Dyna case, the poor reward initialisation can only pollute the value function. With SC, information can also flow from the impaired reward model into the dynamics model, and from there damage the value function further in turn. A different effect can be seen when varying the value initialisation by adding noise to the true value, as shown in Figure 5 (right). Since the value is updated by SC, it is possible for poorly initialised value to be fixed more rapidly if SC is effective. Indeed, we see a slight trend towards a greater advantage for SC when value initialisation is worse. However, this is not a guarantee; certain poor initialisations could lead to relf-reinforcing errors with SC, which is reflected in the overlapping confidence intervals.
Scaling with computation. Self-consisenncy updates, like Dyna, allow use additional computation without requiring further interactions in the environment. In Figure 4b, we show the effect of rolling out different numbers of imagined trajectories to estimate the elff-consistency loss, by randomly choosing a fraction of the states in each batch to serve as starting points (for ratios greater than one we randomly duplicated starting states). In the domains where self-consistency was beneficial, the potential gains often increased with the sampling of additional starting-states, However, in most cases saturation was reached when one imagined trajectory was sampled for each state in the batch (ratio=1 was the default in the experiments described above).

---

5
Related work
Model learning can be useful to RL agents in various ways, such as: aiding representation learning
[46, 28, 33, 20, 25] (ii) planning for policy optimisation and/or value learning [55, 51, 21, 10]; (iii) action selection via local planning [45, 49]. See Moerland al. [36] or Hamrick et al. [24] for a survey of how models are used in RL. Many approaches to learning the model have been investigated the literature, usually based on the principle of maximum likelihood estimation (MLE) [32, 51, 38, 21, 31, 10, 22]. Hafner et al. [23] notably showed that a deep generative model of Atari games can be used for Dtna-style learning of a policy and value only the imagination the model, performing competitively with model-free methods.
RL-informed objectives have been used in some recent approaches to learn implicit or explicit models that focus on the aspects of the environment that are relevant for control [53, 50, 39, 16, 30, 47, 18, 44, 25]. This type of model learning is connected to theoretical investigations of "value-aware" model learning [15], which discusses an equivalence of Bellman operators applied to worst-case functions from hypothesis space; "iterative value-aware model learning" [14] which, closer to our setting, uses an equivalence of Bellman operators applied to a series of agent value-functions; and the "value equivalence" (VE) principle [19], which describes equivalences between Bellman operators applied to sets of arbitrary functions.
All the approaches to model learning discussed above leverage observed real experience for learning a model of the environment. In contrast, our self-consistency principle provides a mechanism for model learning using imagined experience. This is related to the work of Silver et al. [50], where self-consistency updates were used to aid learning of temporally abstract models of Markov reward process. Filos et al. [17] also used ceff-consistency losses, for offinn multi-task inverse RL. Self- consistency has been applied in other areas of machine learning as well. Consider, for instance, back tack-translttion [8, 12] natural language processing. or CycleGANs [58] in generative modelling.
In this paper we considered models that can be rolled forward. It is also possible to consider backward models, that can be used to assign credit back in time [54, 11]. The principle of self-consistency we introduce in this work can be extended to these kinds of models fairly straightforwardly. A different type of celf-consistency was studied concurrently by Yu al. [56], who develop a model-based system where forward and backward model are trained to be cyclically consistent with each other.
Conclusion
We introduced the idea of self-consistent models and values. Our approach departs from classic planning paradigms where model is learned to be consistent with the environment, and values are learned to be consistent with the model. Amongst possible variants, we identified as particularly promising update modelled on semi-gradient temporal difference objectives. We found this update to be effective both in tabular settings and with function approximation. Self-consistency updates proved particularly well suited to deep naluveequivalent model-basee agents, where traditional algorithms such as Dyna were found to perform poorly. The self-consistency objectives discusses in this paper are based a form of policy-evaluation; future work may investigate extensions to enable aauuveequivalent model-based agents to perform policy improvement for states in arbitrary trajectories drawn from the model.
Acknowledgments and Disclosure of Funding
We would like to thank Ivo Danihelka, Junhyuk Oh, Iurii Kemaev, and Thomas Hubert for valuable discussions and comments on the manuscript. Thanks also to the developers of Jax [9] and the DeepMind Jax ecosystem [3] which were invaluable to this project. The authors received no specific funding for this work.
10

---

References
[1] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum Posteriori Policy Optimisation. In International Conference Learning Representations, 2018.
[2] T. W. Archibald, K. McKinnon, and L. C. Thomas. On the generation of markov decision processes, Journal of the Operational Research Society, 46(3):354-361, 1995.
[3] I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, Cai, Clark, I. Danihelka, Fantacci, Godwin, C. Jones, T. Hennigan, M. Hes- sel, S. Kapturowski, T. Keck, I. Kemaev, M. King, L. Martens, V. Mikulik, T. Norman, J. Quan, G. Papamakarios, R. Ring, F. Ruiz, A. Sanchez, Schneider, Sezener, Spencer, S. Srinivasan, W. Stokowiec, and F. Viola. The DeepMind JAX Ecosystem, 2020. URL http://github com/ deepmind.
[4] L. Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pages 30-37. Elsevier, 1995.
[5] P. Baudiš and J. L. Gailly, Pachi: State of the art open source Go program. In Advances in computer games, pages 24-38. Springer, 2011.
[6] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[7] R. Bellman. A Markovian decision process. Journal of Mathematics and Mechanics, 6(5): 679,684, 1957.
[8] O. Bojar and A. Tamchyna. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 330-336, 2011.
[9] J. Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin G. Necula, A. Paszke, VanderPlas, Wanderman- Python+-NmmPy programs, 2018. URL testt://githan com/ google/jax.
and Q Zhang. JAX: composable transformations of
[10] A. Byravan, J. T. Springenherg. A. Ahdolmaleki, R. Hafner, M. Neunert, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models. In Conference Robot Learning, pages 566-589, PMLR, 2020.
[11] V. Chelu, D. Precup, and H. van Hasselt. Forethought and Hindsight in Credit Assignment. In Advances in Neural Information Processing Systems, volume 33, pages 2270-2281, 2020.
[12] S. Edunov, M. Ott, M. Auli, and D. Grangier. Understanding back-translation at scale. arXiv preprint arXiv: 1808, 09381, 2018
[13] I.. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et Impala: Scalable distributed deep-rl with importance weighted learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[14] A. M. Farahmand. Iterative Value-Aware Model Learning. In NeurlPS, pages 9090-9101, 2018.
[15] A. M. Farahmand, A. Barreto, and D. Nikovski. Value-aware loss function for model-based reinforcement learning. Artificial Intelligence and Statistics, pages 1486-1494, PMLR, 2017.
[16] G. Farquhar, T. Rocktäschel, M. Igl. and S. Whiteson. Treeqn and atreec: Differentiable tree-structured models for deep reinforcement learning. arXiv preprint arXiv.1110.11417. 2017.
[17] A. Filos, C. Lyle, Y. Gal, S. Levine, N. Jaques, and G. Farquhar. PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning, arXiv preprint a1X1v:2102.12560, 2021.

---

[18] K. Gregor, D. Rezende, F. Besse, Y. Wu, H. Merzic, and A. Oord. Shaping belief states with generative environment models for arXiv preprint arXiv: 09237, 2019.
[19] C. Grimm, A. Barreto, S. Singh, and D. Silver. The Value Equivalence Principle for Model- Based Reinforcement Learning. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[20] A. Guez, F. Viola, T. Weber, L. Buesing, S. Kapturowski. D. Precup, D. Silver, and N. Heess. Value-driven hindsight modelling. arXiv preprint arXiv:2002.08822 2020.
[21] D. Ha and J. Schmidhuber. World models. arXiv preprint 1803. 2018.
[22] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination, arXiv preprint arXiv:/91 /9/2006020,0 2019.
[23] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering Atari with Discrete World Models. arXiv e-prints, art. arXiv:2010.02193, Oct. 2020.
[24] J. B. Hamrick, A. L. Friesen, F. Behbahani, A. Guez, F. Viola, S. Witherspoon, Anthony, L. H. Buesing, P. Veličković, and T. Weber. On the role of planning in model-based deep reinforcement learning. In International Conference on Learning Representations, 2021.
[25] M. Hessel, I. Damihelka, F. Viola, A. Guez, S. Schmitt, L. Sifre, T. Weber, D. Silver, and H. van Hasselt. Muesli: Combining Improvements in Policy Optimization. CoRR, abs/2104.06159, 2021.
[26] M. Hessel, M. Kroiss, A. Clark, I. Kemaev, J. Quan, T. Keck, F. Viola, and H. van Hasselt. Podracer architectures for scalable Reinforcement Learning. CoRR, abs/2104.0627 2021.
[27] G. Z. Holland, E. J. Talvitie, and M. Bowling. The effect of planning shape on dyna-style planning in high-dimensional state spaces. arXiv preprint
0225, 2018.
[28] M. Jaderberg. V. Mnih, W. M. Czameeki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuuglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.0539) 2016.
[29] L. P. Kaelbling and T. Lozano-Péree Hicrarchical Task and Motion Planning in the Now. In Proceedings of the 1st AAAI Conference Bridging the Gap Between Task and Motion Planning, AAAIWS' 10-01, page 33-42. AAAI Press, 2010.
[30] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5129-5136. IEEE, 2018
[31] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, Levine, R. Sepassi, G. Tucker, and H. Michalewski. Model-Based Reinforcement Learning for Atari. CoRR, abs/1903.00374, 2019.
[32] R. Kumar and P. Varaiya. Stochastic systems: Estimation, ideniiiccation, and adaptive control. SIAM, 2015.
[33] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv: (9X.009907.0095: 2019.
[34] M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562 2018.
[35] H. R. Maei. Gradient Diiforal-iiffeeence Learning Algorithms. PhD thesis, CAN, 2011. AAINR89455.
[36] M. Moerland, J. Broekens, and C. M. Jonker. Model-based reinforcement learning: A survey. arXiv preprint 2rXiv:2006.16712, 2020. 2020.
12

---

[37] R. Munos, T. Stepleton, A. Harutyunyan, and M. G. Bellemare. Safe and efficient off- off-po rein- forcement learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 1054-1062, 2016.
[38] Oh, X. Guo, H. Lee, Lewis, and Singh. Action-conditional video prediction using deep networks in atari games. arXiv preprint arXiv: 1507.08750 2015.
[39] Oh, S. Singh, and H. Lee. Value prediction network. arXiv preprint arXiv:/707.03493 2017.
[40] Y. Pan, J. Mei, and A.-M. Farahmand. Frequency- Frequency-bbsed Search-control in Dyna. In International Conference on Learning Representations, 2020.
[41] T. Pohlen, B. Piot, T. Hester, M. G. Azar, D. Horgan, D. Budden, G. Barth-Maron, H. Van Has- selt, J. Quan, M. Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv: 1593, 2018.
[42] M. L.. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
[43] S. Racanière, T. Weber, D. P. Reichert, I.. Buesing, A. Guez, D. Rezende, A. P. Badia, O. Vinyals, N. Heess, Y. Li, al. Imagination-auummenned agents for deep reinforcement learning. In Proceedings of the st International Conference on Neural Information Processing Systems, pages 5694-5705, 2017.
[44] D. J. Rezende, I. Danihelka, G. Papamakarios, N. R. Ke, R. Jiang, T. Weber, K. Gregor, H. Merzie, F. Viola, J. Wang, et al. Causally correct partial models for reinforcement learning. arXiv preprint (.Xii:2002.028360 2020.
[45] Richalet, A. Rault, Testud, and J. Papon. Model predictive heuristic control. Automatica (journal of IFAC), 14(5):413-44 428, 1978.
[46] Schmidhuber. on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In 1990 IJCNN international joint conference neural networks, pages 253-258. IEEE, 1990.
[47] J. Schritwiissrr, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering Atari, Go, chess and shogi by planning with learned model. Nature, 588(7839):604 609:660996699900 Dec 2020. ISSN 1476-4687.
[48] J. Schritwiiserr, T. Hubert, A. Mandhane, M. Barekatain, I. Antonoglou, and D. Silver. Online and Offline Reinforcement Learning by Planning with Learned Model. arXiv e-prints, Apr. 2021.
[49] D. Silver and J. Veness. Monte-Carlo Planning in Large POMDPs. In Advances in Neural Information Processing Systems, volume 23, pages 2164-2172. Curran Associates, Inc., 2010.
[50] D. Silver, H. van Hasselt, M. Hessel, T. Schaul, A. Guez, T. Harley, G. Dulac-Arnold, D. Re- ichert, N. Rabinowitz, A. Barreto, et al. The predictron: End-to-end learning and planning. In International Conference Machine Learning, pages 3191-3199. PMLR, 2017.
[51] R. S. Sutton. Dyna, an Integrated Architecture for Learning, Planning, and Reacting, SIGART Bull., 264):160-163, July 1991. ISSN 0163-5719. doi: 10.1145/122344112277
[52] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA. 2018.
[53] A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. arXiv preprint arXiv: 1602 02867, 2016.
[54] H. van Hasselt, M. Hessel, and J. Aslanides. When to use parametric models in reinforcement learning. In Advances in Neural Information Processing Systems, volume 32, pages 14322- 14333, 2019.
13

---

[55] P. J. Werbos. Learning how the world works: Specifications for predictive networks in robots and brains. Proceedings of IEEE International Conference on Systems, Man and Cybernetics, NY, 1987.
[56] T. Yu, C. Lan, W. Zeng, M. Feng, and Z. Chen. PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning. arXiv preprint arXiv:2106.041: 2021.
[57] S. Zhang, W. Boehmer, and S. Whiteson. Deep Residual Reinforcement Learning. In AAMAS 2020: Proceedings of the Nineteenth International Joint Conference Autonomous Agents and Multi-Agent Systems, May 2020.
[58] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] We do not provide code, but the experimental setup is described in detail in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please refer the supplemental material.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] All results show 90% CIs estimated with bootstrapping. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to the supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "xFrAQephl2u", "reviewer": "Reviewer_2igq", "summary": "This paper considers the problem of model-based reinforcement learning. The authors introduce a new loss, which drives the learned model and value function to be consistent with each other, rather than the model being consistent with the real environment and the value function being consistent with the real environment and the learned model. Experiments show that the proposed self-consistency loss can improve the sample efficiency and final performance. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing can be improved. The writing for method and intuition are reasonable, but for experiments, a lot of missing details make understanding it difficult. \n\nIt might be a good idea to introduce Muesli to readers who are not familiar with it. It's hard to understand what \"using only the MPO component of the policy update\" means without any proper introduction to Muesli. The paper mainly uses Muesli as its baseline while the experiments also use \"a variant of the Muesli agent\". I didn't find a detailed description of the variant. The lack of code, even implementable pseudocode, and implementation details, together with a heavy computational resources requirement, make reproducing this paper very difficult. \n\nThis paper only applies self-consistency to Muesli (and a grid world) in many domains with discrete control (Sobokan, Atari games, 9x9 self-play Go), while it seems to me that the self-consistent model/values are also applicable to other model-based RL algorithms in continuous control domain, e.g., MBPO. \n\nEq.7: How is G computed? Please give an explicit form rather than simply citing a paper if it's not elementary.  \n\nSec 4.1: What's the model-free baseline? How's the value normalized? \n\n> (L.174) In contrast, the “direct” self-consistency update was able to accelerate learning, even compared to Dyna, for both MLE and value-equivalent models.\nI didn't see a significant advantage of SC-direct over Dyna with VE models. \n\nSec 4.1: Please provide a short description of the environments used (especially Sokoban). \n\nFig.6 (a): 6 curves in the legend, 4 curves in plots? \nFig.6 (d): The figure caption should match the titles of each subplot. \n\nThe improvement of Muesli + SC over Muesli seems to be minor. \n\nSec 4.2: Is the policy pi deterministic or stochastic? \n\n> (L.257-259) We found that self-consistency updates were helpful, even though the model was neither used for value estimation nor policy improvement...\nI don't understand. In both VE aux and SC aux, the model is used to improve the value function. What does \"the model was neither used for value estimation\" mean? \n\nL.273: \"by randomly choosing a fraction of the states in each batch to serve as starting points\". What's the \"batch\" here? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I didn't find where the authors discuss the limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "g03OTRctwr", "reviewer": "Reviewer_TCCY", "summary": "The paper offers a new approach to model-based reinforcement learning with their idea of self-consistency of models and values. In the traditional (Dyna) model-based RL paradigm, an approximate model is learned with ground truth transitions and values (and policies) are updated to be consistent with the learned model. The proposed self-consistency update however, formulates a new update to the approximate model so as to make it consistent with the current value estimates using completely imagined (virtual) experience. This update is inspired by minimizing the bellman residual of the approximate model and value for a given policy (since for the true value and model, the bellman equation is satisfied and the residual is zero). An empirical analysis is presented for a set of variations of the proposed self-consistency objective with two types of (grounded) model updates -- MLE (maximum-likelihood) and value equivalent updates. In both tabular and function approximate settings, improved sample efficiency is demonstrated by the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper presents a novel self-consistency objective with a thorough empirical analysis of its variants on raw performance / sample efficiency (in tabular and deep RL settings), choices of search control (i.e. policy and states to use in imagined experience) and its benefits to representation learning. While not all experiments show a clear cut benefit of using self-consistency in all the cases visited in this paper, the unbiased analysis itself is an important contribution.\nAs a result, I feel that this paper in its current form is a significant and relevant contribution for the model-based RL community.\n \nBelow are more specific strengths and weaknesses.\n \n## Strengths\n- The paper proposes a self-consistency update which offers a new take on model updates in model-based RL. This update is important and distinct from most prior work in that it uses imagined (virtual) experience as opposed to ground truth (real) experience, allowing for freedom to update the model without collecting additional data from the true environment. This also leads to questions about what states and policy to choose for such updates and these questions were answered in the search control strategies section (Section 4.2).\n- The self-consistency update is also motivated as a regularizer and shown to promote information “flow” in the presence of noisy inputs (Section 4.3 - Information Flow).\n- Three different variations of the self-consistency updates are studied -- sc-residual, sc-direct and sc-reverse with a clear explanation of how each affects updates for the policy, reward or transition model. While sc-direct is the primary method, the ablations are also evaluated in the experiments providing valuable insights for future work in this area.\n- The benefits of self-consistency to representation learning are insightful, especially given that the self-consistency is used as a purely auxiliary objective where the model was not used for value estimation or policy improvement.\n- The evaluation of the method on Atari and Go show a clear sample efficiency improvement of the objective on top of the baseline methods.\n \n## Weaknesses\n- The motivation for using value equivalent model learning in addition to MLE for the model-update choice could be made more clear in the paper. Currently, I was able to understand the connection to value equivalence based on the fact that both self-consistency and value equivalence take inspiration from the bellman operator or bellman equation for their updates. However, the following statements were difficult to understand.\n    - (a) Lines 76-78 (“Value equivalent models could also … in literature”) -- I have no idea how this can be achieved without further details. This sentence seems to support the inclusion of value equivalence as a major part of this paper so further details would help. \n    - (b) Lines 124-126 (“Further, if grounded … value-equivalent setting”). It is not made clear how the policy dependence of the transition model can allow for synergy with the proposed self-consistency update. Is it because of the information flow? Further details on this would help as this is another important justification for the use of value equivalence.\n    - Overall, the link between value equivalence and self-consistency is not explicitly addressed but rather is inferred after reading the above two sentences. It is also missing from the introduction. I would recommend addressing the link and the motivation for using value equivalence early on in the paper before existing value equivalent methods are introduced.\n \n- The experiment on information flow in Section 4.3 is an important contribution despite the lack of statistically significant differences between Dyna and SC-Direct. However, I do not understand the justification for the poor reward estimates affecting self-consistency more while the poor value estimates affect Dyna more. The inversion of trend from reward noise to value noise is the issue here. The trends mostly have overlapping CIs other than the first figure, so I would recommend either dropping any strong claims on difference in performance or making the justifications for these trends more clear.\n \n- Line 312 in conclusion (last sentence): This future work sentence is not clear. Why is there a need to perform policy improvement for value equivalent model-based agents in imagined experience? Don’t existing value equivalent model-based agents already perform policy improvement in this way e.g. [11]?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have sufficiently addressed limitations of their work. For example, Section 4.3 demonstrates the effect of noise in reward or value estimates on their method in comparison to the Dyna baseline.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Qyapmo_9nj", "reviewer": "Reviewer_nzxk", "summary": "This work proposes to improve model-based RL methods by enforcing the model and value learned to be “self-consistent”, i.e. to satisfy the Bellman eqn where both the Bellman operator and value function are defined in terms of the model. The motivation behind this is that in classic MBRL, the model and value function are learned separately, and the value function is typically learned to be consistent with the model but not vice versa. The authors propose to enforce self-consistency by adding a \"self-consistency loss” (L_SC) to the base loss and jointly learning the model and value functions. L_SC is the squared TD error with a stop-gradient on the target and is calculated using rollouts from the model. Although in MuZero/Muesli the model and value functions are learned jointly as well, the targets of the loss are all defined with respect to data from the true environment and there is not term enforcing the “self-consistency” between model and value functions.\nExperimental evaluations on a selection of Atari games show that jointly updating the model and value functions with the additional L_SC term leads to improved sample efficiency as compared to Dyna and a model-free baseline. The authors also empirically study the search control problem.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths: \nThe paper is very well-written and clear. Figure 1 is a nice schematic comparison of the closely related methods and helps put the proposed approach in context. The motivation makes sense based on previous work and the proposed approach is straightforward. The empirical evaluations are thorough in looking at the different variations and their effects.\n\n### Weaknesses/Clarifications:\nThe related works section (particularly the first paragraph (Line 280-292)) could have more details and comparisons. \n\nQuestion: Since the model is unrolled for many steps in the computation of L_SC and each step of the unrolling is not “grounded” to the true data, it seems that the errors in the model and the value function would interact and compound with more unrolling steps and gradient updates. Did you observe a challenge like this in your evaluations?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do a good job of addressing the limitations of the method through empirical evaluations (e.g. the susceptibility of the value error to poor initialization of reward).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "HH4KDC7xC7w", "reviewer": "Reviewer_Vnna", "summary": "This paper presents the idea of self-consistent reinforcement learning.  By enforcing \"consistency\" between value functions and approximate  (learned) models, a model-based learning algorithm can use synthetic experience generated from the approximate model.  This paper presents a few minor variations on this theme, and explores the results empirically.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I liked this paper, although I wish the results were more compelling. I lean towards acceptance because I think this paper presents some good ideas, and that those ideas are worth exploring more.  The central idea that imagined experience can somehow be used to improve the model that it is generated from seems like a Really Good Idea, and I would like to see more work on it.\n\n* Originality\n\nAs far as I can tell, this work is original.  While the RL community has explored many of these themes, this particular combination of ideas seems novel.\n\n* Quality\n\nThe paper is of very high quality, blending novel technical insights, a solid positioning in the academic literature, and thoughtful experiments.  I appreciated the candid balance of positive and negative results - that's good scholarship.\n\n* Clarity\n\nThe paper is generally well-written, although much of the more technical parts are difficult to understand how exactly all of the pieces fit together.\n\n* Significance\n\nThe significance is mixed. While the central theme of the paper is potentially significant, and the technical development is solid, the experimental results are weak.  Often, the proposed methods perform only slightly better than competing methods.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not foresee any significant negative societal outcomes, beyond those inherent to all RL research.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a self-consistency approach to model-based RL. The self-consistency refers to jointly optimizing the value function and the model such that (a variant) of the Bellman residual is minimized.

The reviewers are all positive about this work. They believe that it is an original work and written (mostly) clearly.  Most of their concerns are also addressed during the discussion phase, and there is no concern that prevents them from accepting this work. Therefore, I would recommend the *acceptance* of this paper.

There are some minor issues that can be improved though. Please refer to the reviews for more detail. I enlist a few of them:

- Improving the writing, especially in the Experiments section, in which some details are missing.
- Explaining and studying the effect of the number of unrolled steps in the computation of L_SC.
- Some of the reviewers believe that the paper isn't very reproducible. Please try to improve it in this regard, by providing more detail or releasing (some parts of the) source code.
- Better explanation of the relation between value equivalence and self-consistency.
Decision: Accept (Poster)