Counterfactual Regularization for Model-Based Reinforcement Learning
Authors: Lawrence Neal
Li Fuxin
Xiaoli Fern
Venue:  (2020)
Keywords: Counterfactual
Model-Based Reinforcement Learning

Abstract: In sequential tasks, planning-based agents have a number of advantages over model-free agents, including sample efficiency and interpretability. Recurrent action-conditional latent dynamics models trained from pixel-level observations have been shown to predict future observations conditioned on agent actions accurately enough for planning in some pixel-based control tasks. Typically, models of this type are trained to reconstruct sequences of ground-truth observations, given ground-truth actions. However, an action-conditional model can take input actions and states other than the ground truth, to generate predictions of unobserved counterfactual states. Because counterfactual state predictions are generated by differentiable networks, relationships among counterfactual states can be included in a training objective. We explore the possibilities of counterfactual regularization terms applicable during training of action-conditional sequence models. We evaluate their effect on pixel-level prediction accuracy and model-based agent performance, and we show that counterfactual regularization improves the performance of model-based agents in test-time environments that differ from training.


Under review as a conference paper at ICLR 2020
COUNTERFACTUAL REGULARIZATION FOR MODEL-BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In sequential tasks, planning- planning-based agents have number of advantages over model-free agents, including sample efficiency and interpretability. Recurrent action-conditions latent dynamics models trained from pixel-level observations have been shown to predict future observations conditioned on agent actions aceu- rately enough for planning in some pixel-based control tasks. Typically, models of this type are trained to reconstruct sequences of ground-trute observations, given ground-truth actions. However, tions and states other than the ground truth, to generate predictions of unobserved counterfactual states. Because counterfactual state predictions are generated by differentiable networks, relationships among counterfactual states can be included in training objective. We explore the possibilities of counterfactual regulariza- tion terms applicable during training of action-conditional sequence models. We evaluate their effect on pixel-level prediction accuracy and model-based agent performance, and we show that counterfactual regularization improves the perfor- mance of model- based agents in test-time environments that differ from training.
action-conditional model can take input ac-
INTRODUCTION
Recent advancements in the use of variational inference and generative neural network architectures have made it possible to build accurate transition models for high-dimensional sequential decision- making environments. Networks have been trained accurately predict future states conditioned on actions in large Markov decision process state spaces including Atari, VisDoom, and pixel-based robotic control tasks (Kurutach et al. 2018) (Ha & Schmidhuber (0188) (Hafner et al. 2018). Using such transition model, an agent can plan sequences of future actions to maximize expected reward, using model-pra preddccivee control or other search algorithms. These approaches fall into the paradigm of model-basee reinforcement learning (RL), distinct from the standard model-free approach of training a policy or value network. Model-based approaches have better sample efficiency than model- -ree methods, which usually require large amount of training data to perform well.
There are distinct differences between traditional model-based RL, in which a transition between states are defined by Markov transition matrix, and model-basee approaches in which the state representation is learned, in which the state is represented as a continuous vector. This is made more unclear with environments that start with scene described an image, where it is well-known typical transition models that maximize a variational bound on the likelihood of future observations, similar variational autoencoder (Kingma & Welling 2013) tend to generate blurry samples and exhibit posterior collapse (van den Oord et al. (0117) phenomenon showing that the learned states may not be robust enough.
We hypothesize that representation in which states are factored and well-separated will improve the performance of model-based agents. In order to achieve this, we propose a new family of regular- izations applicable to conditooniii sequence prediction models, which we call counterfactual regularization. These regularizations are loss terms based on counterfactual states, which can be generated by feeding non-ground-truth actions states into a transition model. Specifically, we propose regularization terms to achieve two goals. First, the set of possible states should be distinct in such way that any change in input actions should result in a change in latent state. Second, that the dimensions of the learned state should be disentangled into separate factors in such a way that a change one factor should effect only sparse set of other factors.

---

Under review as a conference paper at ICLR 2020
81
8t+
a
a
T(s)
T(s)
St11
Suz
E(o)
D(s)
D(s)
01.11
02.
O3.33
O
Out fu11
Osea. T42
Figure I: Left: The graphical model representing the generative process. Shaded nodes are observed, and empty nodes are not observed. this example, the state at time / has been observed. Right: The latent state space is represented as convolutional feature map, and the transition function ((s) is implemented as a fully-convolutiona neural network.
In order to test these regularizations, we introduce the Starhntruders game, a sequential environment with test-time dynamics that vary from training dynamics to test the generalization performance of trained agents. We propose a convolutional recurrent latent-state architecture well-suited for modeling two-dimensional video environments and trained using counterfactual regularization. Our experiments show that a simple planning agent using an action-conditional model trained with the proposed counterfactual regularization outperforms both non-regularized models, and model-free agent, in test environments that differ from training.
LATENT DYNAMICS MODELS
2.1 PRELIMINARIES
We consider partially-observable Markov decision process (POMDP) in which agent observes at each time step an observation o, from an observation space O and a reward r; from reward space R and executes action a, from discrete set of actions A. The agent executes actions from = 1 until a terminal timestep f = T, generating a trajectory (0), F1, a1)...(o,, TT, ar). An optimal agent will take actions that maximize the cumulative reward Σ; over the trajectory.
In pixel-based domains, 0r may be a high-dimensional vector (e.g. = R 206r26663 and the underlying dynamics function p(or, re/o T't,, (ttaae)) may be known. We are concerned with the problem of learning from experience a parameterized function pe that estimates the distribution of future observations and rewards conditioned actions. We model the environment using neural network similar to the deterministic state space model from (Hafner et al. 2018), trained on trajectories collected by an exploration policy.
Using such function (a model of the environment), planning algorithms such as podel-predietive control can be applied compute an optimal action a, each timestep. Unlike model-free reinforce- ment learning agents, a model-based agent does not learn a policy network or value function estimate; instead, actions are chosen by searching among possible future trajectories.
Previous work has explored the efficiency benefits of model-based methods, which can achieve similar performance to model-free methods using fewer training examples (Hafner et al.) 2018) (Kaiser et al. 2019b). Our work instead explores the capability of model-based agents to generalize to changes in the environment.
The graphical model in Fig. illustrates assumptions about the generative process. We model all dependencies on previous timesteps through an unobserved latent state variable $t, which depends only the previous state and action 8t-1, dt-1- Each observation depends only the current state. Accurately predicting future observations and rewards depends on building a highly accurate latent dynamics model.
2

---

Under review as a conference paper at ICLR 2020
S
T
++1
T
S#2
T
SH3
E
D
D
D
Example sequence predictions in the Starlntruders task. The encoder E converts observed pixels to a latent state s. The recurrent transition function T(s, a) predicts future latent states, and decoder network D decodes states back to pixel space.
Figure 2:
2.2 MODEL ARCHITECTURE
Our latent dynamics model consists of the following components: an encoder O + S, an action-conditional transition function 7' S x A + S, decoder G:: S + O, and reward decoder convolutional deep neural network
R S - RRK, as shown Figure/ 1. Each component with strided and transposed convolutions, and LeakyReLU nonlinearities. Spectral Normalization (Miyato al.] (018) is applied each convolutional layer of T to assist in gradient propagation. For environments in which state estimation requires observation of multiple sequential states, the input layer of E is resized take as input a stack of K sequential frames. Where not otherwise noted, K = 3.
The transition network T' converts each integer action input to a one-hot categorical representation, broadcast across the spatial input to the network. The reward network It predicts positive or negative reward at each timestep, for each spatial location, for each reward type. To improve training stability, reward estimates discretized in the following way: each spatial location is classified as causing -1, 0, or +1 reward, and the output reward is the sum of all spatial locations.
2.3 TRAINING
The model is trained to reconstruct sequences of observations gathered from the true environment by an exploration policy. Given sequence of ground truth observations, the model must reconstruct future observations and rewards (see Figure 2). Similar to (Hafner et al.) 2018), our model consists of separate encoder/decoder networks which convert between observations and learned latent representation, and transition network which operates recurrently in the latent space (See Figure 1). This structure allows the latent representation to keep track of parts the state not visible in current observations.
2.4 MODEL-PREDICTIV CONTROL
Given a predictive model of future observations and rewards, we apply a simple model-predictive control algorithm to select actions at each time step during evaluation. The planning process produces a batch B 64 trajectories for each possible action a² with each trajectory taking a random rollout for a fixed horizon T 20. The agent executes the action a, that maximizes the estimated sum of rewards Di=t1 Ft. observes the next observation ot+1 and then re-plans each step.

---

Under review as a conference paper at ICLR 2020
at-1
at
3
St-1
St
St+1
S
Observed
Simulated Trajectory
Counterfactual Regularization Loss
att
a't+1
S'1+1
S 1+2
Counterfactual Trajectory
Figure 3: During training, in addition to predicting ground truth sequence based on actions dg, Or+-1... our model predicts counterfactual trajectories based on alternate actions d/, 4++1.. for which no ground truth exists. By enforcing desired relationships between alternate simulated trajectories, desired causal relationships can imbued into the model.
2.5 COUNTERFACTUAL TRAJECTORIES
transition network T takes as input latent state 8; and an action ds and produces an esti- mate of the successor state 81+1. The network is trained to simulate ground truth trajectory {(11, ((01,a1); (02, (2)...(O,,,))) aN)} generated by a policy a in the real environment.
The transition network is applied recurrently to generate sequence of states:
2 = T(8++ 1155+1100-11 11-1) ((T(st, az), a++1)
During training, is an observed state and is known ground-truth action, selected by the training exploration policy. However, an alternate action aj or an alternate state s can be input to the transition function, thereby generating alternate or counterfactual state: 8/+1 = T(8,, a/) or s'441 = T(s a.). Repeated application of the transition function then generates a counterfactual trajectory:
442 T(5
For many tasks, especially in physical environments, it may not be possible to repeat previously observed trajectory, and so the "true" value of counterfactual state may never be observable. However, the estimate E441 is computable, and because T is differentiable parameterized function, the gradient loss function involving generated counterfactual states can be computed with respect to network parameters.
Relationships among estimates of real and counterfactual states provide a language by which state- ments about causality can be converted into differentiable loss terms suitable for inclusion in neural network training process. Many useful desiderata can be posed in the language of counterfactuals, and converted to loss terms along with standard pixel-reconstruction and reward losses used in predictive models. By including a counterfactual loss term L in the objective function, the full loss becomes:
L Σ (((ss)) oz [[(ss)) rall3 AL
where and are the decoder and reward prediction networks, or, Pe a ground truth observation and reward for timestep t, & is predicted latent state output by more recurrent applications of T, is a scalar hyperparameter, and L is one of the counterfactual regularization terms described below.
4

---

Under review as a conference paper at ICLR 2020
3 CAUSAL REGULARIZATION THROUGH COUNTERFACTUALS
By generating counterfactual trajectories during training (see Figure 3), we can augment the training objective with new terms. We explore number of possible types of counterfactual regularization, given below.
3.1 ACTION-CONTROL REGULARIZATION
When exploring a complex environment with an untrained policy, it may be the case that the reward function be accurately modeled deterministic function of time, unrelated to the agent's actions. For instance, in a timed game ending with reward -1 or +1 indicating win or loss, a poorly-initialized agent might learn that its actions have no effect the environment, and may model the reward mechanism as a timer that outputs 0 or -1. This failure case could be considered analogous the psychological phenomenon of learned helplessness, the perception of independence between actions and outcomes (Langer 1975).
To address this problem, we might wish to limit learning system consider only models with the property that every action taken by the agent has a causal effect on the state. Using the language of counterfactuals, we can express this property precisely: for any state 8; reached by taking actions 41 -11-11 there must exist some -4.441 such that + s and 3 < t, a; # 04.
Considering
for simplicity, we can convert the above constraint into a regularization term:
min - log [[ - 1(
where si (s:-11 -1-1) is a counterfactual state generated by the alternate a/. This regularization ensures that the learned environment model must include a causal link from a, st, in the sense that for any s, there must exist a counterfactual action af which would have resulted in a different state s/.
In general case, the counterfactual a' might be multiple time steps removed from st. Using T(s, (1...Gn) to denote recurrent application of to simulate multiple timesteps, the term can be expanded to the general case of < 1-11
LR min - log (8 (T8,,, a.... - 04..,
For finite discrete action spaces, d can be minimized by exhaustive search, but for simple environ- ments it suffices to generate af by sampling uniformly from A {ar). This loss term has the effect of biasing the transition model by forcing the latent state change result of every agent action, which is an appropriate assumption for many control tasks.
3.2 DISENTANGLEMENT REGULARIZATION
Many practical environments contain separate and largely independent factors of variation. For example. in the game of Space Invaders, the agent controls one visible object on the screen (the player character) while enemy characters move across the screen, independently of the player. The only interaction between enemy position and player position is through projectiles fired by the player enemies. It might be desired that a model of the game represent the position of enemies as independent of the position of the player, conditioned on projectiles.
In a non-sequential variational autoencoder setting, FactorVAE (Kim & Mnih, 2018) applies regularization based on adversarial training using examples generated by permuting the dimensions of the latent representation of real examples. This forces the network to learn a representation such that any two latent dimensions could be swapped, and the result should still be realistic. The resulting representation maximizes the independence of latent dimensions.
Similarly but in sequential environment, we desire representation with the property that any two latent dimensions indices k,j of a state $, can be swapped to create an alternate s/, and the resulting sequence of counterfactual states $/ would continue with all dimensions that are independent of l and unchanged. That is, changing the values of some dimensions the latent representation should cause sparse and minimal change in other latent dimensions.
5

---

Under review as a conference paper at ICLR 2020
1=0
t= 10
t= 40
Cumuletive Reward:
Cumulative
Figure 4: An agent completing the training version of the StarInlruders task. Destroying enemy generates +1 reward, and destroying human generates -1, regularized model-based agent destroys fewer humans than non-regularized model-based agent or model-free agent, in perturbed versions of the environment.
We represent this sparsity with an LI regularization term:
LD
Σ (ELAA)
$6 1 (9)
where g(6) denotes the ith dimension (or channel) of the latent state vector 8. Here, 8t (1) (8) (J)
[K)] Si ..... ... ..... [kk) is the state vector time and sy st S...; S.... ....8
(1) ()) (k) (K)
is the vector with dimensions j and swapped. A fixed number of time steps H is used between the counterfactual intervention and the regularization. The effect of this regularization is promote sparse, rather than dense. relationships between latent factors. This is appropriate for environments where the underlying mechanics are discrete and deterministic.
EXPERIMENTS
4.1 ACTION-CONDITTONAL STATE PREDICTION
Our model is trained with ground-trun -ruth trajectories sampled by random exploration policy. We train with À = .01 and maximum prediction horizon of = 10 both environments. After training, the model-predictive control (MPC) agent is applied select the action with highest expected reward over the planning horizon at each time step.
4.2 GENERALIZATION EVALUATION ENVIRONMENT
We train and evaluate agents in the Star Intruders game, a custom sequential environment with mechanies similar Space Invaders, built using the StarCraft II Learning Environment (Vinyals et al. 2017). The state space of the task includes 256x256x3 RGB pixel matrix for visualization, and 64x64x4 binary mask input to the network, indicating the positions of each type of object. The
MSE H=3
MSE H-5
MSE H=10
MSE H-20
Avg. Game Score
Ablation .0017
.0021
.0025
.0027
4.43
Action-Control .0019
.0023
.0043
.0072
5.11
Disentanglement .0015
.0017
.0020
.0023
5.83
Table 1: MiniPacMan environment: Mean squared error (MSE) of predicted video frames at selected timestep horizons H. Right Column: Average score the model-predictiv control agent on 100 games. Ablation is trained with the standard reconstruction loss: each other network is trained with one of the proposed regularizations.

---

Under review as a conference paper at ICLR 2020
dynamics of the task are as follows (See Figure 4). The agent controls unit at the bottom of the friendly and enemy units patrol left and right At
game map. Above the agent, randomly-positionn each time step, the agent take one of four actions: move left, move right, fire, or no-op. The fire action fires a projectile from the player's current position, which annihilates upon contact with any enemy or friendly unit.
When a projectile destroys an enemy unit the agent receives a reward of 1, and when a projectile destroys a friendly unit, the agent receives reward -1. The optimal policy will fire projectiles aimed in such a way that all enemy units are destroyed but no friendly units are destroyed.
The training environment contains 8 friendly units and 8 enemy units. Each friendly unit begins at random location in the bottom half of the map, and moves horizontally in a repeating pattern, from its starting location to the rightmost edge of the map, then to the lefumost edge, and so on. Each enemy unit patrols similarly from random starting location in the top half of the map. Each episode continues until either all enemy units are destroyed, or maximum number of time steps TM 300 are elapsed. The maximum cumulative reward in any episode is 8.
4.2.1 GENERALIZATION EVALUATION
After training, each policy is evaluated in three environments. The Training environment operates exactly as described in the previous section. The Test A environment operates in the same way as the training environment, except that instead of patrolling right-to-left, each unit will patrol either right-to-left or left-to-right with equal probability. The Test B environment operates in the same way as the training environment, except that instead of starting random locations within the entire bottom half of the map. all friendly units are positioned within a small horizontal band close to the bottom of the map. From a subjective human perspective, the Test A and Test B environments are no more challenging than the training environment. Colors, textures, timing and input format in the test environments are identical the training environment.
For comparison with model-free methods, we train network with the Rainbow algorithm (Hessel et al. 2018) to maximize reward in the StarIntruders environment. The policy trained for 1M time steps in the original training environment. The trained policy is evaluated in the original training environment, and in each variant version of the environment.
4.3 RESULTS
Table 2 shows game score in the StarIntruders training and test environments. Without regularization, the model-free Rainbow algorithm outperforms our predec-pre control agent in the training environment. However, the model-based method generalizes to new environments more effectively, Applying counterfactual regularization to the predictive model improves the performance of our MPC agent in both the training environment and test environments.
Table I shows both game score and pixel prediction error for the simpler MiniPacMan task. We note that lower MSE does not necessarily coincide with higher score, so the proposed regularizations do not simply improve pixel prediction accuracy. We conclude that counterfactual regularization improves the representation learned by our environment model.
Method
Training
Test A
Test
Rainbow (Hessel et al. 2018)
7.38
4.92
5,05
MPC
6.50
7.30
7.00
MPC, Action-Control
7.60
7.70
7.60
MPC, Disentanglement
7.90
7.90
8.00
Table 2: StarIntruders environment: Per-episode cumulative reward for training and test tasks, average of 10 episodes. The same preee--peedictvve control (MPC) agent is applied using models trained with the standard reconstruction loss, and models trained with each proposed regularization. A model-free method, Rainbow is included for comparison.
7

---

Under review as a conference paper at ICLR 2020
5 RELATED WORK
Much previous work in model-based reinforcement learning involves planning with model-predictive control in low-dimensional state spaces using neural networks (Gal et al.) (Chua et al. 2018) or other models (Deisenroth & Rasmussen, 2011). Other work has applied generative neural network models to high-dimensional video prediction (Chiappa al., 2017) or state prediction in Markov reward processes (Silver et al. (017) where transitions are not action-conditional. Some work has explored cetion-conditional video prediction (Oh et al., 2015) without planning. Recent work has
applied model prediccve planning to high-dimensional image domains (Kaiser et al. 2019a (Hatner ctt al. 2018).
LEARNING WITH AUXILIARY REWARDS
Related model regularization, some work applies auxiliary tasks during training of model-free or model-based agent with the goal of improving performance or generalization. In (Jaderberg ct al. 2016) a representation is learned using auxiliary policies based on intrinsic rewards such as maximizing changes in pixel intensity, in model-free setting. In model- setting, in (Pathak the environment state trained using inverse or forward dynamics, and the prediction error of this dynamics model is subsequently used intrinsic reward signal to train a model-free policy network.
et al. 2017) representation
DISENTANGLEMENT OF FACTORS
Factorized state representations have long been known to improve performance in reinforcement learning tasks (Degris et al., 2006). In (Higgins et al., 2017), the Kullback-Liebler divergence term of variational autoencoder's objective function is reinterpreted as a regularization term promoting conditional independence of latent dimensions. In (Kim & Mnih 2018), independence of an autoencoder's 's latent dimensions is achieved via an adversarial loss.
MODEL-BASEE REINFORCEMENT LEARNING
Learning predictive model the environment has long been recognized as a more sample-efficient approach than model-frce policy learning. Predictive models were applied to discrete grid-world tasks in Dyna-Q (Sutton) 1990). (Chentanez et al.) 2005) and to low-dimensional robotic control tasks in PILCO (Deisenroth & Rasmussen [201T). In (de Avila Belbute-Peres et al. 2018), linear complementarity problem solver is used to simulate two two-dimensione
rigid body dynamics as differentiable layer within network, allowing sample-efficient prediction and high performance on control tasks. Other methods build models of environment dynamics which are then used to plan for control tasks (Lowrey et al.] (018)) (Chua et al. 2018). Many recent methods use generative deep network models trained to predict transitions in high-dimensiioal visual environments. Planning- based agents have been constructed using networks trained to simulate grid-based games such as Sokoban (Racanière al. 2017). Atari games (Kaiser et al. (019a), pixel-based physics tasks (Halner
let al. 2018) and visual robotic control tasks (Ebert et al.) 2018).
6 CONCLUSIONS
The capability to generate and reason about counterfactual states is already considered valuable when deploying agents that require interpretability or explainability (Wachter et al., 2017). We have shown how counterfactual states can also be used during the training process, in order to force desired causal dependencies or independencies that may be appropriate for given task or environment.
We have proposed counterfactual regularization terms appropriate for two-dimensional video tasks, but loss functions based on counterfactual states provide a flexible and general way to encode desired properties into loss function. Future work might explore new applications of counterfactual states to enforce desired causal relationships during the training process. Additional future work includes extending counterfactual regularization to architectures and more complex physical control tasks, as well as considering more sophisticated exploration policies for generating training data, and better planning algorithms at test time.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Nuttapong Chentanez, Andrew G. Barto, and Satinder P. Singh. Intrinsically motivated reinforcement learning. In K. Saul, Y. Weiss, and L.. Bottou (eds.), Advances in Neural Information Processing
Systems 17, pp. 1281-1288 MIT Press, 2005. http::::: http::::/pppp...pee .nips.nipp........ www.siicpppp....... cC/ 2552 intrinsicalll mot reinforcement -twprcemmmee--eaaanngngngn learning.poo.. wwwppppppppppppppppppct..c....ap
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint arXiv: TTXiv:/704.02254, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018.
Filipe de Avila Belbute-Peres Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. In Advances Neural Information Processing Systems, pp. 7178-7189, 2018.
Thomas Degris, Olivier Sigaud, and Pierre-Henri Wuillemin. Learning the structure of factored markov decision processes in reinforcement learning problems. In Proceedings of the 23rd international conference on Machine learning. pp. 257-264, ACM, 2006.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model- based and data-efficiie approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp. 465-472, 2011.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: 2018
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122. 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint Stf.:1811.0455:1 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Max Jaderberg. Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:161 1.166611.000 1.05397, 2016.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari, 2019a.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: h0:::/900 90390000777 2019h.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, arXiv preprint arXiv: arXiv:1802.00 1802.0092. 2018.

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Max Welling. arXiv:13/2.611,, 2013.
Auto-encoding variational bayes. arXiv preprint
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint orXiv:18022 10592, 2018.
Ellen J Langer. The illusion of control. Journal of personality and social psychology, 32(2):311, 1975.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv: 1.0018881...88..... 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-condition video prediction using deep networks atari games. In Advances in neural information processing systems, pp. 2863-2871, 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Auriosity-driverr exploration by sulf-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16 16-17, 2017.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- augmented agents for deep reinforcement learning. In Advances neural information processing systems, pp. 5690-5701, 2017.
David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, al. The predictron: End-to- end learning and planning. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 3191-3199. JMLR. org, 2017.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pp. 216-224. Elsevier, 1990.
Aaron van den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yco, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 1708.0:700.047777 2017.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gpdr. Harv. JL & Tech., 31:841, 2017.
10


Reviews:
[{"review_id": "rylvNBMCFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers regularization based on \"counterfactual\" trajectories.\nNamely, it suggests two losses, action-control and disentanglement regularization.\nIt experimentally evaluates the benefits of such regularization in the StarIntruders environment.\n\nThe paper is well written and explained.\n\nIssues:\n\n1) Authors evaluated the two suggested regularizations in separate. \nI would like to also see numbers from a combination of these.\n\n2) I think the related work is missing a large line of work on \"auxiliary tasks\".\nIt seems to me that this paper would exactly fit within that scope?\n\n3) My main issue is the evaluation.\nThe evaluation is done on a in-house game and compares to very few methods.\nFor a paper that has very little theory and thus most of the value is in the empirical evaluation, I think that is a problem.\nIf authors opted for example for Space Invaders (they do say it is similar) or simply more games, one would have many more existing numbers to compare against.\n\nMinor issues:\n\n1) The first regularization - action control regularization is motivated by the idea that there is always an action that changes the state. While true for most environments, this does not hold in general.\n\nSummary:\n\nOverall, this paper has potential but I don not believe is good enough - I suggest a reject.\nThe main problem is that the idea is relatively simple, there is no theory and thus the crucial piece of the paper has to be the empirical evaluation.\nAnd the evaluation only compares to a single method with no regularization, no auxiliary tasks and reports only experiments on a single game.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkgu8hhqYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents regularization techniques for model based reinforcement learning which attempt to build counterfactual reasoning into the model. In particular, they present auxiliary loss terms which can be used in \"what if\" scenarios where the actual state is unknown. Given certain assumptions, they show that this added regularization can improve generalization to unseen problem settings. Specifically they propose two forms of regularization: (1) enforcing that for different actions the predicted next state should be different (action-control) and (2) enforcing that when certain parts of the low dimensional state are perturbed, over a model rollout the perturbation should only affect the perturbed parts of the state, essentially encouraging the latent space features to be independent (disentanglement). \n\nOverall the idea is well motivated - incorporating counterfactual reasoning into model based RL has potential to to improve generalization. Also, while the assumptions needed for the regularization to be correct are not always true, they do seem to hold in many cases. Lastly, the results do seem to indicate that generalization is slightly improved when using the proposed forms of regularization.\n\nMy criticisms are:\n\n(1) As mentioned in the paper Action-Control assumes that at every single timestep the agent has potential to change the state. However there may be settings where the agent can always change state, but only a small component of the state. In these cases the states should be quite similar. For example a robot only moving a single object when the state consists of many objects. Also as mentioned in the paper Disentanglement will not work in stochastic environments. One concern I have is that since different environments can violate the assumptions to varying degrees, it seems like actually using the regularization and picking the correct hyperparameter to weight it will be very challenging. \n\n(2) The current results are only demonstrated in a single, custom environment. Additionally performance is shown on only 2 test tasks, and in all cases in Table 2 it is unclear how to interpret the reward. Does this performance constitute completing the task? What is the best possible cumulative reward in this case? The performance improvement seems small, but it is difficult to judge without knowing the details of the task.\n\nI think the paper would be significantly improved by (1) adding experiments in more environments, especially standard model based RL environments where the performance of many existing methods is known and (2) adding comparisons to other forms of model regularization, for example using an ensemble of models. My current rating is Weak Accept.\n\nSome other questions:\n- In Table 2 does MPC amount to PlaNet?\n- How sensitive are the current numbers to planning parameters (horizon, num samples)?\n- Can you provide error bars for the numbers in the tables?\n\n______________________________________________\n\nAfter author responses and closer examination of the paper I have some additional concerns about experimental details.  Changing my score from 'Weak Accept' to 'Weak Reject'", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hylr1WGlcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method called \"counterfactual regularization\" whereby the dynamics/transition model is encourage to not have degeneracies where the actions don't influence the state transitions.  Concretely, this is done by, for every state, computing the maximum deviation of Transition model under a different action than the action taken in the history, and encourage that deviation to be as large as possible.  If that maximum deviation is 0, then all actions lead to the same next state. Empirical results show reasonable improvements in the StarIntruders task.\n\nMy biggest complaint (and the only one barring me from supporting acceptance) is that I don't see the body of results as scientifically solid.  Example of additional results that I would find much more convincing are:\n\n-- Experiments on more than one environment.  Currently, this paper should be judged solely for its empirical improvements, because there is little formal analysis or rigorous derivations.  But it's hard to judge that based on only one experiment.\n\n-- Deeper investigation into the effects of counterfactual regularization, including its interaction with learning disentangled representations.  Right now, there is no investigation, just a numerical score of reward attained.  This does not lead to much scientific insight.\n\n-- Exploration of the limitations of the approach.  Personally, I think this approach is reasonable for video games with a few discrete actions, but quickly runs into problems for more complex action spaces.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the reviewers that this paper has serious limitations in the experimental evaluation.
Decision: Reject