Self-Supervised Policy Adaptation
Authors: Christopher Mutschler
Sebastian Pokutta
Venue:  (2020)
Keywords: reinforcement learning
environment representation
representation learning
model mismatch

Abstract: We consider the problem of adapting an existing policy when the environment representation changes. Upon a change of the encoding of the observations the agent can no longer make use of its policy as it cannot correctly interpret the new observations. This paper proposes Greedy State Representation Learning (GSRL) to transfer the original policy by translating the environment representation back into its original encoding. To achieve this GSRL samples observations from both the environment and a dynamics model trained from prior experience. This generates pairs of state encodings, i.e., a new representation from the environment and a (biased) old representation from the forward model, that allow us to bootstrap a neural network model for state translation. Although early translations are unsatisfactory (as expected), the agent eventually learns a valid translation as it minimizes the error between expected and observed environment dynamics. Our experiments show the efficiency of our approach and that it translates the policy in considerably less steps than it would take to retrain the policy.

Under review as a conference paper at ICLR 2020
SELF-SUPERVISED POLICY ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of adapting an existing policy when the environment representation changes. Upon change of the encoding of the observations the agent can no longer make use of its policy as it cannot correctly interpret the new observations. This paper proposes Greedy State Representation Learning (GSRL) to transfer the original policy by translating the environment representation back into its original encoding. To achieve this GSRL samples observations from both the environmen and a dynamics model trained from prior experience. This generates pairs of state encodings, i.e., new representation from the environment and a (biased) old representation from the forward model, that allow us to bootstrap a neural network model for state translation. Although early translations are unsatisfactory (as expected), the agent eventually learns valid translation it minimizes the error between expected and observed environment dynamics. Our experiments show the efficiency of our approach and that it translates the policy in considerably less steps than would take to retrain the policy.
INTRODUCTION
Model-free reinforcement learning (RL) achieved remarkable results the recent past and surpassed human performance on a number of complex tasks that have previously been considered intractable in applications such as playing games (Mnih al. 2015). and created new opportunities in areas such as control in robotics (Schulman et al.) 2015). Such agents directly interact with the environment and do not rely on a model for the environment as input.
While model-free (end-to-end) RL does not require priori information of the (environment) model and its dynamics, usually comes with number disadvantages. First, model-free RI. suffers from sample inefficiency: it is not uncommon that it takes millions of training samples and trajectory rollouts to converge to a good policy. Second, as end-to-end RL trains a policy directly on raw observations it is virtually impossible decision strategy (at least for high-dimensional input). Third, a trained policy that might already be running for long time and that has been proven successful in wolld application cannot simply be transferred to work with a changed environment or different sensory inpul (consider for instance robot whose sensors break down or whose sensor outputs degrade change, e.g., by radiation). Environment changes are also problematic for model-based agents. The first problem is well-studied and usually addressed with simulation nemulation-to-rcaltt----o to-reality- transfer (i.c., train in simulator and later transfer to the real world, possibly taking care of model mismatch) and the second one is comparably novel research field where we might make use of methods such as model extraction (Bastani et al. 2017). saliency maps (Greydanus et al. 2018). or PIRL (Verma et al. 2018).
provide any interpretation of the policy and the agent's s
However, yet there no common -ense approach to address changes to the enviromment, and in particular to the environment representation. Hence, in practice this often requires the policy to be retrained from scratch. If a system has already been deployed to the real-world application retraining of the policy becomes even harder we may not use elaborate techniques such as auxiliary tasks (Jaderberg et al. 2017), reward shaping (Ng et al. 1999). or hindsight experience replay (Andrychowicz al. 2017), which could have been used in a laboratory setting improve initial policy search. We cannot expect perfect extrinsic reward signals from the real world.
To the best of knowledge the adaptation of existing agent to changes in environment representation (without leveraging the value function or the underlying reward process) still remains a vastly untouched research field and we out to provide a first approach that shows promising results.

---

Under review as a conference paper at ICLR 2020
Environment
St
St+1
St+2
(new representationn
Agent
St
execute
m((s))
St+1
= ((
execute
si+2
Forward Model (old representation)
(s,
f(s.a)
St+1
t+2
Figure 1: Greedy State Representation Learning.
Only recently Caselles-Duptte et al. (2018) took state representations from variational autoencoders (VAEs) and use generative replay (Shin et al. 2017) to align the latent state representations that has been used before an environment representation change to the one that is used after this change. However, the application of this approach is limited in practice as it requires new autoencoder to be trained that captures all generative factors of the environment upon the change. In practice, we need to iteratively train and acquire new samples while partially exploring the environment to obtain comprehensive information about the generative factors (approaches that use VAEs always need to bootstrap (Ha and Schmidhuber 2018) unless the observation itself describes the entire environment, e.g. if the representation is a top- view of a maze that needs to be navigated.
The most simplistic idea is to acquire labeled samples from the environment (states for which know both the old and the new encoding), i.e., state-paut state-pairs, and use them to train a model that translates the state representations. However, we cannot obtain such pairs of state representations (i) the old raw input is no longer available, (ii) potentially stored observations from early runs (old encoding) cannot be matched the new observations, and (iii) cannot expect that oracle human provides manual translations for observations as the representation is usually too complex.
Greedy State Representation Learning (GSRL) obtains such state-pairs indirectly and uses them to train a neural network model that serves as a translator for observations obtained after the environment change into the representation corresponding to the environment before the change, so that we can re-use policies trained for the initial environment representation. Figure [ sketches the general idea. The environment (top) works on the new representation while the agent (below the dashed line) works on the old representation (blue). Using an initially randomly initialized model fa we translate a new state observation 8t from the environment into its old representation s (noisy translation), which use to select an action ae from the earlier learned policy π. We now use the obtained action twice. First, we query a forward model f (that captures the environment dynamics and that we have trained with the data acquired before the environment representation changed) to get estimate of the expected next state 4 in the old representation, i.c., our biased state estimation target. Second, execute the action a, in the environment and obtain the new state representation 8c+11 of the next state. We translate this new representation with the neural network model μs into s and calculate loss that we use to minimize the error relative our expected state estimation target 6 such that the translations come close to the target estimated by the forward model. Our experiments on mounta a near environment show the efficiency of our approach and that we can bootstrap a model that translates state representations in less iterations than it would take to retrain the policy.
The remainder of this paper is organized as follows. Section 2 discusses related work. Section 3 recalls basic definitions and foundations from reinforcement learning. Section 4 formalizes the considered problem and introduces our GSRL algorithm. Section describes the experimental setup and Section ddiicssee the results.
2 RELATED WORK
Classic approaches that are commonly used in sensor fusion such Kalman or particle filters only provide limited applicability for extreme changes to the representation. fault-t. control that uses, for instance, Gaussian processes and model-based controllers (e.g.. model predictive control, MPC) Yangg and Maciejowski 2015), provides more flexibility. However, such approaches also struggle with severe changes the representation and can longer reliably control such situations. Cully et al. (2015) propose a two-step algorithm to adapt after severe damage (e.g. a robot losing a limb), that (1) priori calculates behavior map that captures the expected performance of behavioral
2

---

Under review as a conference paper at ICLR 2020
models, and (2) evaluates those behaviors upon damage. However, this requires to enumerate and calculate the maps of all possible (partial) breakdowns a priori (which is usually intractable) it does not scale both in memory and computation time to real world applications. Approaches such as AdaPT (Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems) (Harrison et al. (017) compensate dynamics mismatch with MPC attenuate the bounded dynamics between source and target dynamics. However, AdaPT does not assume that the representation changes.
Domain randomization assumes small discrepancies between source and target domain. Added perturbations in training ensure that the policy does not overfit to the training environment and will later generalize well in the target domain. These approaches adapt the policy in the target domain (Dafur et al.] (016), use adversarial training (Pinto et al.] [017]] Mandlekar et al. (017)
with perturbed dynamics, assume and compensate for dynamics mismatches (Muratore et al. 2018
or adapt the reward that is provided by the environment (Wang et al., 2018 |Rmmof et al. However, they only work for mild changes to the environment but do not cope with severe changes (such completely missing elements).
2018
In contrast, domain adaptation trains on a particular input distribution with a specified reward structure (source domain) and then adapts the agent to a modified input distribution with the same reward structure (target domain). DARLA (Higgins et al.) 201 7b) learns disentangled generative factors of the environment (with 3- V-Es (Kingma and Welling 2014, Higgins et al. 2017a)) and uses this latent variable representation to learn a policy in the source domain. However, DARLA is no solution if perturbations severely affect the encoding of the generative factors (which is our assumption). Tzeng et al.] (2017) address domain shift and adaptation through weak pairs of visual input from two domains with similar structure and elements. A combined minimization of task, confusion, and pairwise loss makes the policy robust to domain shifts. The similarity to our approach lies in the acquisition of labeled training input from unlabeled observations but this approach assumes a similarity between the images, which we cannot expect in our setting.
There is also connection of the presented work here recent work in continual learning and multi-task learning (Parisotto et al. 2016).. E2C (Watter et al.] 2015) derives a latent state space representation based on VAËS which dynamics are locally linear to apply locally robust optimal control. While E2C translates the state representation to a latent space for linear control it does not address changes in the state representation. Finn et al. (2017) use semi-supervised RL to train an agent on set of tasks in environments where reward function is available and use inverse RL to generalize to unknown environments. However, while they deal with changed dynamics and rewards (as for the different tasks) they are not concerned with large, systematic domain shifts and changes to the environment representation. Guppa et al. (2017) propose a multi-agent transfer learning in a setting where two (physically different) agents learn multiple skills. For skills that have been learned by both agents, each of them constructs a mapping from their observed states to an invariant feature space. With a richer feature space an agent can learn a new skill by projecting the executions of the other agent into its own feature space. While common feature space would also be a solution for the problem at hand this approach cannot be applied as (upon a sudden change to the representation) there is no longer a contribution to the feature space by the agent that works on the old representations.
The work closest related to ours is Caselles- et al. (2018) who build a state representation model with VAFs. Upon detection of change to the environment (but not to the underlying dynamics) they sample from the latest VAE using generative replay (Shin et al., 2017) and then train an updated model together with new samples acquired from the environment. However, there are challenges depending on the environment and the exploration as the agent needs to iteratively use a greedy policy and sample all relevant areas of the environment (Ha and Schmidhuber 2018). If the agent does not see the whole environment at once this approach is eventually equivalent to retraining the policy.
PRELIMINARIES
We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the notation and without loss of generality we assume that the environment is fully observable, ie.., that or is a fully observed realization of the (true and fully described) environment state 8 at time step t. A Markov decision process (MDP) is deseribed by set of states S E R", a set of actions A E R. a distribution of initial states p(5n), a reward function S x A R, transition probabilities p(se+1 (8t, at), and discount factor 7 E [0, 1).
3

---

Under review as a conference paper at ICLR 2020
A deterministic policy is mapping from states to actions: : S - A. Every episode starts with sampling an initial state So. At every time step the agent produces an action based on the current state: a, (s²)). Then it receives the reward T, r(s,, az) and the environment's 's new state is sampled from the distribution p(11st, 0t). A discounted sum of (future) rewards is called a return: R2 Disr "r,"r, The agent's goal is maximize its expected return Exg (Ro so). The Q-function or action-value function is defined as Q* az) = IE [R az).
Let n² denote an optimal policy, i.e., any policy x" s.t. Q" (8, a) 2 (s,a) for every 8 E ES,aa E A and any policy л. All optimal policies have the same Q-function which is called optimal Q-fanction and denoted Q²" It is easy to show that it satisfies the Bellman equation:
(s, (x (((s.a) 7aaxmmxxxx ('', Q a'EA
In practice, the Q-function will be approximated with value function approximation (Mnih al. 2015), which also allows to scale RL to nigh-dimensional state spaces such as raw sensory input from cameras in an end-to-end setting. As an alternative an end-to-end approach we can also make use of function (8:) that, given set of parameters é, returns a compact representation of features that are relevant for an agent to control its effect on the environment. For instance, VAEs (Kingma and Welling 2014) have been demonstrated to learn structured latent representations of high dimensional data and also have been applied to RL (Higgins et al.. 2017b Ha and Schmidhuber) 2018).
4 GREEDY STATE REPRESENTATION LEARNING
4.1 PROBLEM FORMULATION
Now let π. 0, a₂ be policy that has been trained using some model-free RL algorithm, see Figure 2 In fact, the observation ot is generated by underlying function o, g(s)) that returns sensor information that we assume that fully describes the hidden environment state 8, (if o, is not Markovian such as frame stacking (Mnih et al. 2013) or recurrent networks to ensure this). In other words, g turns state into a state representation. In practice, such functions 9 are e.g., implemented in cameras to compute pixel arrays from light input. If or is large or complex, e.g., a camera image, we may also apply some post-processing on it (e.g., given by a VAE's encoder that extracts the latent variables from the observations) some manually engineered feature descriptors that provide a more concise representation of o,. Those transformations can be applied directly on 0g but we omit them to simplify the notation. We can o g(s;) to sample action from given policy our assumption is that π is a non-trivial complex, robust, and well-proven (or even certified) control policy for which re-estimation is considered costly. As such, upon change to the environment, our goal is to reuse the policy π rather than to retrain it from scratch.
env/sensor
St
may use methods
g
0r
Figure 2: MDP setting.
In the following
Now let assume further that (e.g., due to some unexpected event) the hidden process/function g(s:) that generates or changed and we instead obtain some perturbed version g g²(s) of. In real-world sensor system this might be caused by (a) damages of single sensors (e.g., dead pixels in camera frames) or by (b) loss of functionality of sensor in a multi-sensoe system (if all the sensor input is collected and provided Markovian realization of the hidden state 8; such that the MDP assumption still holds (in (a) the sensor output is perturbed but still unique and (b) other sensors might compensate for the damage). However, under the assumption that o, and of are too different cannot expect to use the policy to sample actions (and probably we neither can apply the same preprocessing as we did it on oz).
the agent in an end-to-end RL setting). We still assume that of is
Our goal is to infer a translation model with parameters & that converts the observation of to its original value o,, both for the same hidden state s, so that we can reuse the policy. In other words, Me should translate between the distributions that generate the observations from the states. An easy solution is to let the agent sample the environment by following its policy and then to use pairs of observations (ot, of) 8t to train model using supervised learning. However, as the agent only sees
4

---

Under review as a conference paper at ICLR 2020
env/sensor
env/sensor
env/sensor
Sp+1
S
g
0
0f+2
Mig
My
Hig
AIA
ALA
ALA
Oc+1
Uc+2
R
Skc +
Gc+1
monolate to
Dc+2
Ironslate to
Oc+3
Figure 3: Translating the state representation: (I) the new observation is provided by the environment (and optionally further preprocessed); (2) a neural network model translates the observation to its old pendant; (3) using this observation we sample action a, from policy (4.1) through the forward model f we obtain a prediction for the succeeding observation 0c+1 and (4.2) by sampling the environment obtain the succeeding observation 1i (5) train pap with [0 0t+1]-
observations from single distribution at time (either from before after the incident) it will never receive both observations for a single 8, rendering this easy solution impractical.
4.2 MATCHING BIASED PREDICTIONS TO REAL OBSERVATIONS
We want find parameters that minimize the translation error according distance metric (i.e., mean squared error, huber-loss, 12-loss, etc.) over samples of observations from the environment E that we obtain by executing actions a₂ (given the environment's state st):
1 arg min Eat- ((sa)))))) [d(or (p(o)))
dERR
This is a relaxed version of the original problem of translation as it only translates observations from the environment that are actually relevant to the agent. This is why it is sufficient to calculate the expectation over samples that we acquire by following policy л. Note that in particular more frequently sampled actions implicitly receive a high weight in the objective.
The key idea is to estimate a neural network model with set of weight parameter v that we can for state translation: p4o (xx))) Rs g(s)) and that converts the new state representation to the old state representation. While initial policy training (or original operation of the agent) we use observed transitions [g(s;), az g(s++)) to train neural network forward model f with parameters that captures the dynamics of the environment:
g($t+1) = (s))
(1)
in order to predict the next observation. We use this forward model predict biased targets of succeeding observations 0z+1 that we then can use together with the altered environment observation of (i.e., those observations obtained after the state encoding has changed) to train our translation model. Forward models have been proven to work well in practice but may suffer for larger planning horizons, i.c., if they are used to predict several consecutive states.
We illustrate the setup in Figure 3 We observe further preprocessed using an updated encoder. We translate of into o, using an initially randomly initialized model Mgg This gives the observation from the initial distribution over st, i.e., a noisy estimate of g(s)). Next, we use o, to select an (optimal w.r.t. the translated state/observation action a, from given policy
observation of g*{ that we might have
Now, we make use a, along two routes. First, we use the forward model fo a4) to predict a biased estimate of the next observation dt+1 (according to the old distribution g). This provides the
5

---

Under review as a conference paper at ICLR 2020
state that the agent expects to end up in when it follows the policy under the old state representation. Second, we actually execute this action to obtain the real next observation (according to the current distribution g²) from the environment: o1 9 (st),, St Ea.-r- This way obtain pair (oi+1, 0 that we can push to a replay buffer where we later sample mini-batches to train the translation model.
The intuition behind this is that for two succeeding states the translations along the routes result in the same original state representation, i.e.,
fe [pw [ (st)) od) = ps (E(st, aill
(2)
forward model (left)
via environment (right)
To further simplify we first multiply the observation provided by the environment E on the right side with the identity, i.e., y and its inverse g g-
([((((())))) (0((((((()))))))))))))
fo [ (g" (8t) ), = 1o
We assume that so ideally captures the environment dynamics, i.e., Solor, ar) = 9 [E(s,, az)]. Hence, instead of sampling the observation from the environment we sample from the forward model for
fo [ (g" (so) as] Mv [9" (fo (oc, (ot, ai)
(4)
Next, we can further reformulate the right side and eliminate Psg uu (ot) = or = (((gt)-1
(i.e., the inverse of g" applied on of gives 8t and applied on gives o.):
fo J (g* (st)
), arl = (g)) 0- '
(0 (or, ad)
(5)
= fo Iap (g* (s+) at) = 9 [g- (Jo (o..a))) at)]
(6)
+ fo M (g" (s:) at) = folor, 4t).
(7)
We can easily reformulate the left side, as (*x) = of and pw = OF:
felo., az) = fo at).
Note that we do not assume the invertibility of g" and g in practice as we never actually invert them. However, for the relaxed problem space both g" and g are at least locally invertible as this ensured by the Markov property of the state representation.
4.3 MATCHING AMBIGUOUS DYNAMICS MANIFOLDS
Training the parameters & for translator ia with labeled pairs (of, o.) is problematic when applied in practice. While the ob- jective of the training problem captures the desired goal there might be still (local) ambiguity in the dynamics over the state- space that can pose significant challenges. To exemplify this, consider one-dimensional state-space with states x E E and the underlying dynamics from Figure 4 (at state x the observation is g(x) = x and the dynamics behave according to the given function). Assume that g (xx) is now different from g(x) and we instead receive some perturbed version of g(z). Our algorithm tries to find a mapping such that relative changes between observations behave as expected, i.e., we try to find a mapping such that the expected dynamics (given by the forward model) match the actually observed dynamics.
Figure 4: Ambiguous Dynamics.
In Figure 4 the dynamics in [1; are equal to those in [5: 7] (similar for [3; 5] and 17: 9). A (surjective) mapping of pairs of observations (ot, of41) from both regions [1;3] and [5; 7] to either one those in the original encoding will result in zero (target) translation error (only the transitions of the incorrectly mapped trajectory at the (left) borders of this regions will have a non-zero translation error) as the translations suffer from a local view on the dynamics.
We address the ambiguity of dynamics with two property of the observations: instead of only optimizing 14g for the translation error we add
First, we exploit the Markov

---

Under review as a conference paper at ICLR 2020
reconstruction loss regularizer that enforces that a target representation, i.., the old encoding, can be translated back to the given representation, i.., to the new encoding. This adds a penalty if different decoder μ with parameters v that takes the
of's s are mapped to the same or. For this output or of já and decodes it to 0j R oz- We jointly train the parameters vA and ja using stochastic gradient descent and standard backpropagation using the loss function:
[ de; 1/, v) Las (ōx 1áp (ot) + B La [o] - -15 (1s (ot))
where Lg is the huber loss function, the first term is the target translation error, the second term is the reconstruction error, and / is a hyperparameter that balances the influence of the reconstruction error over the translation error.
Our second counter-measure is a lookahead extension that makes use of multi- transitions. Instead of generating translation targets from single steps through the forward model and the environment we perform X > 1 steps into the future. This gives 0.+11..., Ot+A (note that the actions ..,.... + not sampled from the policy using og but based the immediate translations o,). We can reinitiate this process of generating translation targets starting from oc+1 with À then reaching out to oc+A+1- We add the additional samples to the replay buffer as we also did single-step predictions. See Appendix A for more details.
In essence, the interaction between the forward model and the environment by following our policy provides labeled data pairs that we can use to train Hgg. If the system dynamics are constant everywhere, i.e., for states and any actions the derivative of f is constant (in each dimension), then there are infinitely many solutions for parameters v/ for which Equation 2 holds. For instance, consider a simple cartpole, where the state is described by the position of the cart (among other variables). Aside from the environment which decides if an episode ends the actual dynamics of the system are independent of the position. If we assume an infinite rail we would not be able to translate the actual position of the cart. The same holds for any environment where dynamics are invariant to particular variables of the state representation. We cannot resolve this ambiguity as the translation error of the generated targets vanishes even for an arbitrary biased translation of this variable.
EXPERIMENTAL SETUP
Environments and Baseline Policy. To evaluate our method we used OpenAl's
MountainCaz-vo environment with discrete actions E [left: noop: right] and a two- dimensional state space that is defined by the position [-1.2; 0.6] and the velocity E [-0.07; 0.07] of the car. We train dueling Double-DQN baseline policy (Hessel et al. 2018) with a single hidden layer with 64 units and ReLU activations and replay buffer size holding 50,000 transitions (using prioritized experience replay with a = 0.6, 30 = 0.4). We use a learning rate of Le - 3 with the ADAM optimizer, a batch size of 32, and apply parameter noise. We update the target network every 500 time steps and let the agent train for 500k time steps.
As experienced high variance among different runs for both the baseline policy training and our state translation we changed the exploration behavior to better focus on the algorithmic behavior. Figure 5a shows the mean episode reward over the last 100 episodes of different runs of our vanilla DQN agent using an e-greedy exploration schedule (e drops linearly from to 0.01 in 10k time steps (<2 50 episodes)). Depending on the randomness we get different performance for the runs.
Instead of using an e-greedy schedule to foster exploration we use a static = 0.02 and extend the range for the initialization of the environments, i.e., such that the states may take any value from within the state space. Hence, for the monitoring and evaluation of the training process we use two different environments: (i) a training environment (with custom initialization), and (ii) an evaluation environment (with original initialization and a greedy agent where we can clearly measure the achieved reward). Figure [5b] shows the episode reward for this custom policy evaluation. Despite of the collapse of the reward, which is well-known issue and expected for value function approximation (van Hasselt et al. 2018)), we see a much more robust performance across the runs. We will use this custom environment initialization for all our experiments.
Forward Model Architecture. To train the forward model we use the observations that the agent collects during initial policy training of our DQN baseline. We concatenate the input o and a into single vector (as we use the version with discrete actions we used a one-hot encoding) and pass into multilayer perceptron (MLP) with 3 layers having 256 units each. We use ReL.U activation
7

---

Under review as a conference paper at ICLR 2020
-100
110
Run #1
Run #1
-1200
Run #3
-120
Run
meat reward
Run #4
mean reward
Run
-140
-160
-160
-180
-180
-200
-200
episodes
episodes
(a) With e-greedy schedule.
(b) With improved initialization.
Figure 5: DQN baselines on OpenAl gym's MountainCar-va
for the hidden units and linear activation provide the output. To improve the robustness of the forward model we only predict updates, i.c., given o, and or as input the forward model predicts the output ot+1 Ot, which reduces the prediction error for unseen samples and hence results in a better generalization (Deisenroth and Rasmussen 2011). As the dimensionality of the inputs and outputs is defined by the environment we normalized and shifted both the input and output accordingly. We do not apply any regularization or dropout. The loss is calculated using a mean squared error, the network is optimized using the ADAM optimizer and learning rate le 3. We trained 150 epochs with batch size of 32.
The translation uses two sub-models: translation and a recon- struction model. The former maps the input observation of into on and the latter reconstructs of from or- In our experiments both use an MLP with a single hidden layers having 128 units. We apply tanh activations to the outputs of the hidden layers and linear activation to the output layers (i.e., for the translation and for the reconstruction). We do not use any regularization or dropout. We initialize the weights with Xavier and the biases with zeros. For the loss function we use the huber loss with 5 = 1.0 and apply B = 1.0 for the reconstruction error regularizer. We use a batch size of 1.
Translation Model Architecture.
We also use replay buffer with size 10,000 where we sample 16 mini-batches at each time step. We use target networks (i.e., copies of and == that we update after any 500 time steps. For lookaheads À we generate all the targets for À I,..., and push them to the replay memory.
The source code will be made available for download upon publication.
RESULTS
Translating the State Representation. We perturb the state representation (we tried many and they result in similar performance): instead of [pos i i ve l locittty tyy the environment provides the simple distortion [velocity:
We run an agent that takes DQN baseline policy (one from Figures 5b) and that interacts with the environment to obtain data that we process according Section 4 Figure shows the mean (and standard deviation) of the episode rewards for both 10 runs of our algorithm with X = 10 and the DQN baseline from Figure 5b) according to our policy evaluation scheme. We reach convergence after approx. 300 episodes. For any of those runs our algorithm translates the state representation such that the agent can reuse the given policy. To achieve this, we need considerably less samples that we would need to retrain the policy from scratch, At the same time method never any ground truth of both the state representation or the reward.
Note that there are also small reward drops for our translator over some episodes. We noticed that (for the evaluation environment) even small errors in the translation (of the velocity) around the starting point of the cart lead to a different action sampled from the policy: the agent gets stuck in the valley next to the base of the hill it mistakenly thinks that it has some velocity that allows it to go further up the hill. The episode then terminates at a reward of -200, i.c., after 200 time steps. Indeed, we very good and accurate translations for most of the runs even after 100 episodes in training.
8

---

Under review as a conference paper at ICLR 2020
30
CGRL JA
10)
greedl
-1.0
-500
-340
(Po0n
1:0
meen
180
-200
50
150
200
230
400
-210
50
100
130
200
250
302
402
-20
50
100
230
300
350
episodes
episodes
egisoddss
(a) GSRL (A = 10) vs. DQN.
(b) Improved init. vs. e-greedy.
(c) Different lookaheads A.
Figure 6: GSRL on the MountainCar-vo environment.
For the sake of completeness Figure shows run with an e-greedy schedule (orange) and standard evaluation setup of the environment (only one environment, e-greedy policy) next to the custom evaluation (blue). While we see the early rise later the e-greedy policy reaches the plateau en par with the improved initialized environment. However, the performance varies between the different runs due the randomness that is introduced by the exploration-exploitaiie schedule.
Lookahead Study. We also compared the influence of the lookahead parameter À on the performance of the translation. To capture the influence more clearly we did not use the forward model here as we want to leave out any effects from the accuracy of the forward model for longer prediction horizons (as for larger prediction horizons error accumulate). Instead, we here use an implementation of the actual dynamics of the poontainnCrr-v environment and whenever we query the forward model we instead initialize a fresh environment with the observation, step through the environment using dz provided by æ, and take the resulting state/observation as a prediction of the forward model. Figure (6c shows the mean reward and standard deviation over episodes for different values of (10 runs for each). In general we see that larger prediction horizons result in faster/better convergence and (although hardly visible) the variance between the runs decreases. Interestingly, the high variance targets that are generated especially from early samples (resp. from their noisy representations) do not have a negative impact on the performance. For À < 6 we observed that for 50% of the runs the translation gets stuck in local optimum where it cannot escape (this is due to the effects described in Section 4.3) The other 50% of the runs behave similar to those for larger X's.
In practice we need to adjust À carefully as there is a trade-off between convergence, accuracy the forward model, and the dynamics manifold. Hence, X should be chosen based on hyperparameter optimization. Based on this is also conceivable to use a schedule that starts with a low À (as we have high variance in the translations in the beginning) that is steadily increasing as the translations of the targets get less noisy (this would then allow to also escape local minima).
Policy Improvement. Figure 7 shows the policy on the translated input over the training of translator at the beginning, after 5k, 10k, 25k, and 40k iterations/steps. The x-axis denotes the (true) position within [-1.2; 0.6] and the y-axis denotes the (true) velocity within [-0.07: 07]. We generate the images as follows. We initialize the internal state of the environment according to each point and retrieve its perturbed version. We translate this perturbed observation using the current translator and sample the policy for an action. We draw the actions at the respective coordinates and encode them with blue circles (left), green rectangles (ncop), and orange crosses (right) The most right image shows an almost perfect translation of the perturbed observations with respect to the underlying policy, i.e., to gain momentum and if velocity is positive the agent applies right. and it is negative it applies left. Interestingly, the translations lead to the correct action for the majority of the state-space even after 10k iterations (approx. 50-60 episodes). For the remaining iterations the agents only fine-tunes the translations. Hence, depending the actual problem/cnvironmed at hand the agent might reach sufficient translation very early. The rightmost translation of the policy is almost identical to the original policy (this is why we omit the original policy).
We refrain from showing graphs that plot the error of the translated samples (both against ground truth and targets) they behave expected: the error of the model against the ground truth is initially large and decreases towards 0 while the error against the targets starts from a lower value and decreases more slowly (as targets are biased).

---

Under review as a conference paper at ICLR 2020
steps.
5,000 steps.
10,000 steps.
(d) 25,000 steps.
40,000 steps.
Figure 7: Translated policy emerging over iterations. Encoding: left (blue circles), noop (green
rectangles), right (orange crosses). The rightmost policy is almost identical to the original policy.
7 CONCLUSION
This paper proposed a novel algorithm that translates an environment representation using a given policy. To the best of our knowledge this has been the first attempt to address this problem. We sample observations from both the environment and pretrained forward model and bootstrap translator that minimizes the differences between expected and actually observed system dynamics. While we used an RL-based policy our method is agnostic to the controller, e.g., it also works with optimal control or MPC. Most importantly our approach can also be used when there no reward process and no ground truth samples available. We would also like to point out that, while we did not explicitly test for this our method also works for state representations that are generated by VAEs and hence can be applied to nigh-dimensional input (however, training the VAE representation requires careful exploration and bootstrapping). During our work we also came up with much more simplistic solution the given problem, Appendix B However, turned out that this approach has significant downsides compared to the one we presented in this paper.
environment
In future work intend to investigate the influence of stochasticity of the environment to the policy transfer process. While this is very well studied in terms of the effects on a forward model (Racanière) al. 2017) stochasticity affects our approach twice (in the training of the forward model and in the generation of targets). Another interesting direction is to apply a curiosity-based exploration scheme (Pathak al. (017) to generate more informative translation targets.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Agrawal, P., Carreira, J., and Malik, J. (2015). Learning to see by moving. In Proc. IEEE International Conference Computer Vision (ICCV), Las Condes, Chile, pages 37-45.
Andrychowicz, M., Crow, D., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., experiee experience replay. Proc. Advances in Neural Information Processing Systems 30 (NeurIPS), Long Beach, CA. USA, pages 5048-5058
Abbeel, P., and Zaremba, W. (2017). Hindsight
Bastani, O., Kim, C., and Bastani, H. (2017). Interpreting blackbox models via model extraction. CoRR, abs/1705.088 .085444
Caselles-Dupré, H., Garcia-Ortiz, M., and Filliat, D. (2018). Continual state representation learning for reinforcement learning using generative replay. In Proc. Advances in Neural Information Processing Systems 31 (NeurIPS), Montréal, Canada, pages 2450-2462.
Cully, A., Clume, J., Tarapore, D., and Mouret, J. (2015). Robots that can adapt like animals. Nature, 521(7553):503-507.
Daftry, S., Bagnell, J. A., and Hebert, M. (2016). Learning transferable policies for monocular reactive MAV control. In Proc. International Symposium Experimental Robotics (ISER), Tokyo, Japan, pages 3-11.
Deisenroth, M. and Rasmussen, C. E. (2011). Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML., pages 462-472
Finn, C., Yu, T., Fu, J., Abbeel, P., and Levine, S. (2017). Generalizing skills with semi- cumi-supervised reinforcement learning. In Proc. 5th International Conference on Learning Representations (ICLR), Toulon, France.
Greydanus, S., Koul, A., Dodge, J., and Fern, A. (2018). Visualizing and understanding atari agents. In Proceedings 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, pages 1787-1796.
Gupta, A., Devin, C., Liu, Y., Abbeel, P., and Levine, S. (2017). Learning invariant feature spaces to transfer skills with reinforcement learning. In Proc. 5th International Conference on Learning Representations (ICLR), Toulon, France.
Ha, D. and Schmidhuber, J. (2018). Recurrent world models facilitate policy evolution. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianche Bianchi, N., and Garnett, R., editors, Proc. Advances in Neural Information Processing Systems 31 (NeurIPS), Montréal, Canada, pages 2460-2462.
Harrison, J., Garg, A., Ivanovic, B., Zhu, Y., Savarese, S., Fei-Fei, L., and Pavone, M. (2017). ADAPT: zero-shot adaptive policy transfer for stochastic dynamical systems. In Proc. International Symposium on Robotics Research (ISRR), Puerto Varas, Chile.
Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., and Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. In Proceedings 32nd AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, pages 3215-3222.
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M.. Mohamed, S., and Lerchner, A. (2017a). beta-vae: Learning basic visual concepts with a constrained variational framework. In Proc. 5th International Conference on Learning Representations (ICLR), Toulon, France.
Higgins, I., Pal, A., Rusu, A. A., Matthey, L.,, Burgess, C., Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A. (2017b). DARLA: improving zero-shot transfer in reinforcement learning. In Proc. 34th International Conference on Machine Learning (ICML), Sydney, Australia, pages 1480-1490.
Jaderberg, M., Mnih. V., Czarnecki, W. M., Schaul, T., Leibo, J. Z.. Silver, D., and Kavukcuoglu, K. (2017). Reinforcement learning with unsupervised auxiliary tasks. In Proc. 5th International Conference Learning Representations (ICLR), Toulon, France.
11

---

Under review as a conference paper at ICLR 2020
Jayaraman, D. and Grauman, K. (2017). Learning image representations tied to egomotion from unlabeled video. International Journal of Computer Vision, 125(1):136-16)
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In Proc. 2nd International Conference on Learning Representations (ICLR), Banff. AB, Canada.
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., and Savarese, S. (2017). Adversarially robust policy learning: Active construction of physically-plausibe perturbations. In Proc. IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, pages 3932-3939.
Mnih, V., Kavukcuoglu, K., Silver, D.. Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattic, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg. S., and Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540)5529-335
Muratore, F., Treede, F., Gienger, M., and Peters, J. (2018). Domain randomization for simulation- based policy optimization with transferability assessment. In Proc. 2nd Annual Conference on Robot Learning (CoRL), Zürrich, Swirzerland, pages 700-713.
Ng, A. Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application reward shaping. In Proc. 16th International Conference Machine Learning (ICML), Bled, Slovenia, pages 278-287.
Parisotto, E., Ba, L. J., and Salakhutdinov, R. (2016). Actor-mimic: Deep multitask and transfer reinforcement learning. In Proc. 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self- supervised prediction. In Proc. 34th International Conference Machine Learning (ICML), Sydney, NSW, Australia, pages 2778-2787
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. (2017). Robust adversarial reinforcement learning. In Proc. 34th International Conference on Machine Learning (ICML), Sydney, NSW; Australia, pages 2817-2822
Racanière, S., Weber, T., Reichert, D. P., Buesing, L., Guez, A., Rezende, D. J., Badia, A. P., Vinyals, O., Heess, N., Li, Y., Pascanu, R., Battaglia, P. W., Hassabis, D., Silver, D., and Wierstra, D. (2017). Imaginatioo--aggmentd agents for deep reinforcement learning. In Proc. Advances in Neural Information Processing Systems 30 (NeurIPS), Long Beach, CA, USA, pages 5690-5701.
Romoff, J., Piché, A., Henderson, P., François-Laved V., and Pineau, (2018). Reward estimation for variance reduction in deep reinforcement learning. In Proc. 6th International Conference on Learning Representations (ICLR), Vancouver, BC, Canada.
Schulman, J., Levine, S., Abbeel, P., Jordan, M. I, and Moritz, P. (2015). Trust region policy optimization. In Proc. 32nd International Conference on Machine Learning (ICML), Lille, France, pages 1889-1897.
Shin, H., Lee, J. K., Kim, J., and Kim, J. (2017). Continual learning with deep generative replay. In Proc. Advances in Neural Information Processing Systems 30 (NeurlPS), Long Beach, CA, USA, pages 2990-2999.
Sutton, R. S. and Barto, A. G. (1998). Introduction Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition.
Tzeng. E., Devin, C., Hoffmmnn J.. Finn, C., Peng. X., Levine, S., Saenko, K., and Darrell, T. (2017). Towards adapting deep visuomotor representations from simulated to real environments. CoRR, abs/1511.0711 1.0
van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. (2018). Deep reinforcement learning and the deadly triad. CoRR, abs/1882.00244
12

---

Under review as a conference paper at ICLR 2020
Verma, A., Murali, V., Singh, R., Kohli, P., and Chaudhuri, S. (2018). Programmatically interpretable reinforcement learning. In Proc. 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, pages 5052-5061.
Wang, Liu, Y. and Li, (2018). Reinforcement learning with perturbed rewards. CoRR, abs/1810.01032.
Watter, M., Springenberg. J. T., Boedecker, J., and Riedmiller, M. A. (2015). Embed to control: A locally linear latent dynamics model for control from raw images. Proc. Advances in Neural Information Processing Systems 28 (NeurIPS), Montreal, Quebec, Canada, pages 2746-2754.
Yang, X. and Maciejowski, J. M. (2015). Fault tolerant control using gaussian processes and model predictive control. Applied Mathematics and Computer Science, 25(1):133-145
13

---

Under review as a conference paper at ICLR 2020
A LOOKAHEAD EXTENSION
Figure 8 sketches the idea behind multi-step predictions for our translation. The elements above the dashed line show simplified version of Figure we translate any observation of into ogi the policy gives us azi we obtain o+ from the environment and 0r41 from the forward model; and we use (oi+1: 0c+1) to train the parameters for translator J.
E
of
Oc+1
Oc+z
Oc+3
4c
Oc+1
0c+2
Figure 8: Creating translation targets through multi-step predictions.
Similar to eligibility traces (n-step prediction) known from TD-learning (Sutton and Barto 1998) can generate targets by looking À steps into the future. For instance, in the lower part in Figure 8 we take À steps through the environment and the forward model (the original idea uses one step through the forward model), which gives us ōt+1, 02-2, 0t+3 (note that the actions ag 0++1. a++2 are not sampled from the policy using o; but based on the immediate translations o,). We can reinitiate this process of generating translation targets starting from 0s+1 with À = 3 then reaching out 02+4- We add the additional samples to the replay buffer as we also did single-step predictions.
1, i.e.
B ALTERNATIVE APPROACH TO TRANSLATE STATE REPRESENTATIONS
much simpler alternative that is very similar and that we also investigated is depicted in Fig. 9 The key idea of this approach is an inverse dynamics model, i.e, model that gives the action that has been taken to make a transition from o, to O Such models have been investigated to learn features for recognition tasks (Pathak et al. 2017 Agrawal et al.) 2015 Jayaraman and Grauman 2017). We train this model on transitions (or, d2 0++1). ). i.e., we use j to translate of to og and sample policy obtain a,, which execute in the environment. We receive o' and translate it to Oct11 We use the inverse forward model f-1 on (or, 0++1) predict â, We use gradient descent to minimize the distance between a and â and regularize the reconstruction errors on of and oc+o similar to Section
Figure 9: Alternative Transcoding Scheme.
However, in practice this approach does not work well because of numerous reasons. The noisy estimates get out of μ are likely represent invalid state transitions. The inverse model cannot generalize to such unseen (and impossible) state transitions and returns a highly biased action â Even if the estimations from μ would become more accurate the inverse model will eventually introduce bias. As the inverse model cannot work on state updates (i.c., predicting only the delta between states) but instead must be trained to predict full state representation will be as accurate. This leads to a problem when it comes to generalization error for previously unseen states. Hence a, will always have some non-negligible bias that will render this method unstable. In contrast to the approach from Section we cannot use lookaheads to trade variance for bias.
14


Reviews:
[{"review_id": "SygsfAGpKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a means to adapt to new state representations during reinforcement learning. The method works by learning a translation model that translates new state representations to old state representations. The authors evaluate the method on the MountainCar environment and show that the adaptation model is more efficient than training a new policy from scratch.\n\nMy concerns with this work are as follows:\n\n- I don't find the type of changes to state representations very useful (e.g. Section 6, velocity *=2, position /= 2) nor practical. Moreover, learning a translation model to recover the old representation seems easy given this type of simple perturbations. This perturbation is fundamentally different than those used to motivate the problem (e.g. \"a robot whose sensors break down\" or \"whose sensor outputs degrade\", in the introduction).\n- The authors only evaluate with one type of simple perturbation on one environment, hence I am skeptical regarding the generalizability of this work.\n- The premise of this work is to not store old transitions (Section 1, the paragraph \"the most simplistic idea is to...\"), however this model does store old transitions because it uses prioritized experience replay. In this case there is no argument against using this data to train the translation model.\n- A comparison that is missing from the paper is to fine-tune the existing model. I believe this is a more fair comparison in terms of sample-efficiency than training from scratch.\n\n\nOther comments:\n- For the title, to call the adaptation to slightly different state representations \"policy adaptation\" is a stretch.\n- There are a lot of tangential information in the introduction on things like sample efficiency and model-free vs. model-based RL. This is distracting.\n- The paper is excessively long.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx9n-9WqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "NOTE: This paper is 10-pages long which requires a higher bar according to the guidelines.\n\nSummarize what the paper claims to do/contribute.\n* This paper claims to propose the first method to translate an environment representation to a different representation when that changes. \n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\nReject.\n* The results were not adequate. I suggest exploring more environments and more complex ones than MountainCar. \n* I do not believe this is the first attempt in translating an environment represention to a different one. Other techniques in domain adaptation have been working on this for quite some time. It might be the case that the technique proposed here is better eg to pixel-based adaptation or adaptation of features, but it would need to be shown experimentally.\n* Given the higher number of pages, i would have expected more thorough experimentation. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylWnJiXtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates a setting in which the observation function changes while the underlying environmental dynamic stays the same. \nIn order to re-use the policy which was trained on the old observation function, they propose to learn a mapping function to map the new observations to the old ones.\n\nI believe the work is interesting as generalization and reducing the sample complexity of learning policies, for example through re-use of old policies, is of high current interest.  However, I believe this paper requires more work to show the feasibility of the proposed approach. In particular:\n- The proposed method has the problem that matching is done 'locally' and without any guarantee that the mapping function will converge to the correct mapping. This is not a problem of the method itself but of the challenging problem setup. The authors discuss this and propose two approaches to alleviate this. However, no experimental evidence is shown whether and to what extend this prevents wrong local minima, as I don't believe the mountain car has such problems?\n- Evaluation is only done on the mountain-car experiment, which is not sufficient to show feasibility in general\n- The learning curves in the experiment seem highly volatile and unstable. I'm not sure why this is the case, maybe wrong hyperparameters or a bug in the code? \n- Lastly, the paper is over the recommended limit of 8 pages and could, in my opinion, made more concise at many points to shorten it and also make it easier to read.\n\nIn summary: I believe this interesting work, but requires more experiments in different environments and additional ablation studies to show the feasibility of the proposed method. Making the writing more concise would, in my opinion, not only shorten the paper but also make it easier to read. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes to improve generalization in RL environments, by addressing the scenario where the observations change even though the underlying environment dynamics do not change. The authors address this by learning an adaptation function which maps back to the original representation. The approach is empirically evaluated on the Mountain Car domain. 

The reviewers were unanimously unimpressed with the experiments, the baselines, and the results. While they agree that the problem is well-motivated, they requested additional evidence that the method works as described and that a simpler approach such as fine-tuning would not be sufficient. 

The recommendation is to reject the paper at this time.
Decision: Reject