Improving the Generalization of Visual Navigation Policies using Invariance Regularization
Authors: Michel Aractingi
Christopher Dance
Julien Perez
Tomi Silander
Venue:  (2020)
Keywords: Generalization
Deep Reinforcement Learning
Invariant Representation

Abstract: Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment. However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order to be useful. This has not been the case for agents trained with reinforcement learning (RL) algorithms. In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks. Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. 
We propose a regularization method which combines RL with supervised learning methods by adding a term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken. The results of this method, called invariance regularization, show an improvement in the generalization of policies to environments not seen during training.


Under review as a conference paper at ICLR 2020
IMPROVING THE GENERALIZATION OF VISUAL NAVIGATION POLICIES USING INVARIANCE REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment. However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order be useful. This has been the case for agents trained with reinforcement learning (RL) algorithms. In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks. Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. We propose a regularization method which combines RL with supervised learning methods by adding term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken. The results of this method, called invariance regularization, show an improvement in the generalization policies to environments seen during training.
INTRODUCTION
Learning control policies from high- ditheniimeniinall sensory input has been gaining more traction lately due to the popularity of deep reinforcement learning (DRL) Mnih al. (2015) Levine al. (2015) ;
Zhang et al.) [20] Rakelly et (2019), which enables learning the perception and control modules simultancously. However, most of the work done RL chooses to evaluate the learned policies in
the same environment in which training occurred Cobbe et al. (218).
Using the same environments to train and test agents does not give any insight into the generalization abilities of the learned policy. There could be a number of changes in the environment at test time that would degrade the agent's 's performance. Variations could appear in the visual aspects that determine the agent's 's observation, the physical structure that determines the agent's 's state and even some aspects that are related to the agent's goal (Figure Tp. For example, different observations of the same room encountered different times of the day (different lighting conditions). New obstacles could be present. Levels of a game could be different, yet playing a few levels should often be enough to figure out how to play the rest. Such variations might result in new environment where the control model that defined the training enviromment has changed. A robust policy should generalize from its experience and perform the same skills in the presence of these variations.
DRL agents have been notorious for overfitting to their training environments Cobbe et al. 200 (018), An agent could have drastically different performance on testing environments even if manages to maximize the reward during training Zhang et al. (2018a). 8a). Supervised learning algorithms have been shown to have some generalization guarantees when adding proper regularization Mohri et al.) (2018). However, these guarantees are weakened in reinforcement learning al algorithms where the source of the data is not i.i.d. order to make use of the progress of DRI. algorithms in practice we need policies that are robust to possible changes in the sensory inputs, surrounding structure and even some aspects the task.
In this paper study the notion of generalization that is appropriate for visual navigation control policies that are learned with DRL. We present: (1) study of the generalization of visual control policies to certain changes in the underlying dynamical system; (2) an alternative training method that combines DRL with supervised learning, thus using DRL to learn a controller while leveraging the generalization properties of supervised learning. In experiments use the VizDoom platform

---

Under review as a conference paper at ICLR 2020
Kempka et al. (2016) which is easily customizable and enables the generation of numerous variants of given environment.
PRELIMINARIES
Visual navigation for mobile robots combines the domains of vision and control. Navigation can be described as finding a suitable and safe path between a starting state and goal state Bonin-Font et al. (2008). Classical approaches split the problem into sequence of sub-tasks, such as map construction, localization, planning and path following BBniin-Font et al.] (2008), However, each sub-task requires some hand- engineering that specific the environment and task which makes it hard to adapt it to dif- ferent scenarios without performing some tun- ing. Deep learning approaches enable the use of highly non-linear classifiers that can adapt their inner representations to learn to robustly solve complicated tasks Goodfellow et al. (2016).
The figure shows how environments may differ in their visual aspects, like textures of the surfaces. The textures provide a differentiator for each environment, where without them the envi- ronments would have shared the same state space.
Figure I:
In this work, we use reinforcement learning al- gorithms coupled with deep learning approaches to solve the task of navigating agent towards a goal object using only its visual observations as input. The field of view of the agent is limited, i.e. it does not observe the full environment, and we do not provide explicit map of the environment to that agent.
2.1 PROBLEM STATEMENT
We model the problem of visual navigation as a partially observed Markov decision process (POMDP) (Spaan 2012 A POMDP is given by tuple
: (8, A, 2, Po),
where the set of states, AA is the set of actions and is the set of observations, all which are assumed to be finite sets. The reward function is R S A - R. The conditional transition probability mass function T SS A x S + [0. 1]. with the interpretation that T(x, a, a,s") = p(5t+1 s'|s;
s, Gr a) the probability that the next state s' given that the current state is s and that action a is taken. The conditional observation probability mass function is O: S A x Ω2 + [0, 1], with the interpretation that O a, o) p(o: 0 8, at-1 = a) is the probability of observing in state s when the last action taken was a, and we allow for a special observation probability O(8, o) = p(oo = 0/80 = 8) when in the initial state & and no action has yet been taken. Finally, Po is the initial state probability mass function, so that (s) = p(80 = 5) is the probability that the initial state is 8.
In DRL, we work with a parameterized policy Tę(h, = Pe(a, = alh h) with parame- ters E θ, giving the probability of taking action a given observation-action history hr == (oo, ag 01, ....... or)) The objective is to adjust the parameters 0 to attain high value for the discounted reward
o Σ
(P(() = R
R(s,, at)
with discount factor ? [0, 1). The expectation is over state-observation-acttoo sequences (St, Ot, a where the initial state sp is drawn from Po and other elements of such a sequence are drawn from T.6 O and Ta respectively (Sutton & Barto 1998).
Many methods for attempting to approximate optimal policies have been proposed. For instance, policy gradient methods perform gradient ascent on estimates of the expected discounted reward. In
2

---

Under review as a conference paper at ICLR 2020
this work we use the proximal policy optimization (PPO) algorithm, which arguably shows relatively robust performance on a wide range of different tasks (Schulman et al. 2017).
2.2 FORMALIZING GENERALIZZTION
As in classification, we wish to learn from a finite training set but still perform well on previously- unseen examples from a test set. To formalize this, we have a distribution 'D over POMDPs, representing multiple environments or tasks, and we sample n train POMDPs from this distribution Pi, P........ , ,,miis In the context of navigation, these POMDPs might differ in terms of their observation distributions, perhaps representing views of the same environment at different times of day or year, in terms of their transition distributions, perhaps representing maps with different geometries, or in terms of their reward distributions, perhaps corresponding to the specification of different goal states. Given this sample, we then learn policy me from finite collection of sequences from these POMDPs. In order to have a meaningful common policy across these POMDPs, we require that they have common state, action and observation spaces S, A and Ω. By analogy with the notion of generalization risk in classification (Mohri et al. 2018), we say that policy Te generalizes well if it attains a high value for the expectation of the discounted reward over the full distribution of POMDPs, which we call the discounted generalization reward, that Ep-DJp(8)) is high in some sense.The our own terminology as we did not find a semantically-equivlleee term in the literature.
state-observation-ee
not hard to see that the discounted generalization reward is actually the discounted reward Jpp (f) of a single larger POMDP pD, whose state space may however no longer be finite. To see this, let us associate a unique identifier i(P) with any POMDP sampled from D and let ID be the set of all such unique identifiers. In the large POMDP pD, the state space is the Cartesian product x Id of the original states and these unique identifiers, but the action and observation spaces are just A and Ω. The initial state distribution is obtained by first sampling a POMDP P D and then sampling Senv ~ Po from that POMDP's s initial state distribution. The initial state in the large POMDP is then the concatenation (Nenv, i()).. Thus one might succinetly state the problem of generalization in POMDPs as follows: given a distribution D over POMDPs with common state, sequences from sample of POMDPs drawn from D, choose a policy Ta that obtains a high value for the discounted reward Jpp(( (0).
action and observation spaces and access to a sample of state-observationaaci
RELATED WORK
Training in synthetic environments enables the simulation of huge amounts of experience in a span of few hours. Simulations are convenient to use when training reinforcement learning agents that are often highly sample inefficient /Sutton & Barto (1998). There is, frequently, gap between the synthetic world and the rerl-world, mainly due to the manner in which the simulators depict the real-world dynamics and visual appearances. Often, these simulated worlds capture the richness and noise of the real-world with low-fidelity Tobin et al. (2017). Many have tried to propose transfer learning techniques to bridge the reality gap in order to still make use of fast simulators for training Taylor & Stone (2099)
One popular method to bridge the reality gap is by randomizing some aspects of the training environment (Sadeghi & Levine (0166). This domain randomization technique has been shown be successful for the transfer of grasping policies from simulated training environments to the real-world Tobin al. (2017). However, the learned models resulting from that work are not control policies, but perception modules. Previous work has showed some success in transferring the perception module learned in simulation to the real world, but not the controller.
Cobbe et al.] (2018) conduct a large scale study on generalization using a new environment, that resembles an arcade game, which they call CoinRun. They experiment by training on different background images and different level structures. They test with different regularization strategies and network architectures finding that the RL agent has a surprising tendency to overfit even to large training sets. /Zhang et al.) 22088s 8a) reach similar conclusion, when learning in grid-world environments, and state that the agents have a tendency to memorize levels of the training set. Unlike Cobbe et al. (2018), however, they argue that the methods that inject stochasticity into the dynamics
3

---

Under review as a conference paper at ICLR 2020
of the system prevent memorization, such sticky actions Machado et al. (2017) and random initializations Hausknecht & Stone (2013). often do not help. In our work we are interested in generalization when navigating under partial observability unlike the fully observable CoinRun or grid-world environments.
Domain adaptation methods have also been used for simulated to real transfer. They allow models trained on a source domain generalize to a target domain. [oossmaiis al. (2017) train generative model to adapt the synthetic images of the simulator to appear like the real environment. It was shown to successfully transfer grasping policy trained in simulation the real world. However, they do not discuss whether the policy generalizes when variations happen in the target domain.
Another aspect of generalization the transfer of learned skills to solve different tasks. In other words, generalization to the goal of the trained agent g. Achieving different tasks would require the agent have the ability to maximize different reward functions, [Schaul al. (2015) consider working with value functions that contain the goal g part of the agent's 's state. They call them universal value functions. The reward will then become a function of a state-action- -gaal tuple (s, a, g) instead of classical state-action pair. In the paper, the authors present universal value function approximators (UVFA). A method that attempts to learn universal value function estimate (s, They show that UVFA'S can generalize for unseen state-goal pairs in grid-world setup.
Deep reinforcement learning has been used to train control policies. These DRL based methods generally propose to learn motor control commands from raw camera images, thus mapping pixels to commands that control the robot's 's motors Levine et al. (2015). DRL algorithms have been used for various navigation tasks such goal conditioned navigation Mirowski al. (2016); Zhu al. (2016) and mapless navigation Mirowski et al. (2018).
4 GENERALIZATION IN VISUAL CONTROL
Control policies learned from high-dimensional visual input are often brittle and lack the robustness operate in novel situations. Our main contribution to propose regularization term that can be added to the RL objective to improve the robustness of the learned policy to variations in the observations, presented in Section 4.2 However, to motivate the necessity of our proposed method we study domain randomization in Section 4.1 one of the current main practices that aims at learning a policy that generalizes well.
4.1 DOMATN RANDOMIZATION
Domain randomization is typically used to train policies that can generalize to variations and noise in the observations. It is done by training on several POMDP's that share the same S, A, S2 spaces, however they could in their observation distribution. The motivation behind domain randomization is that it is assumed to be an effective technique provide a policy that is invariant to the changes that would appear in the observations. We explore the problem of navigating the agent towards a goal object with random noise added to the agent's ss observations. If the agent is able to perform the task in an environment defined by POMDP Pi then it should still be able to perform the task in another POMDP P2. if certain features ff the environment that are specific to successfully achicving the task exist and are invariant to these variations, i.c., f(P) = f(P2).
In Section 5.I
we study domain randomization when added to RL training and the ability of resulting policies generalize in unseen POMDPs. We want to investigate if the policy does in fact overfit to the training POMDPs and whether we mitigate that overfitting by training the policies on multiple POMDPs.
4.2 INVARIANCE REGULARIZATION (PROPOSED METHOD)
In the previous sections, we discussed how overfitting to the training environment can be a big problem in RL. Furthermore, should be careful not to jump to the conclusion that training different environments will ensure policies that generalize well to new environments. It is merely an assumption that has been shown to empirically hold up when used in supervised learning context.
4

---

Under review as a conference paper at ICLR 2020
However, we show in this work that this assumption might not hold for reinforcement learning techniques. This is compatible with the findings in Cobbe et al. (2018) and ZZang et al. (2018a).
We reason that in order to generalize well, the training objective should include term that encourages policy gencralization. Therefore, putting the weight of the problem of generalizing explicitly in the objective function itself. Formally, function h of variable x: is invariant to a transformation & of (()) = h(((z)). We can deduce the same definition for the invariance of a policy и to changes in the observation given by some transformation T, T(o) ग (T())). We add this regularization penalty term to the RL objective as shown in Equation Q
N
M d(To(o,), no (T((())))
max 0
Lppo (O; TT)
Σ
(1)
NM
j=1
where Lppo is the PPO objective Schulman et al.
2017), A is the set of parameters that define the policy may d is distance function between the two conditional distributions, and À is weighting coefficient of the penalty. O = {oi, 02, ...ON) J is a sequence of N observations.
T is transformation of the observations. Given observation and a transformation that observation T where the transformation still holds the semantic context of the underlying state, but with added visual variations. We can think of the difference between observing a room with observation o and observing the same room with observation T(o) as the color of the wall for example. Therefore, let us say that we observe in POMDP P and observe (o) in POMDP PJ then f(P) = f(PT), where f(P) is the set of invariant features of the environment defined by the POMDP P. We further discuss the nature of in the experiments section. M the number of transformations of each observations.
The penalty d in Equation T resembles adding constraint on the PPO objective, where the new objective dictates that the policy should simultaneously obtain a high reward while behaving similarly for the observations o and (()). The idea is similar, in spirit, to trust region policy optimization Schulman et al. (2015) where penalty term, resembling that which would result from imposing a trust- region constraint, is added to ensure monotonic improvement of the average return with each policy update. We call this method in Eqquation jnvariance regularization (IR) since the regularization term indicates the invariance of the learned policy to transformation of given observations.
We propose ways solve the RL problem in Equation 1 The first is directly optimize the full objective by adding the penalty to the original PPO loss. The second method splits the training process to two stages of training RL first and then performing supervised learning step to minimize ((T((), < (T())) which presents an elegant form that combines reinforcement learning with supervised learning, more details of the second method is available in Appendix A
In the next section, we will discuss experiments using both methods. Before that, we will describe a study on the effectiveness of domain randomization mean reducing overfitting DRL agents.
EXPERIMENTS
In this section we present the results of two experiments. The first is about training RL with domain randomization. We discuss the ability of the learned policies to generalize to unseen environments when trained on variations of the training environment. The next part presents the results obtained when using the invariance regularization (IR) method, proposed in Section 4.2 with domain randomization and shows that it improves the success rate considerably.
We performed these experiments because we are interested in the following questions: (1) Does training on environments with random variations (as domain randomization suggests) learn repre- sentation of the invariant f with which the policy can generalize to other environments that share the same invariant features? (2) Can we find training algorithm that would empirically guarantee finding these invariant features f?
5.1 DOMAIN RANDOMIZATION
We leverage the customizability of VizDoom maps Kempka et al. (2016) with hundreds of unique textures to generate train/test scenarios. The agent is required to reach an object in order to get
5

---

Under review as a conference paper at ICLR 2020
Num training envs:
10
50
100
500
PPO
RGB
0.21 + 0.04
0.17 + 0.04
0.35 + 0.13
0.35 + 0.16
0.34 + 0.14
RGB-D
0.05 i 0.04
0.89 + 0.05
0.90 1 0.05
0.61
0.37
0.77 1 0.33
Grayscale
0.36 + 0.04
0.33 1 0.13
0.37 I 0.04
0.47 I 0.14
0.41 E 0.22
PPO-IR (split)
RGB
0.64 + 0.05
0.69 + 0.03
0.72 + 0.016
0.75 + 0.02
RGB-D
0.85 + 0.02
0.90 + 0.05
0.94 + 0.01
0.95 + 0.02
Grayscale
0.69 + 0.01
0.76 + 0.02
0.75
0.02
0.76 1 0.02
PPO-IR (full objective)
RGB
0.79 + 0.05
0.79 + 0.03
0.81
0.03
0.81
0.02
RGB-I)
0.98 + 0.01
0.97 + 0.01
0.99
0.01
0.99 + 0.01
Grayscale
0.79 + 0.03
0.79 + 0.01 0.79 - 0.02
0.80 1 0.02
Table I: Average success rate and standard deviation agents, that are trained on a different number of randomly environments, when tested on 50 test environments whose textures not seen during training. The bold values represent the algorithm that resulted in the best average success rate according an amount of training environments and an input type. We see that our method brings stability to the average results and improves generalization even when no depth is added.
reward. We train an actor-critic style agent Konda & Tsitsiklis 1999) to solve the task. The network consists of three convolutional layers and 2 fully connected layers, followed by the policy and value function estimator layers. The policy output is a four- -our-dimensional fully- sally-connected layer, where the four dimensions corresponds to four actions; move forward, turn right, turn left and do nothing. The ouput of the policy layer is a log-probability of each action. The value layer is a single unit that predicts the value function. This network architecture was proposed by Mnih et al. (015)). ReLUs used as the non-linear operations in all layers Nair & Hinton (2010). As mentioned, the PPO objective [Schulman et al. (2017) with a binary reward function (+1 if goal is reached, otherwise) and discount factor y 0.99.
optimize
We generate the variations of the training environment by changing the textures on the surfaces using the numerous textures provided by VizDoom Kempka et al.. (2016). We train agents on subset of 1, 10, 50, 100 and 500 rooms from the generated environments and test on 50 rooms with textures from hold-out set which are not seen in training. We detail this experimental setup in Appendix
We experiment with different types of visual input; RGB, RGB-D and Grayscale. The number of training iterations is fixed at 5 x 10th ensure repeatability of our experiment. The results are therefore potentially pessimistic, and in future work we would like to choose the number of iterations for each network independently so as to maximize generalization performance. The agent and the goal object are initialized at random positions in the environment at the start of each episode.
The role of depth. Adding a depth channel to the observation plays significant role in generaliza- tion. Depth is invariant to many changes in the visible spectrum of the observations. This might lead the training agent to partly find an invariance in observations in its implicit perception model. which in this case can be as simple as focusing on the depth channel only. Therefore, it was not surprising to see, in Table I, that the depth agents (RGB-D) generalize better than the agents without any depth information.
Table I shows the success rate of the PPO models with respect the number of training environments used and the input type (RGB, RGB-D). The results are averaged over 5 seeds, a standard practice in the RI. literature today. We notice the superior performance of the agent with depth than the agent without depth. The fact that the RGB agent is not able to generalize well even when exposed to numerous environments tells us that it might not be learning the invariance relating the environments. On the other hand, the RGB-D -DBDDaan agents perform well the testing environments even when the agents are only exposed to 10 random training environments.
Looking at the RGB and RGB-D experiments, the agents trained on 100 and 500 environments generalize worse on average than the ones trained on 10 and 50, which indicates that some agents might be overfitting. This is inspite of the fact that these agents are able maximize the reward in

---

Under review as a conference paper at ICLR 2020
the training set regardless of the set size. Looking at the max statistic of these results (not shown in this paper) the 100 and 500 experiments outperform the rest. However, the 100 and 500 experiments have a higher variance in the success rates of different seeds than the 10 and 50 experiments. High variance in the test results of the 100/500 RGB-D experiments shows that some seeds are able to achieve near perfect score the testing environment and others completely fail, thus there is then no empirical guarantee that RL agents will generalize when exposed to numerous environments.
The average success rate for the RGB input without the depth shows that domain randomization alone might not be an effective method to adapt the policy to variations in the observations, at least not the context of RL. In fact, shows little progress, e.g., the RGB agent exposed one environment achieves around a 20% success on the testing environments and the agents exposed to 50+ environments achieve less than 40% success. These results are consistent when running with a grayscale channel (see TTable 1).
While training by randomizing the environment did show some success in making supervised learning models generalize better, it fails to do so in RL policies. It is clear from these results, that adding random variations and relying solely on the RL objective not enough to ensure generalization. Much of the success of domain randomization in previous works Tobin et al. (2017) was reported using supervised learning. Also, the generalization abilities of machine learning algorithms have been linked to supervised learning setups. Therefore, it would make sense to adapt supervised learning techniques to regularize the models trained with DRL.
5.2 INVARIANCE REGULARIZATIO EXPERIMENTS
In this section we will discuss the results obtained from training the agent using the method proposed in Section 4.2 As mentioned in Section 4.2] we propose two methods of using the proposed IR penalty. The first is to add the PPO objective as in Equatioo I] this method is referred as (full objective) in the results. The second, which is referred to as (split) in the results, is to split the objective into two parts; RL step and supervised learning step (more details available in Appendix A). The value of À in Equation used in all IR (full objective) experiments is 1.0.
As for the nature of transformation T of the observations, we tested with the same randomly textured environments from VizDoom, that were used in the previous section, in order to be able to make fair comparisons with the pure RL and domain randomization agents. Regarding the distance penalty term d in Equation [] we did preliminary experiments with the KL divergence. L1, L2 and cross-entropy losses and the KL divergence returned the best results. Table I] shows the results for combining PPO with the IR penalty using the two proposed implementations.
Observing the split method's s results, we see that the proposed training procedure returns stable success rates that are improves more environments are added. The split version was able to outperform vanilla PPO and substantially improve the generalization especially in the cases of RGB/Grayscall inputs. Training with the full objective, however, returned the best results that outperform vanilla PPO with domain randomization and the split version of the IR algorithm. Similar to the split version, training on the full objective shows stable performance for the different inputs across different seeds.
The results, in Table I also show that the trained models, on the full objective, are achieving test success rate, with only 10 training environments, that is close if not identical the agents trained on 50, 100 and 500 environments. These results suggests that training with the full objective version of the IR algorithms does not require a large number of environments to learn the invariant features. Notice the average testing success rate is similar across the different number of training environments since the model learns the invariant features from only 10 environments and adding more environments that share the same invariant features will not make a difference. We can verify that hypothesis when looking at the RGB-D testing results in the full objective part. All agents achieve a near perfect score which we attribute to the availability of an invariant feature map in the input (the depth channel) which only the agents trained with the full objective are able to catch.
5.2.1 COMPARISONS WITH OTHER REGULARIZZTTION TECHNIQUES
Regularization has been shown to help in the generalization of supervised learning models Srivasstaal jet al.] (2014). Using regularization in supervised learning often improves the performance of the trained models test sets. However, regularization has not been frequently used in DRI. setups,
7

---

Under review as a conference paper at ICLR 2020
0.45
0.43
0.65
0.8
0.55
Batchnorm Drapout
0.35
0.33
(full)
10
50
100
500
10
30
100
500
Number Training Environments
Number of Training Environments
Figure 2: Different regularization methods tested the RGB input. (Left): The average success rate show that some of these methods achieve similar results to ours in some instances. (Right): The lower SPL of the other regularization methods relative to ours indicates some randomness in their learned policies
possibly due to the previously-common poor practise of testing and training in the same environment so there no generalization gap Cobbe et al. (2018).
We compare our method with some regularization techniques that are frequently used; dropout, batchnorm and L2. The first experiment has dropout layer added after each convolutional layer
Srivastava et al. 22014), the second has batchnorm layer added after every convolutional layer loffe & Szegedy 2015) and the last uses L2 regularization. We choose the dropout probability be 0.1 and the L2 weight to be 10-4, 4, the same values that were proposed by Cobbe et al. (2018). As in the previous setup, we train five models (different seeds) for each technique and evaluate on 50 environments whose textures are sampled from hold-out set. We report the experiments done with RGB input only as it poses a harder problem and a larger gap than RGB-D.
Figure 2 (left) shows the average success rate over 5 seeds for the four methods. We see that our proposed method the only one that steadily improving when more environments are added. The batchnorm models performed worst while dropout and L2 achieved similar success rates to the split version of our method given 50 and 500 training environments. However, the entropy of the learned policies is substantially higher when dropout and 1.2 are added the model.
We hypothesize that the high entropy policies are able to generalize by acting randomly in some instances and this makes them more robust in certain situations. We show the success weighted shortest path length (SPL) in infuigure (right). A random behavior that displays robustness (has a high success probability) would return a relatively lower SPL due to the fact that this random behavior will probably not take the shortest possible path to the goal. Details of the formulation of SPL are available Appendix C Figure (right) shows that the dropout and L2 agents have a lower SPL than the IR agents indicating that these policies with higher entropy are inefficient.
DISCUSSION AND CONCLUSIONS
We present study of the generalization capabilities of visual navigation agents trained with deep reinforcement learning algorithms. We formalize what it means to generalize in the context of POMDP. We find that the tendency of RL agent to overfit even when exposed to large training sets is quite visible. We show that using domain randomization with RL, without adding invariant features to the input such as the depth maps, is not enough generalize. In the second part, we proposed Invariance Regularization (IR), method that attempts to regularize the RI. model with supervised learning loss. improves the generalization success and displays stable performance across different seeds.
In this work, we focused our experimentation on generalization to changes in the input observation. However. it is also interesting to generalize the learned skills to different architectural designs of the environment, just as one one wishes to generalize to different levels of the game as proposed in the retro competition Nichol transformation function of the observations. One might consider an adaptive form of learned with data augmentation Cubuk et al. (2018) adversarial examples Goodfellow et al. 2015).
(2018). Another avenue of future work is to explore the appropriate
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navigation agents. CoRR, abs/ foss/880.cdl 0607006757, 2018. /RL.sttps//aaravv..or//aaa.
1807.06757
Francisco Bonin-Font Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots: A survey. Journal of Intelligent and Robotic Systems, 53:263-296, 11 2008. doi: 10.1007/
$10846-008-9235-.
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakr- ishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. CoRR, abs/ abs/1709.00857, 2017.
Karl Cobbe, Oleg
Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. CoRR, ab/18822.02341, 2018. URL htth:///rxiii
org/abs/1812 1812. 02341
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. CoRR, 0tbs/1805009500 2018. URL htw.httpp/// //axiv org/abs 1805. 09501
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//oww.deeplearrinngbooo org
lan J. Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, ab1/1241, 2015.
Matthew Hausknecht and Peter Stone. The impact of determinism learning Atari 2600 games. In AAAI Workshop on Learning for General Competency in Video Games, January 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL http::/arxxv.orgg/
2015.
abs/1502. 03167
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. ViZ.- Doom: A Doom-based Al research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pp. 341-348, Santorini, Greece, Sep 2016. IEEE. URL
http://arxiv.org/ hwtp:///bb........ 02000
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems 12, pp. 1008-1014, 1999. URL
176--atoo--riii---allo--tt
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. CoRR, abs/1504.00700 2015. URL htttt::/aaxxi...gg//
00702
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. CORR, abs/1709.06009, 2017. http::/t////rxivvcoo wwwp:iivorrgabb.. abb//
4709.06003
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning navigate in complex environments. CoRR, abs/1611.03677. 2016. URL
http://arxiv.orxt wtt.v//.ggbbbbb/...../ 03673
Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith An- derson. Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning to navigate in cities without a map. CoRR, abs/1804.00168, 2018. URL
http://a wwwoor//aaao... hwtpgg/////ai 804.0016888

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533 533, 2015.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT Press, second edition, 2018.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, pp. 807-814, USA, 2010. Omnipress. ISBN 978-1-60558-907-7. URL
http:/ /ttp://al.. acm. 3ttp://caaa....ooooaaa2230232222 31044255
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in RL. CoRR, abs/1 800.03720, 2018. URL http:://axix:
org/abs / 03720
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforreemen learning via probabilistic context variables. CoRR, abs/1903.08233 2019. URL htk.littpp//axxx..orrg////
08254
Fereshteh Sadeghi and Sergey Levine. Real single-image flight without single real image. CoRR.
abs/1611.04201, 2016. URL
org/ awwsabsss1.. 04201
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML'15, pp. 1312-1323 1320. JMLR.org, 2015. URL http: acm.
cfm? 71 d= 304511111... 3045118.3045255.
3045258
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, Abs/1502..0547,, 2015.
05477
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL htttp://arxiv.. abs/
1707. 06347]
M.T.J. Spaan. Partially observable Markov decision processes. In Reinforcement Learning, pp- 387-414. Springer, 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdi- nov. Dropout: simple way to prevent neural networks from overfitting. Journal of Ma- chine Learning Research, 15:1929-1955 1958, 2014. http:// jmlr. wtgppapers/vv5//
srivastaval aa..stavalla..ht...
html
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Mach. Learn. Res., 10:1633-1685, December 2009. ISSN 1532-4435. URL http: //dl //dl.acm..
1755839
Joshua Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world.
CoRR, abs/1703.06900 2017. URL htRphttp: /ttpttt:///rii..../// http://awww.v.oraa////aaaa/ /17033 117.0 06907
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. CoRR, abs//804.00683,, 80. URL http::/ ort.abs.rgabb..... 1804. 06893
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SOLAR: deep structured latent representations for basel-based reinforcement learning. CoRR, abs/1 09100, 2018b. Utl.ftttpp//arriv.orr//aaa//
09105
10

---

Under review as a conference paper at ICLR 2020
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation indoor scenes using deep reinforcement learning.
CoRR, 0bs/1600.05144, 2016. URLhttp: / aww.v.org/ab..ao.oooo 1609. 05143

---

Under review as a conference paper at ICLR 2020
APPENDIX
IR SPLIT VERSION
Algorithm 1 RL with iterative supervision
Initialize kn, k2, f,r Ti=(I....N] env while not converged do for =
ki do // Train To on env on the RL objective 0, <- maxØ Lo To._.)
end for for j = li2 do
// Train env and ((env)
Sample {of=e, TBz, (of=e)}
Generate for (TTeeen)) Tee, (0g ( ) )
0, f ming d(TaN, (Deno) To, (Ti(Oeno));
end for end while return We
The first part consists of training RL on the observations of the original training environment, while the second part can be seen as supervised learning objective on the transformed observations, as shown in Algorithm| [
The first step trains RL on one environment and then use the actions that the trained policy would have taken that environment tune the model with supervised learning on the textured environments. In the reported experiments using the split version, the model is trained with one iteration of the algorithm. Therefore, the training process has two stages, train RL then train with supervised learning setup, without iterating between both.
B EXPERIMENTAL SETUP
As stated in Section 5.1 we run training on a subset of I, 10, 50, 100 and 500 rooms where the surfaces in each room are sampled from the variety of textures available Vizdoom. The resulting policies are tested on 50 rooms with textures from hold-out set which are not seen in training. During training we run several agents in parallel to quickly collect inpervation-action-eewwee multiple environments. Another advantage of this parallelization is the ability to run each agent on a variation of the training environment. Due to hardware limitations, we cannot run one agent for each environment, at least not when we have large number of training environments, i.e., 100 or 500. Therefore, each agent samples one environment from the training set and runs on for some n episodes before sampling another one (n = 25 episodes).
data in
C SUCCESS WEIGHTED SHORTEST PATH LENGTH (SPL)
SPL was proposed by Anderson et al. (2018) as way of measuring the navigation agents success rates while taking into account the time it takes agents to succeed] II
N
SPL
where is the number of runs, S, the binary indicator of the success of episode i, 1, is the length of the shortest possible path and p; is the length of the path taken by the agent.
'The SPL formula suggested by Anderson al. 2018) contains a max 4) at the denominator instead of Pr. written in in(Equation We removed this factor the redundant since p: I.
12


Reviews:
[{"review_id": "HkgGADihKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to achieve generalization under appearance change by explicitly optimizing for invariance under the visual transformations present in the training data, by leveraging privileged information about which states should be treated identically. Results demonstrate improvements over both a separated RL+supervised version and the naive domain randomization approach.\n\nI have settled on a weak rejection, because although the results are reasonable, I feel that the approach is not particularly novel, requires significant access to privileged information, and is not sufficiently situated in the literature.\n\nThis is essentially a generalized visual place recognition problem, in which representations are sought that are invariant to visual changes in the environment such as texture and lighting. This is an extremely well-studied problem, and many important references are missing in the discussion of related work [1]. This problem has also been tackled with deep learning approaches [2,3,4] and so it is somewhat surprising that none of this work is referenced in a discussion of visual invariance.\n\nThe approach of using privileged information (image pairs that are the same place with different appearance) to train representations to be similar [3,4] is a very straightforward and not particularly novel auxiliary loss, and it is not surprising that it would improve generalization here. A more interesting and important direction is how to identify these points of invariance without supervised labels.\n\nThe results in Fig. 2 look very noisy and don't have error bars. Are these results statistically significant?\nThis paper could be improved with clearer experimental results showing some kind of monotonic improvement with the number of environments in Fig. 2, perhaps this could be achieved by running more experiments.\n\nThe paper would also be improved by considering unsupervised ways to provide the labeled pairs required for the method. For example, what about generating random transformations of the observations instead of using privileged information? The authors mention an adversarial approach; results from a technique like this would significantly improve my rating.\n\n[1] Lowry, Stephanie, et al. \"Visual place recognition: A survey.\" IEEE Transactions on Robotics 32.1 (2015): 1-19.\n[2] Chen, Zetao, et al. \"Deep learning features at scale for visual place recognition.\" 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.\n[3] Arandjelovic, Relja, et al. \"NetVLAD: CNN architecture for weakly supervised place recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n[4] Carlevaris-Bianco, Nicholas, and Ryan M. Eustice. \"Learning visual feature descriptors for dynamic lighting conditions.\" 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2014.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJcaN-MKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe goal of the paper is to improve generalization of RL agents to a set of known transformations of the observation. \nThe authors propose to explicitly include a term into the PPO loss function that incentivizes invariance to transformations of the environment which should not change the policy, in their case changing textures of walls. \n\nThe idea itself is straightforward but worth exploring, although it does make the fairly strong assumption that one has access to or knowledge of the transformation function of the environment that can be applied to the observations. \nOverall, the paper is clearly written and easy to read. \n\nI'm currently recommending rejection based on the experimental evaluation as I don't believe that, in their current form, they sufficiently show the utility of the method (see detailed comments below).\nHowever, I'm happy to change my rating if some or all of my comments and questions are addressed. \n\nMy main concerns are with the experimental evaluation:\n- I would encourage evaluation on a second environment. In particular, the CoinRun environment which is cited in the paper would in my opinion make an excellent second evaluation setting as it also allows randomization of the texture, so the setup is fairly similar to the setup here and, importantly, it allows comparison to published results. \n-I am surprised that training on _more_ environments reduces the performance for the baseline agent. My suspicion is that training on 100 or 500 task is, at first, much harder to learn for the agent as it sees individual levels much more rarely. With the number of training steps fixed, I think it is possible that those agents haven't finished training yet. It would be good to include the training curves for the agents or at the very least their final performance on the training set (for example in the appendix).\n- Another question that was not clear to me from the text: When training the IR objective, is the transformation function restricted to produce observations from the set of limited environments? I.e. when training on 10 envs, does the transformation function produce only observations from those 10 or potentially from all 500 in the 'maximum' training set? Having access to all 500 would explain why the success rate for IR is constant across all number of training set sizes.\n- I think figure 2 needs more random seeds and needs to show the standard deviation across them, as it might be fairly large. \n\nEdit:\nThank you for your response and your comments. \n\nOverall I think this is interesting and very promising work. Consequently, I am raising my score to \"weak reject\". Below, I discuss several points how I think the paper could be improved. \n\nAdditional environment: I understand that Coinrun requires a significant amount of compute and that's not easily affordable for everyone. However, I do strongly believe that this work would profit a lot from additional experiments. For future versions of the paper, I do think Coinrun would be very interesting, but alternatively, maybe the Multiroom environment used in [1] might be an easier alternative? (With transformations being symmetry transformations?). Just an idea. \n\nMore environments leading to a deterioration of performance: Unfortunately, I still don't understand how more data can lead to more overfitting. I believe it would be useful to investigate this further, either to avoid an unfair comparison to PPO if there's some training instability that can be easily avoided. Or alternatively, if this result holds, this could be a very useful insight as well if the authors can explain why this is the case. \n\nGood and constant performance of IR: I still find it surprising how constant the IR performance is independent of how many levels are being used. Providing more data/ablation studies to better support this result and provide further insight into IR would strengthen the paper a lot. Additionally/alternatively: Submitting the code for reviews to inspect would help here as well. \n\n>> \"We hypothesize that adding this penalty encourages the model to find the invariance in the environment and the model is able to deduce the shared invariance within these environments and reach a certain performance with this added information only by observing a few .  \n\nIn particular, if you can show some further support for this claim, I think this would strengthen the paper a lot. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BklkqyTxFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Claims: They formalize generalization in a reinforcement learning setting by defining a distribution over POMDPs from which they can sample multiple POMDPs for training. The authors posit that using domain randomization causes overfitting to those domains, and propose an invariance regularization to add to the training objective that prevents this overfitting.\n\nDecision: Weak reject. While I believe the problem setting is important and the framework interesting to evaluate RL in a multi-POMDP or MDP setting and assuming a distribution over those multiple MDPs, the assumptions made in order to perform this invariance regularization are too strong. In Equation 1, in order to add the regularization penalty term, it requires that they have access to the transformation \\mathcal{T} or to observations that correspond across the MDPs. If the authors can explain why this is a reasonable assumption to make or explain how this assumption can be relaxed, I will consider changing my score. It is also unclear from the writing how this assumption is enforced. Is the initial state always the same across environments, so the agent is always performing the same actions to stay sync-ed across environments?\n\nCan you give some intuition as to why the agents trained on 100 and 500 environments are generalizing worse compared to ones that are trained on fewer environments? It seems one would expect the opposite to be true. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers recommend rejecting the submission. There is no basis for acceptance.
Decision: Reject