Domain Invariant Q-Learning for model-free robust continuous control under visual distractions
Authors: Tom Dupuis
Jaonary Rabarisoa
Quoc Cuong PHAM
David Filliat
Venue: Submitted to ICLR 2023 (2023)

Abstract: End-to-end reinforcement learning on images showed significant performance progress in the recent years, especially with regularization to value estimation brought by data augmentation \citep{yarats2020image}. At the same time, domain randomization and representation learning helped push the limits of these algorithms in visually diverse environments, full of distractors and spurious noise, making RL more robust to unrelated visual features. We present DIQL, a method that combines risk invariant regularization and domain randomization to reduce out-of-distribution generalization gap for temporal-difference learning. In this work, we draw a link by framing domain randomization as a richer extension of data augmentation to RL and support its generalized use. Our model-free approach improve baselines performances without the need of additional representation learning objectives and with limited additional computational cost. We show that DIQL outperforms existing methods on complex visuo-motor control environment with high visual perturbation. In particular, our approach achieves state-of the-art performance on the Distracting Control Suite benchmark, where we evaluate the robustness to a number of visual perturbators, as well as OOD generalization and extrapolation capabilities.

Under review as a conference paper at ICLR 2023
DOMAIN INVARIANT Q-LEARNING FOR MODEL-FREE ROBUST CONTINUOUS CONTROL UNDER VISUAL DIS- TRACTIONS
Anonymous authors Paper under double-bling review
ABSTRACT
End-to-end reinforcement learning on images showed significant performance progress in the recent years, especially with regularization to value estimation brought by data augmentation (Yarats et al.) 2020). At the same time, domain randomization and representation learning helped push the limits of these algo- rithms in visually diverse environments, full of distractors and spurious noise, making RL more robust to unrelated visual features. We present DIQL., method that combines risk invariant regularization and domain randomization to reduce -uttof-distribution (OOD) generalization gap for temporal-difference learning. In this work, we draw link by framing domain randomization as a richer exten- sion of data augmentation to RL and support its generalized use. Our model-free approach improve baselines performances without the need of additional repre- sentation learning objectives and with limited additional computational cost. We show that DIQI, outperforms existing methods on complex visuo-motor control environment with high visual perturbation. In particular, our approach achieves state-of the-art performance on the Distracting Control Suite benchmark, where we evaluate the robustness to number of visual perturbators, as well as OOD generalization and extrapolation capabilities.
INTRODUCTION
Data augmentation is used extensively in computer vision models for regularization. One can not imagine reaching state-of-the-art performance on usual benchmarks without using a careful combi- nation of transformation on images. Yet reinforcement learning lags behind on the usage these techniques.
First, this stems from the high variance reinforcement learning suffers during training. This es- pecially true for off- off-policy algorithms such as Q-learning (Watkins & Dayan 1992; Mnih et al.) 2013). where noisy Q values caused by uncertainty induced an overestimation bias which renders training extremely difficult. A number of methods directly tackle the problem of overestimation with algorithmic or architectural changes to the value function (Van Hasselt et al. 2016)] Wang et al.) 2016 Bellemare et al. 2017, Kumar al., (02I]. Regularizing the value estimation with light data augmentation is another successful approach (Yarats et al. 2020 Laskin et al., 2020b) but exten-
sive data augmentation in reinforcement means adding even more noise and can lead difficult or
unstable training (Hansen et al.) 2021).
Secondly, standard computer vision tasks mostly focus on extracting high level semantic information from images or videos. Because classifying the content of an image is a substantially high level task, the class label resilient lot of intense visual transformation of the image (e.g. geometric transformations, color distortion, kernel filtering, information deletion, mixing). Features such as exact position, relative organization and textures of entities the image is usually not predictive of the class label and data augmentation pipelines take advantage of it. From causal perspective, data augmentation performs interventions on the "style" variable which is not linked to the class label in the causal graph. It happens that for classification tasks the dimension of the style variable is much bigger than in visuo-motor control tasks where reinforcement learning is involved. Intuitively, change lot of factors of variation in the visual aspect of particular object without

---

Under review as a conference paper at ICLR 2023
Extrapolation domain
Interpolation douain
Qu
Qo
o
D
DIQL
D2
Q
Qi
c+1
Figure Domain Invariant Q-Learning. For training, DIQL uses 2 visually different domains (Interpolation domains) based on the same inner state of the environment to learn Q-function that is invariant to spurious visual features. DIQL promotes risk extrapolation and prevents drastic collapse of the accuracy of the Q-function in out-of-distribution settings (Extrapolation domains).
being not recognizable anymore: we can still recognize car on the street with very sparse and highly perturbed visual cues. With visuo-motor control, these factors of variation (dimensionalitt of the style variable) are in fewer number and less obvious. In particular, geometric deformations or occlusions could destroy crucial information for control such as relatives distances of objects in the image. Simple data augmentation, under the form of random shift, proved to be crucial for boosting RL performance (Yarats et al., 2020 2021 Laskin et al., 2020b a Haasen et al. 2020) and is now used as a baseline in of methodologies. Though, it remains less clear which combination of image transformations optimal for reinforcement learning (Hansen et al. 2021
Raileanu et al. 2021).
related technique, Domain randomization (Tobin et al. 2017) ) was introduced in robotics to close the gap from simulation to real world, by randomizing dynamics and components in simulation. We
argue in this paper that domain randomization a more general case data augmentation more suited for reinforcement learning that allows for finer control over visual factors of variation by directly changing the hidden state of the system in simulation. Contrary to data augmentation in general, domain randomization directly acts on the causal factors instead of adding uncorrelated noise to the observation, which could destroy useful information.
Starting from this observation, we present Domain Invariant Q-Learning (DIQL) for robust visuo- motor control under visual distractions. We show that DIQL is able to efficiently train an agent with visual generalization capabilities without losing on convergence speed and asymptotic performance on the original task. In particular, we derive domain-invariant lemporal-difference loss combining domain randomization and risk extrapolation [F. We show that domain randomization can be better integrated in reinforcement learning than is classically done in order to improve performance at very low cost. Our main contributions are:
novel methodology for robust visuomotor control based on temporal-difference learning images using invariance principles,
empirical results on the Distracting Control Suite benchmark (Stone et al. 202 with state- of-the-art results raw training performance and out-of-distributtin (OOD) generalization under the hardest setting of dynamic distractions
2 PRELIMINARIES
Value-based reinforcement learning We define the MDP M AS where S is the set of states. A the set of actions, P the transition probability function, R the reward function and a discounting factor for future rewards. We also define the transition containing state, action, reward and next state at timestep T (St, A ri, S411). Reinforcement learning aims maximize
2

---

Under review as a conference paper at ICLR 2023
the total reward received by the agent. Mathematically, it corresponds to finding policy that maximizes the (discounted) expected return ET~ 2'r S(+1))
We are interested here in value-based reinforcement learning and in particular deep Q-learning, which approximates the state-action value function with deep neural network to handle high- dimensional observations. The Q-network is optimized with gradient by minimizing the following temporal-differeen loss:
JQ( = Bls. (CQ a rr, St+))
(1)
where Lo ar, rt, 8141) (St, ar) Tt + max (g(s++1,
0 follow the trainable weights with an exponential moving average: 0 = 70 + (1 -]0 with the smoothing factor.
Risk minimization and OOD generalization The empirical risk associated to a model with pa- rameters 0 trained with a loss function / with data X situated on a domain D is the expected Ex~pl((X:6)] When working on out-of-
loss value of the model over that domain (D:0)) distribution generalization and domain shift problems, we sometimes have multiple training do- mains. The true joint distribution P(X, Y) of the data X and labels/predictions Y is often unknown and the simplest solution we can imagine to improve generalization is to minimize the averaged empirical risk above across data points and domains. Empirical risk minimization (ERM) is conser- vative and averages the training risk over the training distribution, without making further assump- tions on out-of- disttribbttion behavior. Formally, if the training dataset is composed into multiple domains D (Di he,.' then using empirical risk minimization to optimize parameters 0 of the model equivalent to ming DD E, R(D 0) where R(D', ,0) is the risk or cost function computed with data of domain D'. However, ERM is naïve the minimizer of the sum of the risks is not necessarily& minimizer of each subdomains. We can easily imagine scenario where the joint ob- jective of ERM is optimized even though the performance on one or more of the training domains is suboptimal. A fortiori, there is no guarantee of generalization to out-of -it-of--disributiin domains at test time. Multiple improvements over ERM have been made, through causal inference (Peters et al, 2016) and invariant prediction (Arjovsky et al., 2019).
more recent approach, V-REx (Kueger et al., 2021), directly tackles the issue of OOD general- ization which is overlooked by previous methods. The problem is formulated as robust optimization and the objective to minimize the risk ROOD maxp<ep R(D*)). To practically mínimize this worst-case risk, the empirical risk is complemented with penalty based on the variance between training risks:
RV-REa (0) = Σ R(D', 0) + (Var((
(2)
This principle of invariance of risks promotes risk extrapolation and provably enables causal discov- ery.
METHOD
Our method builds upon deep Q-learning by regularizing and enriching the temporal-differnnce emror with multi-domain and cross-domain risk minimization, taking fully advantage of domain randomization.
3.1 ON THE USE OF DOMAIN RANDOMIZATION
Data augmentation is transformation applied a posteriori to an image. The transformation itself is most of the time completely independent from the image and agnostic to its content. Domain ran- domization on the other hand is acting directly inside an environment before an image is produced. Its subtlety lies in its ability to change appearances of scenes without changing its semantic content and inner state such as object classes and relative positions. Because our objective is to have an agent robust to visual perturbations, we will restrict the definition of domain randomization only to visual changes which do not causally change the inner state of the environment, usually called style variables or spurious features, as they are irrelevant for control.
3

---

Under review as a conference paper at ICLR 2023
Let's consider the causal graphical model between the environment and the Q-function with data aug- mentation domain randomization in Figure We are dealing with image observation, thus the true inner state (s) of the environment is hid- den. S; is the controllable part of the state whereas 5, is control-irrelevant. We can think of S, as the part of the state containing the causes of spurious visual features in the observation o₂ rendered by the environment. Q: denotes the true Q-value as- sociated with 8, and a, and not affected by 8,. The difference between data augmentation and do- main randomization is where the perturbation is in- jected. While data augmentation adds uncorrelated noise XD directly to the observation, (visual) do- main randomization XDD directly acts inde- pendently from S, before rendering o,, From causal perspective, domain randomization performs do-interventions on the style variable Sg. This can help disentangling of the inner states S; and Sy. In particular, this help the Q-function performing causal discovery by uncovering the effect of & on
$1
at
XDA
St
S
XDR
Then, by encouraging invariance to visual do- main randomization, the Q-function forced to ig- nore S, make predictions and has to extract 8; and rely entirely on it. We call the resulting model domain-invaricn
Figure 2: Causal Graphical Model of the en- vironment and Q-function. (Top): data aug- mentation. (Bottom): domain randomization. Grey variables are hidden, white are visible. Dashed arrows indicate causality through in- ference by the policy and Q-value modules of RL, full arrows are invisible causal relation- ships.
Q- D-function. This is not possi- ble with data augmentation, as the noise is applied after rendering. which prevents disentagnling of the inner states. This motivates us to use domain ran- domization and to perform interventions with it each step of learning.
3.2 DOMAIN INVARIANT Q-LEARNING
To maximize the utility of domain randomization and hopefully learn an invariant Q-function, we constantly generate 2 different views from the same scene each time step the environment (Figure Formally, we can define the two views as two instances of the same scene where the style variable belongs to different domains, where of belongs D and of D2 Because accuracy of Q-values estimation is the main driver of performance for value-based based model-free algorithms, our goal is to regularize the Q-function to make it invariant to visual perturbations irrelevant to reward and control.
We first regularize the Q-function by averaging the TD error over the two training domain previously defined, 2 and D2 by averaging the temporal-difference error over the two observations:
LERM Lo(o), ar, Tt. 1.41) + Lo a,, of+1)
where Lg is defined in Equation I Action and rewards passed to the loss function are the same for both domains, as the unknown underlying state.
We add domain translation term to the loss to further encourage domain invariance for the Q- function. For each additional TD error term, bootstrapped values are computed using the next ob- servation 0:+1 from the opposite domain of the current observation O,:
CDr Tor=Looop Lo(o) a,, rt. o) Lo a, Tt, o
As mentioned in section only averaging training risks over multiple domain is not enough to increase OOD generalization. In order to increase generalization to unseen domains, we regularize the losses with variance minimization term that enforces invariance of training risks (here TD
4

---

Under review as a conference paper at ICLR 2023
IQM
Mean
Median
DIOLE DIOL-V DIOL-M ERM SAC
0.42
0.48
0.54
0.60
0.48 0.52
0.56
0.44
0.48 0.52
0.56
Normalized Score
Figure Evaluation metrics aggregated over all benchmarks, tasks, 21 episodes and 4 seeds at the end of training. Black stripes and colored bars represent aggregates and confidence interval computed with stratified bootstrap sampling using the rrliable library (Agarwal et al. 2021b).
losses). Motivated by Krueger al. (202I), this term prevents one loss from dominating the others by "flattening" the risk landscape across domains.
Cvar = Var (LLold, a ar, rz,
(4t1 (.))E(12)))
(5)
Putting all three losses together gives us our DIQL objective to train an invariant Q-function:
Je DIQL (0) [o) o141 of 02 -B [CERM + LpT + @L
(6)
where controls invariance risks. 3 oo would force all risks to equal but prevent minimization, while 3 recovers ERM applied to multiple domain to the Q-function.
Coming back to the causal inference framework, Krueger et al. (222)) show that under three specific hypothesis, equalizing training risks between domains is equivalent to performing causal discovery. The three hypothesis are the following: causes the target variable are observed, training domains corresponds to interventions on X and the Bayes error rate of the model is the same over all training points. The first hypothesis is verified: Q, has s, and a, as parents. Although s, is hidden, all the information in S, is contained in o, due the MDP framework. Thus, causes of Qr are observable. Second hypothesis is true by definition of domain randomization in our graphical model: domain randomization performs interventions on 5.. We perform random visual intervention, yet the con- trollable state of the environment is deterministic. In practice, we ensure that visual perturbations do not destroy useful information the image: all of s, is contained in og for all states. Because we are using a single Q-network with fixed capacity on the training domains, we deduce that the irreducible error, ie the Bayes error rate, is constant across all training samples. This validates the third hypothesis. Per Krueger et al. (2021 ) this means that equalizing training losses between each temporal-differenn term in DIQL allows the Q-function to perform causal discovery. Thus, the Q-function can learn the hidden causes of the real Q-value and disentangle s; from 5y; we have iinaaiiniivaiantt Q Q-function by definition.
4 EXPERIMENTS
4.1 BENCHMARK AND BASELINES
We aim to evaluate the training stability, robustness and out-of-distribution generalization of our methodology with continuous control from images. We first present the Distracting Control Suite (DCS) that will serve to train our agent and benchmark for robustness to visual distractions. We detail training procedures, baselines that will be used for comparison and evaluation protocol to quantify OOD generalization performance. Finally, we showcase aggregated and detailed evaluation measures all six tasks of DCS, and present some ablations.
Environment. We use Distracting Control Suite (Stone et al. 2021 for our experiments. DCS a variant of the Deepmind Control Suite where visual distractions are dynamically added to the rendered observations. The perturbations consists in the following non-exclusive dimension of vari- ations: color randomization physical bodies, background randomization with videos and camera movement. Distractions dynamic, temporally consistent and continuous. Colors of bodies
5

---

Under review as a conference paper at ICLR 2023
continuously changing at each time step. The background is displaying frame by frame randomly selected video from the DAVIS dataset, which is played forward then backward to avoid discontinu- ities. The camera's 's orientation is rotating with random angle at each step while keeping the agent in the field of view. As descrited in part 3 we produce two observations at each time step of the environment with two different visually randomized observations. In practice, instead of generat- ing two randomized views, with generate a clean observation from the original environment and noisy observation using a combination of camera, color and background distractions. This reduces computational burden and helps training by reducing variance in the data, which helps training as shown by Hansen et al. (2021). We define 5 variants of the environment, based on different com- binations of distraction intensities and parameters, We use DCS with all three types of distractions applied together and varying the intensity value of the distractions. We refer to Easy, Medium and Hard benchmarks for 0.1, 0.2 and 0.3 intensity respectively. Furthermore, we define Very Easy benchmark as DCS with intensity 0.1 and without camera movement. Clean refers to the clean environment with all distractions disabled (see example images in the appendix).
Training. We compare our methodology with sev- eral baselines. SAC refers to Soft Actor-Critic ap- plied with only access with one observation at each time step. This corresponds to the usual implemen- tation of SAC on top of a visually randomized en- vironment. ERM average policy and Q-value losses of SAC over two different observations of the same step at each step, while keeping the action and re- ward fixed. DIQL refers to our methodology, which is implemented with SAC and described in section Main results with DIQL, ERM and SAC are ob- tained by training on the Easy benchmark. We might refer to DIQL-VE, DIQL-E or DIQL-M which de- scribe DIQL trained with the Very Easy, Easy and Medium benchmarks respectively to compare them. Otherwise, ERM and SAC are always trained with the Easy domain. For each experience, we train 4 random seeds for over 500k steps of gradient descent with Adam optimizer. We use the SAC implementa- tion of ACME (Hoffman et al. (020) in Jax (Brad-) bury al., 2018) for faster training. Both policy and Q-network are implemented with convolutional stack followed by a MLP. The policy and Q-network only share weights of the convolution stack to compute lower-dimensional visual features.
SAC
ERM
DIQL-M
DIQLIVE
DIQL-E
1.00
Fraction of runs with score
0.75
0.50
0.25
3
0.00
0.2
Normalized Score
Performance profile aggregated over all benchmarks, tasks, 21 episodes and 4 seeds at the end of training. Shaded areas represent stratified bootstrap confidence in- tervals.
Figure 4:
Evaluation. We evaluate our models during and at the end of training over the 5 benchmarks defined above: Clean, Very Easy, Easy, Medium and Hard. We run evaluation episode on each benchmark every 50000 steps of training and accumulate the return over each episode. Episodic return adds up to 1000 but normalized the score. We systematically use the rlliable library (Agarwal et al., 2021b) to evaluate our models, using stratified boostrap over seeds and/or tasks on the benchmark to provide robust evaluation metrics. In particular, (IQM) as a robust replacement to the mean while being more sample efficient than the median. To properly test for generalization, the evaluation environment uses different dataset of videos for the background even when training and evaluation have the same distraction difficulty.
the inter-quartile mean
4.2 RESULTS
Aggregated metrics. As shown in figure 3 DIQL largely outperforms all baselines when aggre- gating results over all benchmarks. One the one hand, ERM averages both policy and Q-function losses over both training domains but does not improve over plain SAC trained directly on a sin- gle distracting domain. On the other hand, DIQL-E completely outperforms these baselines for the same computational costs, thanks to the efficient use of domain randomization to enforce vavalue invariance. DIQL-EE improves IQM by roughly 30 % over the baseline and is significantly above on

---

Under review as a conference paper at ICLR 2023
DIQL-E
ERN-E
SAC-E
Walker
Finger spin
Normeliee IOM net setum
Normelized low set
Normaltaed LOW Rot
9.8
0.4
22
300
900
930
330
230
300
400
500
100
200
410
500
Stops
Stops
oasy
Corppote Swingup
Nomaatood LOM Razur
Normalizee JCM Return
0.0
0.6
Bormalizee
0.4
0.0
400 530
Steps
Steps
Figure 5: Sample efficiency plots for each task the Easy evaluation benchmark. Models evaluated every 50k steps and scores are aggregated over 10 episodes and 4 seeds. Shaded areas represent stratified bootstrapped confidence intervals.
0.9
DIOLE
0.9
DICL
ERM-E
0.8
DICLEE
Norma normalized (OM retumm
0.7-
0.6
Normainne (QM
0.5
0.4-
0.4
0.3-
0.1
0.1
Clear
Vary Easy
Easy
Medium
Hard
Clear
Vary Easy
Easy
Modium
Hard
Evalustion domain
Evaluation domaan
(a)
(b)
Figure 6: Evaluation the end of training of DIQL variants and baselines across all benchmark difficulties. Points are normalized IQM episodic returns, and vertical bars are bootstrapped confi- dence intervals. (a): Comparison of DIQL, ERM and SAC results trained on Easy. (b): Comparison of DIQL results trained on Very Easy, Easy or Medium.
performance profile of figure Figure shows sample efficiency curves each six tasks on the Easy benchmark (the other benchmarks are in the appendix). We observe that DIQL especially improve performance more complex tasks like Walker walk and Cheetah run. They both have high-dimenssonal action space, which hints at potential successful transfer of our method to harder tasks in robotics. However, all tasks are overall affected by the difficulty of the benchmark and its dynamic aspect compared to clean environment. In particular, Cartpole Swingup which should be solved easily actually shows poor performance because of the original action repeat value of 8; in between consecutive frames seen by the agent, camera and background will have moved by frames which breaks the continuity of distractions and make the task particularly hard.
OOD generalization. We evaluate generalization capabilities of the agent in figure 6 We seek robustness to out-of-distribution shifts and look for extrapolation in the evaluation benchmark. In this setting, DIQL trained on each Very Easy, Easy and Medium benchmarks and evaluated on all five benchmarks. Because the evaluation benchmarks are ordered by distraction intensity, each evaluation domain included in the next domain along the axis. Evaluation domains easier or equal than the training domain considered in-distribution: they interpolate between the Clean domain and the training domain. Domains strictly above the training domain are out-of nut-of-distribution they require the model extrapolate. Figure 6a shows final normalized IQM aggregated over tasks

---

Under review as a conference paper at ICLR 2023
Walker
Bell-in-cup catch
1.0
Normalized (QM return
0.8-
0.8
0.6-
Vary Sasy
Hormalizedd
0.0
Easy
Medium
0.4
0.4
Uleen Very Easy
0.2-
Easy
0.2
Medium
0.0
Hand
10a1
10
10-1
30
10
13a
10
10-1
10
10-
Figure 7: Evaluation score on all benchmarks for different values. Shaded areas are bootstrapped confidence intervals.
for each testing benchmark and all baselines and variants presented. ERM increases total return in the Clean benchmark. However, it fails to improve generalization to all distracting benchmarks and degrades results compared to plain SAC. Pitfalls of empirical risk minimization are to blame: much easier optimize the RL objective over with non-noisy observations which leads to most capacity being dedicated to overfitting on the Clean domain. DIQL is able to correct this problem and redistribute the model capacity between both training environment by enforcing invariance of risks with the penalty term. Performance is lifted on all distracting benchmark result, but is also decreased on the Clean domain. Overall, the risk landscape across domains flattens, phenomenon already observed and described in the seminal paper on risk extrapolation (Krueger et al. 2021).
This flattening of risk landscape is also observed by changing the training distribution used with DIQL. We trained DIQL on three different benchmark difficulties: Very Easy, Easy and Medium. We evaluate on all benchmarks and show results Figure/6b) Training on harder domains helps gen- eralization to harder benchmarks as expected, but reduce overall performance: the network capacity and training time remained constant while the training task became harder. Even though Figure [6a] shows clear improvement by DIQL in the onttof-distribution. of performance when going from in-distribution to OOD evaluation regime. particular, DIQL-VE score drops by more than 50 % between Very Easy and Easy evaluations while DIQL-E score drops less than 40 % between Easy and Medium. DIQL-M only drops by rougly 30% between Medium and Hard but overall performance is lower.
in Figure 6b an expected drop
The risk extrapolation term controlled by 3 plays a significant role in flattening the risk space across domains shown in Krueger al. (2021) In particular, higher 3 forces equalization of training losses of DIQL which prevents overfitting on easy domains. Figure (ouure7/shooe s/oow evaluation scores as function of B. Each task has its own different optimal 3 value depending on the task dynamics and complexity, which makes this parameter important to optimize for maximum performance.
Ablations. We performed a detailed ablation of LERM and LDT described in section 3 and show that both term in their symmetric form are necessary for good performance across all benchmarks. Results are shown in appendix
RELATED WORKS
Data augmentation. Although not studied in this work, data augmentation is an efficient and easy to implement method to regularize models and increase generalization performance. Some methods study which type of data augmentation better suited for reinforcement learning. Apart from the standard random shift/random crop (aarats et al. 2020; 2021 Laskin et al. (020b), some work ex- plored data mixing strategies (Wang et al. 2020 Zhang Guo 2021 Zhou et al.) 2020) inspired by mixup style data augmentation. Lee et al. 2200 Zhou et al. (2020) explored random convolutions and convolution mixing respectively. Recents works (Seo et al. 2022, Xiao et al., 2022) used patch- based masking, inspired from masked autoencoders in computer vision. Automatic adjustment of data augmentation is also studied, with Raileanu et al. (202T) using upper-confidence- manfidence-bound bandit algorithm to find the transformation that maximizes return, Zhang & Guo (2021): Agarwal & Chin-
8

---

Under review as a conference paper at ICLR 2023
chali 2202) opting for an adversarial objective making the task difficult as possible for the policy
and Lu et al. (2200) using learned causal models to apply visual interventions. Other works study
how and when data augmentation should be applied. Haansen & Wang [ Hansen et al.
(2021)
apply DA assymetrically to source and target encoder and/or Q-value functions. Ko & Ok
(021)
show that data augmentation shouldn't be applied uniformly during training, while/Fan et al. (2021)
only use strong data augmentation for behavior cloning of an expert policy. While efficient, these methods are sub-optimal if our goal is increase data diversity for control because data augmenta- tion is only post- post-processing transformation as explained in our paper.
Visual domain randomization. Contrary to data augmentation, domain randomization directly provided by the environment, as the changes are applied at render time or even while building the scene components. [Hansen & Wang (202T): [Sonn et al. (202T); Grigsby & Qi (2020) mod- ify the original Deepmind Control Suite (Tassa et al., 2018) to include visual distractions, Xing] et al. (202T); Zhu al. (2020) involve robotic tasks and Ahmed et al. (2020) specifically focus on causal relationships in robotic toy environments. Domain randomization can also be integrated in- side the methodology, by enforcing inductive bias to the agent such that it remains robust to visual changes. Akkaya et al. (2019) dynamically adapts the level of domain randomization to the agent's performance while Ren et al.
(2020) use an adversarial objective similar to Zhang & Guo (2021). Other works run multiple versions of the same environment in parallel and use adversarial objectives (202T) or enforce invariance between environments /Zhao & Hospedales
Ren et al. (2020); Li et al.
(2021): ; James al. (2019): Zhang et al. (2020).
Invariant representation learning is another approach to ensure good generalization across visual perturbations and is usually implemented an auxiliary self-supervised task to the reinforcement learning algorithm. Zhang et al. (202T): Agarwal et al. (202Ta); [Bertran et al. (2022) use bisim- ulation and behavioural similarity to learn invariant representations. Other works use the causal inference framework to isolate causal feature sets. IZhang et al. (2020) learns a model of the envi- ronment with invariant causal prediction (Peters et al. 2016) in the block-MDP setting by unrolling multiple versions of the same environment in parallel while [Sonar et al. (202T) uses invariant risk minimization (Arjovsky et al. (219)) across multiple domains to learn representations that are invari- ant action prediction. Lu et al. (022)) combined data augmentation with counterfactuals to learn a structured causal model with an adversarial GAN-like objective. Li et al. (2021) also uses an ad- versarial objective combined with gradient reversal to learn a representation that is less predictive of interventions on style variable. [Mzziian et al.] (2020) combined bisimulation and risk extrapolation (Krueger et al., (202)) learn robust representations under domain randomization for robotic tasks. We difter from this line of work as DIQL is model-free and does not use an auxiliary representation learning loss.
CONCLUSION
We presented in this paper DIQL (Domain-invarrian Q-learning), a model-free method that extends deep Q-learning for robust control from images under visual distractions. Our method combines efficient use of domain randomization with risk extrapolation to learn domain-invariann Q-function. We demonstrated strong results and sample efficiency on the Distracting Control Suite benchmark. To our knowledge, DIQL shows state-of-the-ar performance on Easy and Medium benchmarks (intensity 0. 0.2) with all three types of dynamic distractions at the same time, especially camera movement distractions. Due to the simplicity of implementation and low computational burden of the method compared to representation learning modell-aseed approaches, advocate for more generalized integration of domain randomization into reinforcement learning algorithms.
An obvious limitation of DIQL is its need for domain randomization, although one could adapt the loss function to work with well-suited data augmentation as in Hansen et al. (202T). Yet, recent development of simulations with increased speed and photo- rhaloss call for systematic integration of domain randomization into RL pipelines for robust agents acting from images. Future work involve scaling the method to more realistic environment, with more interventions per step maximize sample efficiency and testing results on real robot learning.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide in Appendix B all hyperparameters associated with the method, implementation de- tails for the method and the environment, as well links the github repositories we based our implementation on.
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021a.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Informa- tion Processing Systems, 34, 2021b.
Shubhankar Agarwal and Sandeep P Chinchali. Task-driven data augmentation for vision-based robotic control. arXiv preprint arXiv:2204.06113, 2022.
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Ben- gio, Bernhard Schölkopf, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. In International Conference on Learning Representations, 2020.
Igge Akkaya, Marein Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv: 1900009100001133 2019.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: /907, 0000002299,,.. 2019.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement
learning. International Conference on Machine Learning, pp. 449 458.. PMLR, 2017.
Martin Bertran, Walter Talbott, Nitish Srivastava, and Joshua Susskind. Efficient embedding of se- mantic similarity in control policies via entangled bisimulation. arXiv preprint 12300 2022.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http:
//github. cat.uuu.coo/qooogagg ccom///ogl///a.../co/////a//a/aa
Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree
Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In International Conference on Machine Learning, pp. 3088-3099, PMLR, 2021.
Jake Grigsby and Yanjun Qi. Measuring visual generalization in continuous control from pixels. arXiv preprint arXiv:2010.06740, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- machine learning, pp. 1861-1870. PMLR, 2018.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data aug- mentation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp- 13611-13611 IEEE, 2021.
Nicklas Hansen, Rishabh Jangir, Yu Sun. Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. International Conference Learning Representations, 2020.
10

---

Under review as a conference paper at ICLR 2023
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems, 34: 3680-3693, 2021.
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stañczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Amold, Manu Orsini, Alexis Jacq. Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 2020.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12627-12637, 2019.
adaptation networks. In Proceedings of
Byungchan Ko and Jungseul Ok. Time matters in using data augmentation for vision-based deep reinforcement learning. arXiv preprint arXiv:2102.0858/ 2021.
David Krueger, Ethan Caballero, Joemn-Henri Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribttion generalization via risk extrapo- lation (rex). In International Conference Machine Learning, pp. 5815-5826. PMLR, 2021.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma. Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations, 2021.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances neural information processing systems, 33: 19884-19895, 2020b.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple tech- nique for generalization in deep reinforcement learning. International Conference on Learning Representations, 2020.
Bonnie Li, Vincent François-Lavet, Thang Doan, and Joelle Pineau. Domain adversarial reinforce- ment learning. arXiv preprint arXiv:2102.07000 2021.
Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hermández-Lobato, Kun Zhang, and Bernhard Schölkopf. Sample-efficient reinforcement learning via counterfactual-batt based data augmentation. arXiv preprint 2rXiv:2012.09092, 2020.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600 2013.
Melissa Mozifian, Amy Zhang, Joelle Pineau, and David Meger. Intervention design for effective sim2real transfer. arXiv preprint arXiv:2012.02055, 2020.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series (Statistical Methodology), 7845):94711012, 2016.
Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34:5402-5415
2021. 2021.
11

---

Under review as a conference paper at ICLR 2023
Yangang Ren, Jingliang Duan, Shengbo Eben Li. Yang Guan, and Qi Sun. Improving generaliza- tion of reinforcement learning with minimax distributional soft actor-critic. In 2020 IEEE 23rd International Conference Intelligent Transportation Systems (ITSC), pp. 1 1-6. IEEE, 2020.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. arXiv preprint arXiv:2206. 14244, 2022.
Anoopkumar Sonar, Vincent Pacelli, and Anirudha Majumdar. Invariant policy optimization: To- wards stronger generalization in reinforcement learning. In Learning for Dynamics and Control, pp. 21-33. PMLR, 2021.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting trol suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint urXiv:1801.006900 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. Advances in Neural Information Processing Systems, 33: 7968-7978, 2020.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279 2979 1992.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203. 06173, 2022.
Eliot Xing, Abhinav Gupta, Sam Powers, and Victoria Dean. Kitchenshift: Evaluating zero-shot generalization of -bate policy learning under domain shifts. In NeurIPS 202 Workshop Distribution Shifts: Connecting Methods and Applications, 2021.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representa- tions, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2021.
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In International Conference on Machine Learning. pp. 1 11214-11224. PMLR, 2020.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn ing invariant representations for reinforcement learning without reconstruction. In International Conference Learning Representations, 2021.
Hanping Zhang and Yuhong Guo. Generalization of reinforcement learning with policy- ad- versarial data augmentation. arXiv preprint arXiv:2 /5587, 2021.
12

---

Under review as a conference paper at ICLR 2023
Chenyang Zhao and Timothy Hospedales. Robust domain randomised reinforcement learning
through peer-to-peer distillation. In Asian Conference on Machine Learning. pp. 1237-1252. PMLR. 2021.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2020.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martín robosuite: A modular simu- lation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 2020.
A CONTINUOUS CONTROI. WITH DIQL
Soft Actor- Critic extends Q-learning to continuous control with an entropy maximizing actor- critic algorithm. The policy loss is defined as follow:
J Ex [EaL,T
(a log (Io (a. s:)) Qo (St, ac]]] EN [T. (s:)]
The temporal cemporal-difference difference loss used to learn the action-value function is similar equation I with an additional entropy regularization term and sampling a,+1 ~ (s,):):
Qurget = rr Y (Qa(s++1, a4+1) a log n (at+1 (41))
where a is temperature parameter controlling exploration and is either fixed trainable.
Deep Q-learning in itself only implements a Q-function and can only output discrete actions. For continuous control, we adapt the invariant Q-learning loss Soft Actor-Critic (Haarnoja et al.) 2018) where the temporal-different of augmenting the soft TD error with the DIQL loss, we average the policy loss of SAC over the two observations of the same state. In theory, an invariant Q-function would suffice to learn an invariant policy with SAC as the policy takes actions that maximizes Q-values. practice, this is rarely true during training. We enforce the policy to also stay invariant by averaging the policy loss over both observations.
error is naturally replaced with the soft Bellman error. On top
J )DIQT(
(6) +
1
where Jn is defined just above.
B IMPLEMENTATION
Each experience in the paper is run 4 different seeds for reproducibility. We the SAC im- plementation of ACME (Hoffman et al. 2020) in Jax (Bradbury et al., 2018 for our DIQL imple- mentation, and only change the critic and actor losses described in section 3 and A Both policy and Q-network are implemented with convolutional stack followed by a MLP. The policy and Q- network only share weights of the convolution stack to compute lower-dimeeniooal visual features. The convolutional stack (or "encoder) is composed of 4 convolutional layers with 32 filters and 3 x 3 kernel sizes. Stride is for the first convolutional layer then 1 for the rest. Outputs features are flattened and put through a linear layer to reach final dimension of 50. Layer normalization and tanh activation is applied the features before passing them to actor or critic's 's MLP. Encoder layers are initialized with delta orthogonal initialization, while all linear layers used Lecun uniform unitialization. All networks use ReLU activations units. Trainig is done with the Adam optimizer, and all hyperparameters used are described in table [ Importantly, 3 was optimized only for two tasks (Walker and Ball in cup) for compute availability reasons and kept at a constant value of I for others. Results can be improved for the four other tasks by finding more optimal B for each of them.
We the githut? implementation of the Distracting Control suite for the Easy, Medium and Hard benchmarks and only modify the Very Easy by removing camera movement.
ng-
13

---

Under review as a conference paper at ICLR 2023
Table 1: Hyperparameters
Hyperparameter
Value
Replay buffer size
100000
Initial collection steps
25000
Optimizer
Adam
Actor learning rate
3e-4
Critic learning rate
3e-4
Weight decay
0
Initial temperature a
0.1
Temperature learning rate
3e-4
Batch size
128
T EMA
5e-3
Actor hidden layers
[512, 512]
Critic hidden layers
[512, 512] 3
Frame stacking
Action repeat
if Cartpole; 2 if Finger, Walker; 4 otherwise 5 if Walker; 0.1 if Ball in Cup; otherwise
DIQL variance penalty 8
C DISTRACTING CONTROL SUITE
14

---

Under review as a conference paper at ICLR 2023
Ball in cup Catch
Cartpole Swingup
Cheetah Run
Finger Spin
Reacher Easy
Walker Walk
Clean
Very Easy
Easy
Medium
Hard
Figure 8: Distracting Control Suite tasks and evaluation benchmarks used in the paper.
D DETAILED RESULTS
D.1
SAMPLE EFFICIENCY CURVES
15

---

Under review as a conference paper at ICLR 2023
DIOL-F
FRM-F
SAC-F
Walker walk
Ringer spin
Cheetah Run
Normalized IOM Returnn
2.0
Normalized IQM Return
Nomalized 1OM Return
0.8
0.8
0.6
0.0
0.3
0.2
0.3
0.0
0.0
0.0
100
700
300
400
500
100
200
300
400
500
100
700
300
400
100
Staps
(M)
Steps (M)
Steps (M)
easy
Ball-in-cup
Cartpole Swinguup
Nomallizdd (OM Return
altredd IOM Return
Norm allzed TOM Return
0.0
0.8
0.8
0.0
0.0
Normalized
0.4
0.4
0.2
0.0
700
700
300
500
Stops (M)
Stops (M)
Steps (M)
Figure 9: Sample efficiency curves on Clean benchmark. Line is normalized IQM and shaded area bootstrapped CI.
DIQL-E
ERM-E
SAC-E
Walker walk
Ringar spin
Cheatah Run
Normalzed IQM Return
Normalized IQM Return
0.e
Normalized LOM Return
0.6
0.0
0.0
0.7
0.0
0.0
0.0
100
200
300
400
500
200
200
300
400
500
100
200
300
400
300
Steps (M)
Steps IM)
Steps
(M)
easy
Ball-in-cup
nquu
Nornalized (OM Return
3.01
Normalized IOM Retur
Normalized aeaaaateedde IOM Return
M
0.8
0.6
0.0
0.4
0.4
0.3
0.2
0.2
0.0
Steps (M)
Steps (M)
Steps (M)
Figure 10: Sample efficiency curves on Very Easy benchmark. Line normalized IQM and shaded area bootstrapped CI.
16

---

Under review as a conference paper at ICLR 2023
DIOL-F
FRM-F
SAC-F
Walker walk
Ringer spin
Cheetah Run
Normalized IOM Return
0.0
Normalized IQM Return
Norm alized IOM Return Normalized 1OM Return
0.3
0.6
0.4
0.2
0.0
0.0
0.0
100
700
300
400
500
100
200
300
400
500
100
760
300
400
100
Staps (M)
Steps (M)
Steps (M)
Reacher easy
Ball-in-cup
Cartpole Swinguu
Nomalized (OM Return
Normalized alteed IOM Return
0.0
0.0
0.0
0.3
0.4
0.4
0.4
0.0
200
700
300
Stops (M)
Stops (M)
Steps (M)
Figure 11: Sample efficiency curves on Easy benchmark. Line is normalized IQM and shaded area bootstrapped CI.
DIQL-E
ERM-E
SAC-E
Walker walk
Ringar spin
Cheatah Run
Normaized LQM Return
Normalized IQM Return
Comaaized 10M Return
0.3
0.3
0.10
0.2
0.05
0.1
0.3
0.0
0.20
100
200
300
400
500
200
200
300
400
500
100
200
300
400
300
Steps
(M)
Steps IM)
Steps
(M)
easy
Ball-in-cup
Swingup
Normalized IOM Return
0.0
Normalized IQM Retur
Cormalized IOM Return
0.8
0.35
0.0
0.30
0.6
0.4
0.4
0.20
0.2
0.2
0.15
0.0
Steps (M)
Steps (M)
Sleps (M)
Figure 12: Sample efficiency curves on Medium benchmark. Line is normalized IQM and shaded area bootstrapped CI.
17

---

Under review as a conference paper at ICLR 2023
DIQL-E
Waller walk
Finger spin
Cheetah Run
lormelizee 1O1 Return
Normelized LOM Return
lormelized LOM Return
0.23
0.12
0.20
0.20
0.00
0.04
0.09
0.00
0
100
200
300
400
300
0
100
200
300
400
530
3
100
200
300
400
500
Steas
(M)
Steps (M)
Steps (M)
casy
Saingup
Normalized IOM Return
Normalizee IOW Return
0.8
Narmalized JQM Return
0.25
0.4
0.6
0.3
0.20
0.2
(M)
Steps
Figure 13: Sample efficiency curves Hard benchmark. Line normalized IQM and shaded area bootstrapped CI.
18

---

Under review as a conference paper at ICLR 2023
D.2 GENERALIZATION
Woltee walk
Finger spir
Cheetan Run
300- ERAE
DIQUEE BIM-E SAC
3.5
DICLE EUN-E
SAC
S4C
2.3-
Kormabed
6.4-
Narmalzeed
0.4
0.2-
62-
6.0
Dean
Medium
Very
Mediums
Dean Easy
Medum
Hardd
Eva uation comain
Evallaaion domain
Evcluation damain
Peacher
Ball catch
Carpeete Saringuo
1.0
DIOLE
1.0
DIGLE BUM-E
ERVAE
SAC-2
C.6-
D.r
SACE
C..
Normatzed
led
Normalzedd
2.5
0.4
0.4
0.4
2,3
DIQU-E SAC E
32-
3.1
Dean
Medium
Medium Ecaluation damain
Mertumm
Here
ualian domain
Eveluation
domaii
Figure 14: Evaluation at the end of training of DIQL, ERM and SAC across all benchmark difficul- ties splitted per task. Points normalized IQM episodic returns, and vertical bars are bootstrapped confidence intervals.
'Niibwr wnik
Rhgger splr
Cheetan Rum
DICLAVE
Diguatt
DIOI-E
3,5m
DIQUEE
0.8
DACI -M
Eagt 44
C.6-
0.4
0.6
Cormaalzzes/oNN IOM
0.3-
C.4
0.4
2.2
Owan: Very Easy
Easy Medium
Hard
Clean
Very Easy
Easy
Medurs
Hard
Owan Wery Easy
Easy Rudhum
Bard
Evo untion domsin
Evalustion domain
Evoluntior domain
Reacher easy
Ball nueup catch
Comppote Swinngu
1.0
0.8
DIGU-V
TAGI
Naraltzed ret.a
0.8
DIOL M
C.6-
2.7-
DILL
10M
2.6
C.6-
2.5
0.4
Normal
n.
0.2
DIOLVE
3.34
0.2-
3/2
Usem
Medium
Clean
Vey
Medars
land
Medium
Evo uatton comain
Evalustion domain
Evelustion domain
Figure 15: Evaluation at the end of training of DIQL-E, DIQL-VE and DIQL-L across all 5 bench- mark difficulties splitted per task. Points normalized IQM episodic returns, and vertical bars bootstrapped confidence intervals.
19

---

Under review as a conference paper at ICLR 2023
E ABLATIONS
We perform ablations on the ERM and DT (domain translation) terms from the DIQL loss and show that all four TD error terms are necessary for good performance. To perform the ablation, disabled risk extrapolation by fixing 3 to O. We define the following notations for the ablations: LERM composed of ERM1 and ERM2, LDr DI, and DI2 (each corresponding to one of the two TD loss of each term). ERM refers to LERM- DT refers to CDr. ERM + DT is equivalent DIQL with 3 O. Training done on the Easy benchmark for all ablations.
IQM
Mean
Median
ERM DT ERM 2 + DT 1
ERM 2 + DT_2
DT
DT 2 DT 1 ERM
ERM 2
ERM 1
0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00 0.25 0.50 0.75 1.00
Normalized Score
Figure 16: IQM and bootstrapped CI of ablations on Clean
IQM
Mean
Median
ERM + DT
ERM 2 DT 1
ERM 2
2
DT
DT
DT
ERM
ERM 2
ERM
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
Normalized Score
Figure 17: IQM and bootstrapped CI of ablations on Very Easy
20

---

Under review as a conference paper at ICLR 2023
IQM
Mean
Median
ERM + DT
ERM 2 + DT
ERM 2 + 2
DT
DT 2 1
D
ERM
ERM
ERM
0.25 0.50 0.75
0.2
0.4
0.6
0.8
0.25
0.50
0.75
Normalized Score
Figure 18: IQM and bootstrapped CI of ablations Easy
IQM
Mean
Median
ERM + DT
ERM 2 + DT
ERM 2 + DT_2
DT
DT 2
D
ERM
ERM
ERM 1
0.15 0.30 0.45
0.15
0.30
0.45
0.15
0.30
0.45
Normalized Score
Figure 19: IQM and bootstrapped CI of ablations on Medium
IQM
Mean
Median
ERM + DT
ERM 2 DT 1
ERM 2 + 2
DT
DT 2
DT 1
ERM
ERM
ERM
0.15 0.30 0.45
0.15
0.30
0.45
0.15
0.30
0.45
Normalized Score
Figure 20: IQM and bootstrapped CI ablations on Hard
21


Reviews:
[{"review_id": "hdHN7BPOkAz", "reviewer": "Reviewer_AhUs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach for augmenting Q-learning to take advantage of domain randomization, name DIQL.  By producing observations from two different domains (here defined as visual observations with different distracting perturbations, as in the distracting control suite), DIQL can train its q-network to be invariant to the sorts of perturbations that do not affect the task.  ", "summary_of_the_review": "The paper presents an interesting method for learning task-relevant representations of high dimensional state.  However, I am not convinced that the method is feasible, since it requires knowledge of the ground truth state to generate pairs of observations, and this knowledge seems like it should be off-limits to any approach that attempts to learn task-relevant features.  I'm hoping I've misunderstood a main point in the paper, but if not I do not think it passes the bar for acceptance.", "strength_and_weaknesses": "The paper makes an interesting comparison between data augmentation and domain randomization, and suggests that RL methods can take advantage of invariance built in to different domain randomization techniques.  The method is described well and the experiments show empirical advantages of the DIQL approach.\n\nThe paper currently has the following weak points, but these may be a result of a misunderstanding on my part: \n- DIQL depends on having $o^1_t$ and $o^2_t$ where each comes from a different domain, but where the underlying state $s_t$ is the same.  How can this correspondence be found?  If this is just assumed known (as it must be to generate different domains), why not give direct access to the state $s_t$ to the learning algorithm rather than the more roundabout domain-translation terms in the loss?  Many other task-relevant state representation methods I am aware of do not assume this correspondence is known, but rather must select positive and negative examples for, say, contrastive learning.\n- I would have liked to see augmentation-based representation learning methods, since the paper makes a point to contrast domain randomization with data augmentation.  Additionally, some selection of bisimulation-based and contrastive methods (like the ones cited in the paper) would be good to compare against empirically.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ik3jPEJ_bE", "reviewer": "Reviewer_N4B4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of distraction robustness and out-of-distribution (OOD) generalization in model-free RL. To this end, the paper proposes to use domain randomization combined with some modification to the Q-learning objective. Specifically, the model takes two views, one clean and one noisy. The noisy view is generated by adding random noise to the groundtruth state (as opposed to pixel-space augmentation). For each view, there are two loss terms, regressing the Q-network toward the two Q-targets computed from both views (as opposed to only a single Q-target computed from the same view). To further encourage invariance of the Q-network, the paper proposes to also minimize the variance among these loss terms, following a previous work on OOD generalization. The experiments focus on the Distracting Control Suite, with SAC being the backbone RL algorithm and main baseline. It is shown that the proposed method outperforms SAC in distracting environments, and can generalize better than SAC to larger noise during evaluation. Ablation shows that all loss terms have effect.", "summary_of_the_review": "I recommend reject for the paper in its current form, due to potentially limited significance and weak baselines.", "strength_and_weaknesses": "- Strengths\n    - The paper is generally well written and easy to follow.\n    - Ablation study shows the effectiveness of each loss term.\n- Main Weaknesses\n    - It seems unclear whether the problem setting is interesting or meaningful to tackle. This paper assumes access to the clean state in order to be robust to distractions. This sounds impractical, and most work in this field (e.g., [DBC](https://openreview.net/forum?id=-2FCwDKRREu), [TIA](https://proceedings.mlr.press/v139/fu21b.html), [TPC](https://proceedings.mlr.press/v139/nguyen21h.html), [Dreaming](https://arxiv.org/abs/2007.14535), [DreamerPro](https://proceedings.mlr.press/v162/deng22a.html)) does not have access to clean states when dealing with distractions. It is even unclear whether the access to clean states is necessary, as the paper does not compare to these methods.\n    - The baseline is SAC with no access to the clean state. This seems to be a weak baseline, and it is not surprising that the proposed method, with access to the clean state, can outperform this baseline. A reasonable baseline could be [DrQ](https://arxiv.org/abs/2004.13649), with the augmented views replaced by the clean and noisy views.\n- Questions and Minor Issues\n    - Why use one clean view and one noisy view, not two noisy views? What happens when you use many noisy views?\n    - It is a bit confusing to put models trained on different noise levels into one figure (e.g., Figure 3).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AWL-Dx8rDlP", "reviewer": "Reviewer_3X9n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies reinforcement learning across domains in the presence of visual distractions. A visual distraction is a part of the underlying state that is uncontrollable, i.e., its change is independent of agent's action, and does not affect the Q values. The proposed approach uses domain randomization in a Deep Q-learning setup. At each time step $t$ in an episode, one assumes access to two observations $o^1_t$ and $o^2_{t+1}$ which denote the same controllable state but different uncontrollable state. Here the superscripts 1 and 2 are used to denote the two different domains. The proposed approach includes two additional loss:\n\n- a domain transfer loss: $L(o^1_t, a_t, r_t, o^2_{t+1}) + L(o^2_t, a_t, r_t, o^1_{t+1})$, where $L$ is the temporal difference loss defined by $L(o, a, r, o') = (Q(o, a) - r - \\gamma \\max_{a'} Q_{target}(o', a'))$. This is done to ensure that Q functions remain invariant to the uncontrollable noise. This claim is not proven.\n \n- a variance loss that computes variance of $\\{L(o^1_t, a_t, r_t, o^2_{t+1}), L(o^2_t, a_t, r_t, o^1_{t+1})\\}$. This is used so that losses for both domains are roughly of the same scale. It is not explained why this would happen but a reference to Krueger et al., 2021 is made.\n\nExperiments on Distracting Control Suite (Stone et al, 2021) are presented and show promise of the approach.\n", "summary_of_the_review": "I currently lean towards weak reject based on two concerns: (i) strong assumptions on generating a pair of observations with the same state but different style variable, and (ii) issues with writing. For (i), I would like to understand how to satisfy this assumption in practice outside simulations. I'll change my score based on author response (and paper revision) to these concerns. ", "strength_and_weaknesses": "Strength:\n\n1. Robust reinforcement learning (RL) is an important topic and visual distractions are one of the important things an RL agent needs to be robust to. \n2. Proposed approach is simple to implement and test\n\nWeakness:\n\nTwo main concerns are:\n\n1. The proposed approach requires access to a pair of observations at each time step that have the same controllable state $(s)$ but different uncontrollable state $(\\bar{s})$. If I understand correctly, experiments use the knowledge of how visual distractions are added to generate these observations. How will this be accomplished in practice? E.g., consider a driving scenario where the agent needs to drive. The observation space is an image that shows what is ahead of the car. On the side of the car, there can be noise from people walking or varying weather conditions. How will one generate this pair of observation that only changes these distractions but not the controllable state? This may require domain knowledge almost equivalent to knowing the state. In contrast, the domain knowledge required in computer vision pipeline, as abstracted in a data augmentation setting, is very basic and includes simple operations such as rotation and flipping. \n\nIt is also assumed that different domains differ only in the uncontrollable state noise but otherwise have the same state. This is a particular kind of domain generalization. For example, if one considers navigation in two different buildings, then even the underlying state is different.\n\n2. The writing in this paper is very low quality. This includes mathematical notations used without definition, using words without definition, etc. See clarity section below for examples. Even though the main ideas are easy to understand for an RL audience, the paper needs serious editing for readability.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject