Offline Deep Reinforcement Learning for Visual Distractions via Domain Adversarial Training
Authors: Jen-Yen Chang
Thomas Westfechtel
Takayuki Osa
Tatsuya Harada
Venue: Accepted by TMLR ()

Abstract: Recent advances in offline reinforcement learning (RL) have relied predominantly on learning from proprioceptive states. However, obtaining proprioceptive states for all objects may not always be feasible, particularly in offline settings. Therefore, RL agents must be capable of learning from raw sensor inputs such as images. However, recent studies have indicated that visual distractions can impair the performance of RL agents when observations in the evaluation environment differ significantly from those in the training environment. This issue is even more crucial in the visual offline RL paradigm, where the collected datasets can differ drastically from the testing environment. In this work, we investigated an adversarial-based algorithm to address the problem of visual distraction in offline RL settings. Our adversarial approach involves training agents to learn features that are more robust against visual distractions. Furthermore, we proposed a complementary dataset to add to the V-D4RL distraction dataset by extending it to more locomotion tasks. We empirically demonstrate that our method surpasses state-of-the-art baselines in tasks on both the VD4RL and proposed dataset when evaluated on random visual distractions.

Published in Transactions on Machine Learning Research (10/2024)
Offline Deep Reinforcement Learning for Visual Distractions via Domain Adversarial Training
Men-Yen Chang
chou@miit.uttoogo..e...
Graduate School of Information Science and Technology, The University of Tokyo
Thomas Westfectel
thomas@@mi...-tookk.......
Graduate School of Information Science and Technology, The University of Tokyo
Takayuki Osa
os@@mitta--okkoo.c...
Graduate School of Information Science and Technology, The University of Tokyo RIKEN Center for Advanced Intelligence Project (AIP)
Tatsuya Harada
harulas@mi...utkkky...co
Graduate School of Information Science and Technology. The University of Tokyo RIKEN Center for Aduanced Intelligence Project (AIP)
Reviewed on OpenReview: https: // openreview. net/ forum? id= dce620kJJ2
Abstract
Recent advances offline reinforcement learning (RI.) have relied predominantly on learning from proprioceptive states. However, obtaining proprioceptive states for all objects may not always be feasible, particularly in offline settings. Therefore, RL agents must be capable of learning from raw sensor inputs such as images. However, recent studies have indicated that visual distractions can impair the performance of RT agents when observations in the evaluation environment differ significantly from those in the training enviromment. This issue is even more crucial in the visual oflline RL paradigm, where the collected datasets can differ drastically from the testing environment. based algorithm to address the problem of visual distraction offinne RL scttings. Our adversarial approach involves training agents to learn features that more robust against visual distractions. Furthermore, V-D4RL distraction dataset by extending it to more locomotion tasks. We empirically demonstrate that our method surpasses state-of-the-art baselines in tasks on both the V- D4RL and proposed dataset when evaluated on random visual distractions.
this work, we investigated an adversarial-
proposed complementary dataset to add to the
1 Introduction
Common model-free (Kunar et al. 2020 Fujimoto & Gu 2021 An et al. 2021 Kostrikov et al. 2022) and model-based (Kidambi et al.J 2020 Guo et al. 2022 offline reinforcement learning (RL) algorithms rely on learning from the proprioceptive states address continuous control tasks. However, obtaining proprioccptive states is almost impractical in certain scenarios, such as largo-scale environments, where defining the states of all objects nearly unfeasible. Therefore, RL agents must learn from raw sensory inputs such as images. Training offline RL agents based on visual observations provides opportunities to make RI. more widely applicable to real-world settings. Unlike offline RL agents trained from proprioceptive states, studies on offline RL agent training using visual observations (Lu et al. 2023) for continuous control tasks and well-designed benchmarks are scarce.
Frequently, using visual encoder, we can estimate the latent proprioceptive features of an imago- basoo observation and use these latent features train the RL backbone algorithms (Hansen et al.) 2021a Yarats et al. 2022 Although this well-established approach is effective in scenarios with static backgrounds, it has

---

Published in Transactions on Machine Learning Research (10/2024)
domainy
typical visual encoder
ours
adversarial loss pushing
æ1
F1,bg
encoder
Zugent
Sagent
22
22,02
domaing
21 Fugent V F1,0g
more common
adverserial loss pushing
22 Engynt 1220
agent features
Figure 1: Our problem formulation. The agent is trained on two lypes of domains: a normal observation and a distracted observation. Intuitively, we can say that the latent features consist agent features and distraction features. The visual encoder incentivised by the adversarial training loss to extract latent features that are common to both domains by suppressing domain-specifie visual distractions..
limitations when confronted with visual distractions. These distractions may include changes in background clements, vicwpoints, or variations in the agent's colour scheme. Existing RL agents. whether offline or online, underperform in the presence of random visual distractions (Cobbe et al. 2020 Stone et al., 2021 Wang et al. 2021 Dupuis et al. 2022). By contrast, humans possess a remarkable ability to disregard visual distractions when observing actions from a visual input. This adeptness highlights our capacity to concentrate on domain-invarian features, focusing on aspects that remain consistent across diverse settings, such agent's 's movement, patterns.
We hypothesised that existing vision-based RL algorithms are susceptible to random visual distractions because the visual encoder in these RL algorithms tends to overfit certain visual distractions during training. Consequently, these algorithms struggle to estimate robust latent features when exposed to unseen visual distractions during the evaluation. Our objective is to train the visual encoder in RL agents such that they can learn nonainninvarianl features within ofline RL settings.
In this work, we investigated an adversarial learning approach to address the visual distraction issue in offline RL settings by extending a technique originally developed for domain adaptation (Ganin & Lempitsky 2015 One important assumplion is the accessibility to dalasets from two lypes of domains, such as dalaset without distraction and another dataset with visual distractions.
We consider a framework where the agent is trained on two types of domains and subscquently evaluated on other unseen visual distractions. We introduce a domain discriminator that attempts to classify the latent features estimated by a visual encoder into two classes. The visual encoder trained to minimise the discrepancy between the latent features of the to maximise this discrepancy. The problem formulation and schematics of the adversarial training method are presented in Figure 1 The visual encoder is incentivised by the adversarial training loss to extract latent features common to both domains by suppressing domain-specific visual distractions, Furthermore, we empirically demonstrated that incorporating DropBlock (Ghiasi et al. 2018 into a visual encoder is essential for improving the robustness the proposed domain-adversant adversarial training scheme.
domains, whereas the domain discriminator aims
To investigate the performance of offline RL algorithms in the presence of unscon distractions, we empiri- cally evaluated existing oflline RL algorithms the V-D4RL distraction dataset. For a more comprehensive comparison, collected offline datasets from additional locomotion tasks. We observed that these agents performed well when evaluated with previously seen distractions present in the training dataset, but exhib- ited a significant drop in performance when encountering unseen distractions during evaluation. Figure 2 illustrates this phenomenon. summary, the contributions of this work are as follows:
We empirically demonstrated that commonly used offline RL methods are frequently susceptible to visual distractions. This highlights the need develop techniques to enhance policy robustness against visual distractions.
2

---

Published in Transactions on Machine Learning Research (10/2024)
2. We adopted adversarial algorithm to visual-based offine RL settings and address the problem of visual distraction. The adversarial algorithm allows the visual encoder to learn domain-invariant features. Furthermore, we introduce DropBlock (Ghiasi et al. 2018) into our proposed offline RL algorithm for further robustness against unseen visual distractions.
3. We extended the existing V-D4RL distraction dataset, which contains only one visual distraction task, to four additional locomotion tasks for a comprehensive benchmarking of visual distractions between offline RL algorithms.
4. We empirically demonstrate that the adopted domain-advvrsarria learning method improves perfor- mance. Additionally, we compare our method with of offline RL algorithms and related based RL algorithms and empirically demonstrate that our method more robust in visual distraction settings in both the V-D4RL dataset and our proposed novel dataset.
2 Related Work
2.1
Domain Invariance in RL
Recent advancements in RL, such as RAD [Laskin al.) 2020). DrQ (Yarats et al. 2021). and DrQv2 Yaarats al. 2022). have demonstrated the effectiveness of data augmentation techniques in improving the RL performance, sample efficiency, and gencralisation. Particularly in vision-based RL, data augmentation has emerged a prominent strategy for reducing the reliance specific training domains. These augmee- tation schemes enhance the robustness of the encoder representations and subsequently improve the policy, yielding superior results compared with previous methods (Yarats et al., 2022). For online visual-based RL, (Cobbe et al. 2019) investigated the effectiveness of common computer vision techniques for improving RL generalisation, and (Li et. al. 2021) employed gradient reversal layer in online RT. settings.
Various visual-based generalisation methods include PAD (Hansen al. 2021b) which employs self- supervised learning during evaluation; SVEA (Hansen et al. 2021a which updates typical CNNs in encoders to transformers and uses stronger image augmentation; and ILA Yoneda et al. 2022 which uses dynamic models to adapt during evaluation. In broader context, [gl et al. (2019) Islam et al. (023) proposed using the variational information bottleneck theory to enhance domain generalisation in RL. They introduced an additional policy gradient objective to simultaneously minimise the mutual information between different inputs, alongside actor-critic methods, thereby achieving better feature robustness. Our work focuses specif- ically on the robustness of the visual encoder, distinguishing from previous methods. Domain invariance is also an important topic from the RL safety perspective, and (Haider et al.) 2021 provided an overview and demonstrated how RL agents struggle to provide clear safety boundaries when deployed on safety-related tasks.
evaluation visual distractions included in dataset
evaluation visual distractions dataset
1000
AWAC-BC
AVAC-8C DO004-CC
800
DVOx240C
000
rewerds
600
200
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
1e6
1e6
Figure 2: Initial result of baseline agents DCQ and AWAC+BC trained on the V-D4RL cheetah-run medium-experl. easy-distraction dataset (Lu et al. 2023). When evaluating on the seen visual distractions that can found in the training dataset, we can observe offline agents achieving expert-level performance (left). However, the agents perform very poorly when exposed unseen random visual distractions (right).
3

---

Published in Transactions on Machine Learning Research (10/2024)
2.2 Benchmarks for Offline Continuous Control
D4RL et al. 2020) is prominent benchmark for continuous control of proprioceptive states in oflline RL. The large variety of data distributions has allowed for the comprehensive benchmarking of state-of-the
offline RL. algorithms (Kidambi et al., 2020 Kostrikov et al.) 2022 Kumar et al.) 2020) and an understanding
of the strengths and weaknesses of different lasks. Florence el al. (2022) discusses when offline RL algorithms
would outperform behavioural cloning (BC) in learning from proprioceptive states. Conversely, vision-based datasets for discrete control were created for Atari by (Agarwal et al, 2020): however, they contained only 50M samples per environment. Hence, V-D4RL (Lu et al., 2023) aims to develop comprehensive benchmark for visual-based continnous-control tasks. V-D4RL's 100 K benchmark represents significantly more approachable challenge. Another issue with the current datasets is that they are all iinulatiionbbsedd Zhou et al. 2022)) proposed using data collected in real-world settings and demonstrated its suitability for robot learning in realistic environments. However, visual distraction data are lacking in the original V- D4RL dataset. Our novel datasets aimed to complement the V-D4RL dataset for visual distractions, which originally included only one lask.
2.3 Dropout and its variants in RL
The application of dropout (Srivastava et al. 2014) to the RL. domain not new. For instance, Hiraoka et al. (2022) employed dropout in critic network, demonstrating improved sample efficiency and achieved comparable performance with only two critic networks compared to the ten critic networks required in REDQ CCen et al., 2021). Jaques et al. (2019) deployed dropout to obtaín uncertainty estimates of the target Q values to alleviate the Q-lcarning overestimation bias. Unlike prior studics that focused on critic networks to enhance sample efficiency, this study largets the encoder as a source of improvement. To the best of our knowledge, this is one of the first studies to apply DropBlock (Ghiasi et al. 2018 to RL.
3 Preliminaries
3.1 Reinforcement Learning from Images
A standard RL problem for image- control can be defined as an infinite- Markov Decision Process MDP = (S, A, P, R. 7), where S the set of observations, is the set of actions, p is the transition probability function, R is the reward function, and y (0, 1) is the discount factor for future rewards. Generally, in image settings, image rendering of the system not sufficient to fully describe the underlying state of the system. To this end, and per common practice (Mnih et al. 2013), approximated the current obscrvation of the system by stacking three consecutive prior images as an obscrvation. We define the replay bulfer D containing the observation, action, reward, and next observation at time step I as D = (8t, Tt. 8++1)- The RL agent aims to maximise the discounted expected return Ex E 7 R(ss, at)]. which is the expected cumulative sum rewards when following the policy in the MDP, where the importance of the horizon is determined by 7. Consequently, the goal is determine policy æ that maximises discounted expected returns.
3.2
Visual Actor-Critic Methods
We consider architecture in which the imput observation is first transformed using a data augmentation scheme. Subsequently, the encoder f with parameters 0, maps to a lower-dimensional latent feature vector
z == fs;00)) Q-networks have parameters Og. For the replay buffer D and policy n, the policy objective
becomes:
D.a (Q(f(se;)) a7: 0,)
We employ clipped double Q-learning (Fujimoto et al.
2018 to roduce overestimation bias in the target Lu al.) (202)). Target Q networks are Q networks with parameters 0g which are slow-moving copy of 0g- The overall critic objective is
value following previous works Yarats al.) [021]] 2022
4

---

Published in Transactions on Machine Learning Research (10/2024)
min L
[(y ((f(s);0)) ae: 0g))
(2)
Q
where larget value y - min (f 0g)
1=1,2
3.3 Offline Reinforcement Learning
Offline RL problems can be defined as a data-driven formulation of the RL problem. In offline RL, an agent can no longer interact with the environment and collect additional transitions using its learned policy. Instead, the agent was provided with a static dataset of transitions loaded into the replay buffer {(s), af, ri, 1)+1) In visual offline RL, following DrQv2+BC (Lu et al. 2022). if we consider an additional policy constraint behavioural cloning (BC) term with its strength regulated by hyperparameter Xbo the policy objective becomes
argmax -a ()(((s;))).of) 0y) Abcla, a.)
(4)
3.4 Visual Distractions Dataset
We describe the visual distraction datasets as static datasets of transitions, where observations are sampled from different observation distributions, and the reward and action are sampled from the same distribution.
We can define datasets with and without visual distractions as Duormal i,normal J.normal i.normal ,5+1 ) and Dais {s di ,t' ,841 ( respectively. The V- DARL benchmark (Lu et al. 2023) is
{(s) dr Jr.
i,dis i,dis
commonly used offline dataset for visual-based offline RL methods but contains only one visual distraction task. The V-D4RT, dataset was generated with fixed visual distraction; that is, the same visual distraction persisted throughout the dataset, whereas during the
evaluation, visual distractions were generated randomly.
4 Domain Adversarial Training for Visual Distractions
We hypothesise that the subpar performance of the current state-of-the-art basclines can be attributed visual distractions present during the evaluation but absent from the offline RL training dataset. These unseen visual distractions confuse the encoder, making the encoder estimate less robust latent features Consequently, the actor-critic backbone, which relies the latent features 2, cannot learn robust policy and state-action estimations. Baseline agents can achieve expert-level performance in an environment where the same visual distractions exist in the training dataset. However, when we evaluated the same agent in an environment with random visual distractions, the agent's performance diminished significantly, as shown previously Figure 2
In this section, we discuss the proposed approach for improving the robustness of visual-based offiine RL for visual distractions. Our proposed method is based on DrQv2+BC (Yarats et al. 2022: Lu et al.) 2023) and incorporates two components. We adopt (i) domain discriminator that trains ad adversarially against. the encoder, and (ii) DropBlock (Ghiasi et al. 2018) layers added to the encoder to achieve robustness against unseen visual distractions. An overview of this architecture is presented in Figure 3
To address this issue, we propose a framework in which an agent trained using datasets from two domains: a normal observation domain denoted by Doormal and a visually distracted domain denoted by Ddis Intuitively, the presence of domain-specife visual distractions may allow the discriminator to accurately classify latent features belonging to particular class. However, aim to induce the opposite and train the visual encoder such that the estimated latent features are indistinguishable from the domain discriminator.
For the actor-critic backbone, we chose the same actor-critic RL backbone as presented in DrQv2 Yarats et al. 2022 Lu et al.) 2023), which is a variant of the actor-critic structure in TD3 (Fujimooo et al. 2018) using a stochastic policy. We use only the distraction latent features dii = di ) to train the actor-critic backbone. Only the image encoder, which we hypothesise the source of underperformance, is trained with both Duormal and Ddis.
5

---

Published in Transactions on Machine Learning Research (10/2024)
replay buffer
adversarial training
backward gradient reversal 1
Discriminator
St
din
24
visual encoder
distractions replay buffer
dis
actor
#2
Ais
critic
RL agent
Figure 3: An overview of general visual actor-critic RL methods with our additional proposed discriminator: The lower branch & common actor-critic backbone where an visual encoder is used to estimate latent proprioceptive features z and uses z to train the RL backbone. The upper adversarial branch is where we reverse the sign of gradients from discriminator loss from the discriminator g such that the encoder is incentivised to learn features that cannot be classified successfully by g.
4.1 Domain Discriminator
To facilitate domain adversarial learning, we introduce a discriminator g which trains adversarially against the encoder via gradient reversal (Ganin & Lempitsky/ 2015). The domain discriminator g(2; maps the latent features
fs; fy) with parameters fg. and outputs the probability of the latent feature 2. We use k as the label for classes & E 10, representing normal and distraction observations respectively. Both normal and distraction observations underwent the same data augmentation scheme, which was the same as that in DrQv2 (Yarats et al. 2022) and DrQv2+BC (Lu et al. 2023). Given that we sample n observations from the replay buffers Duorma and Ddls the discriminator objective T.p can be expressed as a binary cross-entropy loss, and is definod as follows:
La
Σ &LE(D UDdis))
Kg log(o(2;00)
log(1 ((z;; 0,)) where sss;0,)
(5)
N.
Contrary to conventional binary classification tasks, our objective was not discriminate between different domains. Rather, we aimed ensure that the discriminator wnable to classify encoded latent features when they similar, irrespective of the domain from which the observations encoded. To achieve this goal, adopted the gradient reversal technique proposed for classical domain adaptation tasks (Ganin & Lempitsky 2015
Although the discriminator learns to use the encoded latent fcatures 2 to classify the domains, the encoder learns features that cannot be used to classify the domain successfully because the gradients are reversed. In other words, while the discriminator learns maximise the latent feature discrepancy between the domains, the encoder attempts to minimise this discrepancy. Consequently, the encoder learns extract latent features that are shared across both domains, that is, domain-invariant features, while simultancously suppressing features that are exclusive to one domain. The parameter updates rule for the encoder f and for discriminator are, in addition to the gradients from the critic objective L, as follows, where 771 and are the lcarning rates:
0L
OL
0, - 0, 7
0g
0, 172
(6)
00,
00,
00,
6

---

Published in Transactions on Machine Learning Research (10/2024)
The encoder parameters By are jointly updated based two gradients: one based the critic objective and the other based on the discriminator objective, as illustrated in Equation ( The key distinction in a typical update is the -μ hyperparameter. This hyperparameter is crucial in adversarial training. This -p hyperparameter is necessary for preventing a typical encoder update from learning the dissimilarities between leatures across domains. For all experiments, we selected μ μ=
4.2 DropBlock in Encoder
Common to supervised learning approaches, forcing the encoder not to rely on specific features discourages overfitting the visual distractions found in the offline dataset. To this end, we added DropBlock (Ghiasi eet al. 2018| the visual encoder, driving the encoder rely on wider patch of features during the learning process to improve its robustness, thus preventing less robust latent feature estimations when unseen distractions are present. DropBlock drops patches of convolutional map (c.g. 7 x 7 map) rather than a single cell (as in dropout (Srivastava et al.) 2014p). The visual encoder would be less likely to overfit the detailed observations that are present in the dataset but would gather information that exists in both Dnormal and Ddis, consequently reducing the reliance on background information. We empirically found that adding DropBlock to the visual encoder f frequently enhanced the performance of the proposed domain adversarial training approach. The results are presented Table
5 Novel visual distractions dataset
Table 1: Comparison between V-DARL distraction benchmark and our novel dataset. v denotes that the data exists. A denotes that although the distraction data exist for walker-wall random, it was only collected in 64 64 pixels, whereas the cheetah-run medium-expert data were collected in 84 84 pixels. In contrast, ours are all 84 x 84 pixels.
Task
Difficulty
V-D4RL distractions our datascts
cheetah-run
random
cheetal-ruun
medium-expert
mall-in-cup-catcl
random
hall-in-cup-cater
medium-expert
reacher-casy
random
reacher-easy
medium-experi.
reacher-hare
random
teacher-hard
medium-oxpert
walker-
random
A
walker-walk
medium-expert.
V-DARL Lu et al. 2023) provides a starting benchmark for visual distractions in offline RL settings. How- ever, specifically for distraction observations, they only collected the cheetah-run medium-ezpert set (in 84 x 84-pixl format) and the walker-wall walk random set (in 64 x 64-pixel format). Therefore, it suitable extend the existing V-D4RI datasct to provide a more comprehensive benchmark for offline continuous control with visual distraction. We collected the Distracting Control Suite (Stone al. 2021 data and the original DeepMind Control Suite (Tassa et al. 2018 for four additional tasks with three difficulties each, all in unified 84 84-pixel format: medium, expert and random sets (medium and expert are combined to form the medium-cxpert set). Specifically, we collected for the following tasks: cheetah-run, walker-walk, ball-in-cup catch, reacher-easy, and reacher-hard. For comparison, we also collected our set for the existing molkerr-walk. random set and the cheetah-run medium-expert set. A brief comparison between the proposed novel datasot and the V-D4RL benchmark is presented in Table [ For dataset collection, we fol- lowed the procedures described in V-D4RL (Lu et al. 2023). more detailed comparison of the benchmarks is provided in A Appendix

---

Published in Transactions on Machine Learning Research (10/2024)
6 Comparison with offline RL algorithms
To evaluate the proposed method, we train four bascline offline RL agents: DrQv2+BC, visual AWAC+BC, and visual IQL and offine-adapted DreamerV? V2 (Hafner et al. 2021). noted as DV2 (following to V-DARL). Visual-WWA++BC is a variant of DrQv2+BC BC that uses AWAC (Nair et al. 2020) as an actor-critic backbone in addition to the BC term, and visual-IQL uses IQL (Kotrrkovv et al. 2022) as the backbone (without the BC term). Both use the same visual data augmentation and encoder scheme in DrQv2+BC and only replace the actor-critic method. All four baseline algorithms were trained dirrctly using the distraction dataset. As mentioned in the proposed method, the input domains for the discriminator are normal and dis-x and dis-x e {easy, medium, hard]. This is to ensure that the RL backbones were trained on the same offline data, and to isolate the effects of the proposed method on the encoder. The imput domain for the RL backbone contained distracted data x, x E feasy, medium, hard], as noted in the training set column in Table g The discussion and results on two alternative formulations where 1) the encoder of baseline methods were trained on both distracting data, and 2) where our proposed method is trained with two difficulties of distractions (no normal observation) are included in Appendix [D] A full list of hyperparameters are shown in Appendix L.4 We follow the original works' hyperparameters for bascline methods. A brief robustnoss analysis can be found in Appendix G
for all algorithms.
All reported numbers were unnormalised, with a maximum reward of 1000 and evaluated over 30 episodes. All reported numbers represent the evaluation results at 1M stops. We trained five seeds: seed E {0,1,2,3,4] For the best practices (Agarwal et al. 2021), we report the IQM and 95% stratified bootstrap confidence intervals (CTs). The drop rates for different tasks are presented in Appendix I
6.1 V-D4RL Distraction Dataset
As defined in V-D4RL (Lu et al., 2023) and the Distracting Control Suite (ttone et al. 2021), ), there are three levels of distractions: distraction-eass
instraction-mddinge and distraction- The training results evalu- ated on all three types of distractions (noted respectively as e, m and h), alongside the normal observations (noted n), are summarised in Table g Overall, the experimental results demonstrate that the proposed method consistently outperforms other model-free offline RL baselines in most tasks. DrQv2+BC and IQL were the second-best algorithms overall. Interestingly, training directly on more challenging datasets does not result in better performance, even on difficult tasks, compared with agents trained on simpler datasets and evaluated on more difficult tasks. We hypothosisc that this is because the datasets do not contain suf- ficient variations in distractions for the more difficult distractions; thus, the agents have difficulty learning to deal with them, whereas training on easier tasks allows the agent to infer some domain invariance across distractions.
6.2 Our Distraction Dataset
The selected results on the walker-walk medium-expert distraction dataset are shown in Table 3 and an overview of the results is presented in Table 4 for our proposed novel dataset. We observed that our proposed method outperformed all the tasks overall. For more detailed results for our dataset, please refer to Appendix B The drop rates for different tasks differ; more details regarding the hyperparameters are presented in Appendix]] [
7 Comparison with visual generalisation methods in RL adapted to offline settings
In this section, compare the existing generalisation methods in RL that utilise both normal-domain and distracted-domain data. These methods are closoly related to our proposed method. The main difference from our work is that these methods were generally developed for online RL with access to two environments, and we assume that we have access to two datasets. Another difference is that these met often require consecutive training, whereas our method can be trained in an end-to-end manner.
Specifically, we adopted PAD (Hansen et al. 2021b), SVEA (Hansen al. 2021a), and ILA (Yoneda et al. 2022) for purely offline settings. To facilitate comparison, we adapted these methods to offline RL by initially
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Training result with V-DARL cheetah-run medium-expert distraction dataset. The reported num- bers are IQM and 95% stratified bootstrap CIs in between parentheses. We mark numbers in bold when algorithm performs statistically significantly better using t-test with S 0.05 compared to other algorithms on IQM. Abbreviations: N-normal, E=easy, M-medium, H-hard.
cheetah-run medium-expert (V-D4RL)
train
eval
DrQv2+BC
AWAC+BC
IQL
DV2
Ours
N
115.1
(61.4, 177.6)
82.2 (96.0, 250.8)
113.9 (66.0, 165.6)
59.6 (21.9.
229.7
(160.6,
E
83.6 (57.9, 96.0)
60.9 (533, 72.8)
67.1 (b4.8, 76.9)
70.0 (42.2, 92.1)
130.6 (102.0 181.0)
M
35.0
(30.1, 44.39
28.5 (20.2, 47.5)
41.9 (39-7, 54.4)
76.2 (36.0, 134.6)
81.0
(66.3, 100.3)
H
18.1 (14.2, 28.0)
18.6 (14.8, 22.7)
20.3 (14.8, 27.0)
90.2 (42.6, 130.9)
55.4
(42.2, 60.5)
N
69.6 (4b.1, 67:2)
35.1 (3.1, 106.8)
55.8 (37.6, 115.3)
84.2 (49.7, #8.5)
84.4 (36.2, 116.0)
E
32.5 (22.3, 56.2)
18.4 (17:5, 26.4)
39.3 (26.1, 61.5)
55.3 (41-1, 121-1)
60.5 (33 0, 72.8)
M
M
34.4 (21.3, 40.4)
24.5 (17.2, 29.8)
36.9 (29.7, 40.1)
61.8 (34.8, 88.0)
49.2 (87.0, 68.9)
H
13.5 (12.5, 16.6)
22.0 (17.3,
20.4 26.9)
65.4 (48.0, 98.8)
20.2
N
17.3 (9.9,
9.9 (5.8,
4.9 (2.4,
73.6 (15.6, 134-3)
39.2 (16.9,
E
14.2 (9.8,
16.6 (18.9,
8.5 (6.0, 12.0)
76.5 (37.6. 104.7)
29.4 (21.0,
M
11.6 (10.4, 22.3)
15.8 (12.5, 21.1)
10.8 (9.2, 19.5)
105.5 (62.7, 118.9)
24.9 (17.7, 28.5)
12.4 (9.1, 15:7)
16.6 (13.3, 82.7)
9.8 19.2, 14.0)
58.5 (37.6, 81:77
27.3 (16.7, 33.5)
IQM total
457.3
349.1
430.2
876.8
831.8
Table 3: Results comparison on our dataset walker-walk medium-expert distraction dataset. The reported
numbers are IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal: N, dis-easy: E, dis-medium: M, dis-hard: H.
walker-walk medium-expert (ours)
train
eval
DrQv2+BC
IQL
DV2
Ours
469.8
(296.3. 511.3
184.7 (07.1, 381.3)
30.6 (27.3, 45.2)
633.4
(600-4, 768.3)
E
231.9
(184.7, 264.0)
246.4 (165.9, 204.1)
21.0 (19.4, 28.7)
293.8
(284.4, 401.8)
M
118.0
(97.4, 188.4)
96.7 (70.0, 156.8)
26.4 (26.2, 81.0)
156.5
(124.1, 251.7)
H
70.1 (41.9,
56.2 (10.4; 74.8)
29.1 (24.5, 30.3)
104.0
(50.5, 142.3)
N
229.8 (121.2, 817.7)
143.6 (89.5, 346.1)
31.5 (27.8, 41.8)
362.9 (145.6, 407.9)
E
163.8 (133.9, 173.6)
105.6 (76.1, 170.7)
32.0 (28.0, 39.7)
209.1 (178.5, 259.2)
125.2 (108.4, 186.2)
130.6 (94.7, 140.8)
35.6 (31.2, 42.1)
191.1 (164.3, 249.7)
69.0 (38.9, 76.4)
70.4 (57.6, 96.0)
31.4 (30-4, 37.3)
112.5 (104.5, 131.6)
30.0 (25.5, 66
30.6 (18.8, 41.9)
45.1 (85.5, 02.7)
42.5 (39.2, 81.7)
E
32.3 (28.1, 97.9)
31.4 (25.6, 38.6)
47.5 (35.9, 64.1)
47.6 6 (39.4, 80.2)
H
M
38.9
(84.3, 42.03
31.1 (24.2, 40.0)
40.2 (34.0, 60.8)
56.6 (42.5, 70.0)
H
37.1
(33.3,
30.9
38.4 (30.4,
51.2
IQM total
1615.9
1158.2
408.8
2261.2
training them on one distracted observation, then adapting them to the respective test environments, and subsequently demonstrating their performance post-adaptation. All hyperparameters were in accordance with the original authors; we did not make algorithmic changes but only data changes.
In PAD Hansen et al. 2021b), we used the V-D4RL dataset (distracted observations) as the replay buffer during training and used the inverse dynamics task the auxiliary training task because it was the best- performing setup in their work. During deployment, we directly adapted to the distracting control suite environment. In the ILA (Yoneda et al. 2022), we train the source domain (in Figure in their paper) purely on the V-D4RL dataset (distracted observations). For the target domain, we adopted the V-D4RL. distraction dataset. We used the dynamic model introduced in ILA. For the SVEA, we trained directly on the V-D4RL dataset (distracted observations). The selected results shown in Table 5. and the full results

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Aggregated IQM totals on V-DARL distractions dataset and our novel visual distractions dataset,
evaluated on medium, expert and medium-cxpert difficulty of respective datasets. For some of the tasks
below, only train DrQv2+BC for baseline comparison. Please refer to Appendix B for the full results.
Task
DrQv2+BC
Ours
cheetah-run (V-D4RL)
1407.0
2298.0
cheetah-run (ours)
853.9
2035.0
ball-in-cup-catch (ours)
15298.4
18465.9
reacher-casy (ours)
1792.1
2432.6
reacher-hard (ours)
52.0
231.6
walker-walk (ours)
4644.7
6594.3
Table 5: Results comparison with related domain adaption methods in RL trained with V-D4RL cheetah- run eepput distracting dataset. The reported numbers are 1QM and 95% stratified bootstrap Cls in between pasentheses.
cheetah-run medium-expert (V-D4RL)
train
eval
PAD
ILA
SVEA
Ours
65.8 (48.1, 65.3)
100.4 (00.7 124.6)
39.2 (28.7, 56.83
229.7 (180.6, 251.8)
F.
28.7 (21.7, 39.8)
13.7 (8.7, 25.3)
28.1 (23.0, 40.8)
130.6 (102.0, 161.0)
E
M
17.5 (18.8, 19.3)
25.9 (15.6, 31.7)
36.8 (26.7, 46.4)
81.0 (66.3, 109.8)
H
22.4 (16.3, 24.8)
29.0 (20.1, 40.1)
57.9
82.03
55.4 (42.2,
N
23.6 (17.6, 26.9)
155.3
160-7)
51.2 (38.7;
84.4 (36.2, 116.0)
E
19.2 (16.4, 28.8)
60.0 (43.6, 60.8)
35.0
60.5 (38.0, 72.8)
M
14.1 (11.9, 19.8)
34.2 (29.1, 39.5)
35.8 (28.4, 58.7)
49.2 (37.0, 66.9)
21.3 (16.7. 26.8)
26.4 (23.8, 33.0)
41.2 (25.7, 50.2)
20.2 (13.8, 30.8)
16.8 (14.2, 28.2)
96.4
(80.3, 108
31.0 (22.6, 37.0)
39.2 (16.9, 60.8)
16.3 (11-4, 20.8)
38.3
(22.7, 66.1)
25.9 (20.1, 28.03
29.4 (21.0, 41.2)
M
18.8 (13.3, 92.0)
40.5
(23.1, 45.7)
25.3 (21.4; 32.1)
24.9 (17.7, 28.5)
H
20.3 (12.9, 23.3)
23.4 (22.5, 27.5)
21.5 (18.0, 34.4)
27.3 (18.7, 33.8)
IQM total
284.8
607.5
430.9
831.8
are summarised in Appendix distracted environments.
C
Our proposed method outperforms three domain adaptation approaches in
Interestingly, PAD and SVEA usually exhibit fewer performance drops when trained on an easier distraction dataset and are evaluated more difficult environment. Similar to ours, the ILA Yonnda al.] (022) uses adversarial approach, which may explain why our proposed method and the ILA suller from significant performance drops when distraction difficulties increase, whereas the PAD and SVEA generally less affected.
8
Ablation Studies
8.1 DropBlock drop rate
We investigated the ellect of DropBlock drop rate using the original V-D4RL cheetah-run distraction dataset.
To achieve this, we train agents with p= (0.0, 0.1, 0.2, 0.3, 0.5). To understand how the DropBblock helps, we train the proposod adversarial discriminator without a DropBlock (p 0.0) in the enr. We obscrve that p 0.3 is the best performing overall, whereas having no DropBlock (p 0) is the least performing variation. Selected results is shown in Table and the full results is shown in Table 19
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: DropBlock ablation results on V-D4RL cheetah-run distraction dataset. The reported numbers are IQM and 95% stratified bootstrap Cls in between parentheses. Abbreviations for dataset type: medium: m, expert: e, medium-expert: m-e. Abbreviations for visual distractions difficulty: normal: N, easy: E, medium: M, hard: H. For example, 'e H' equals 'expert Hard'.
Task
eval
p 0.0
p 0.1
p 0.2
0.3
p 0.5
N
85.4 (45.4, 119.5)
185.9
(172.9,
171.7 (94.4,
229.7
(180.6, 251.5)
194.3 (75.8,
E
71.7 (96.9, 80.0)
130.6 (102.0, 161.0)
121.7 (99.9, 141.2)
110.6 (83.2, 141.2)
96.8 (91.9, 116.6)
m-E E
M
42.4 (31.9, 48.8)
68.0 (06.7. 64.9)
94.0 (87.6, 102 8)
81.0 (66.3, 109.3)
77.0 (65.5, 82.0)
H
34.3 (32.0, 36.0)
50.9 (31.3, 53.4)
49.1 (48.0, 50
55.4 (42.2, 60.5) 5)
42.3 (36.7, 46.7)
N 63.7
(36.7. 110.4)
33.5 (14.3, 116.4)
43.0 (4.5, 106.4)
84.4 (36.2, 110.0)
60.2 (26.1, 70.6)
m-E M
E 33.4
(27.8, 47.4)
27.3 (22. 6, 96.2)
40.5 (31.1, 55.0)
60.5 (33.0, 72.8)
53.0 (39.8, 90.8)
M 45.0 (31.0, 69.6)
29.7 (21.0. 92:2)
31.2 (25.1 44.0)
43.9 (22.0, 59.63
49. (37.0, 60.9)
H 25.3 (22.1, 27.5)
13.2 (10.4, 31.7)
20.2 (13.8, 30.8)
13.4 (9.0, 25.7)
15.3 (12.1,
N 21.9 (16.7,
6.0 (2.4, 7.7)
21.5 (17.1,
39.2 (16.9,
24.4 (11.3,
E
12.0 (10.1, 21.9)
16.4 (13.0, 21.0)
29.4 (21.0, 41.2)
21.5 (14.7, 48.8)
18.0 (15.6, 23.8)
m-E H
M
14.9 (18.0, 10.0)
18.6 (11.0. 22.5)
24.9 (17.7, 28.5)
19.0 (11.2, 26.2)
15.0 (11.3, 19.2)
H
17.1
(11.5, 23.2)
18.1 (13.2, 22.4)
27.3 (16.7, 33.8)
12.7 (10.9, 22.9)
12.8 (9.5, 21.0)
IQM total
467.1
598.2
674.5
771.3
658.3
8.2
Different types of distractions
It is important to understand which types of distraction are casier for the agent to learn, and which types are more difficult. As noted previously, there are three lypes of distractions: background, viewpoint, and colour, with three levels of distraction respectively. We performed ablation studies on different types by enabling only one distraction at a time and testing at three levels each. The evaluation task is cheetah-run. We used p = 0.3 our proposed algorithm as the basis for ablation. The results for the medium- expert dataset are presented in Table 7 and the full results are presented in Appendix [.] Generally, the proposed method performs the best overall. Interestingly, robot colour changes had minimal effect on the agents' performance. The background changes also had slight impact. Agents are generally heavily impacted by viewpoint changes, and the final performance is significantly related to the agents' ability to adapt to viewpoint changes.
9 Limitations
As mentioned previously, one of the assumption of our proposed method is that two datasets are available. In addition, we also have not discussed in depth the impact of the quality of the two datasets effect performance. For example, one interesting question to ask is, how different must the two datasets be for proposed method to work? We briefly discuss using two distraction dataset to train our proposed method in Appendix D An interesting future direction is to further propose novel methods using only distracting data for training.
10 Conclusion
In this work, tackle the issue of visual distraction in offline RL settings, where the distractions that exist in the dataset might significantly differ from those in evaluation. We investigated an adversarial-bass algorithm for visual-based offline RL using additional discriminator with gradient reversal and introducod DropBlock to the visual encoder. Our approach trains the encoder such that it will learn domain-invariant features that are more generalisable when unseen visual distractions are present during evaluation. Our method can be casily implemented by adding a discriminator branch while reversing the gradient flow and adding DropBlock. Despite the simplicity and efficiency of this method, we empirically demonstrated that it achieved excellent performance across various difficulties in the V-D4RL visual distraction dataset, as well
11

---

Published in Transactions on Machine Learning Research (10/2024)
in our proposed datasets. We demonstrated that our proposed method is more robust to various types of distractions, both individually and in combination. We also showed the importance of having DropBlock to achieve better performance. Additionally, we empirically found that agents are significantly affected by changes in viewpoint, a topic that, to the best of our knowledge, has not been extensively discussed. Future studies should address this issue.
Acknowledgement
This work was partially supported by JST Moonshot R&D Grant Number JPMJJS2011, CREST Grant Number JPMJCR2015 and Basic Rescarch Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo. This work is also partially supported by JSPS KAKENHI Grant Number JP23K18476.
Open Source
We have open-sourced our codebase on GitHuu with instructions at: https://github.coom comm/tine/ C advrrrrrral and our collected dataset can be found in https://drive.googlee com/
12

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Distractions ablation results on V-DARL cheetah-run distraction medium-erpen dataset, which is
essentially extended version of Table 2 The 'all' row is the same in Table 2 The reported numbers are IQM and 95% stratified bootstrap Cls in between parentheses. Abbreviations for dataset type: medium-
expert: m-e. Abbreviations for visual distractions difficultyy: normal: N, easy: E, medium: M, hard: H. The
ablation studies for the expert and medium distractions datasets are presented in Appendix
train
eval
type of dis
DrQv2+BC
AWAC+BC
IQL
Ours
E
background
91.3 (G0.8, 111.8)
69.1 (06.0.
97.8 (71.1,
215.2
(187.3, 220.5)
E
viowpoint
47.1 (25.0, 74.1)
51.3 (28.22 119.8)
106.3 (60.4, 120.1)
170.7
(345.5, 192.3)
E
colour
69.0 (45.4, 140.77
95.6 (40.5, 143.5)
114.0 (75.5, 158-1)
227.9
(202 269
E
83.6 (37.9, 95.0)
60.9 (50-3, T2.8)
67.1 (94.8: 76.8)
130.6
(102 0, 161.0)
M
background
111.4 (109.3, 123.1)
95.8 3, 129.5)
157.3 (95.7, 186.6)
236.8
(109 6. 244.9)
m-e E
M
viewpoint
32.9 (14.2, 44.7)
28.3 (16.7, 68.6)
48.2 (20.6, 59.2)
94.3
(81.5, 119.0)
M
colour
95.0 (53.2, 146.8)
98.2 143.2)
96.0 (68.8, 142.0)
217 217.0 (189 282.3)
M
all
35.0
28.5
41.9
81.0
H
background
92.7 (78.6, 160
71.1 (61.3 104.5)
107.7 (91.5, 165.1)
194.7
(149.4. 222.3)
H
viewpoint
21.0 (10.5, 25.3)
16.7 (12.2, 80.4)
25.4 (17.1, 41.7)
65.3
(47.3, 74.8)
H
colour
71.2 (51.5, 122.13
105.8 (50.8, 168.7)
98.8 (39.5, 125.2)
200.2
(162.4, 239
H
all
18.1 (14:2, 29.0)
18.6 (14.8, 22.7)
20.3 (144.5, 27.6)
55.4
(42:2) 60.5)
E
background
25.0 (14.3, 47.0)
27.6 (21.6, 79.9)
18.2 (12:9, 51.7)
49.3
(88.6, 63.9)
E
viewpoint
37.2 (27-1, 41.8)
53.9 (22.3, 85.0)
29.3 (19:2; 78.3)
80.8
(51-8,
E
colour
55.3 (38.7, 74.6)
49.9 (10.2, 84.6)
53.9 (32.0,
70.8
(36.7,
E
all
32.5
18.4
39.3 (26.1.
60.5
M
background
44.0 64.0)
40.2 (36.0, 85.6)
25.3 (20.9,
56.0 (47.6,
M
viewpoint
23.5 (12.5, 30.4)
48.1 (21.8, 70.0)
17.1 (13.8, 35.1)
44.7
(288.6 6, 61.1)
mne
M
colour
39.2 (94.3, 50.7)
57.3 (36.8, 72.7)
44.3 (32.5, 64.5)
76.2 (38.3, 99.9)
M
all
34.4 (21.3, 49.4)
24.5 (17.2, 20.8)
36.9 (29.7, 40.1)
49.2 (87.0, 69.9)
H
background
39.4 (29.8, 61.7)
41.7 (31-7, 61.8)
34.6 (28.1, 69.0)
68.5 (39.2, 85.4)
H
viewpoint
15.5 (10.6, 18.9)
24.5 (16.7, 82.0)
16.4 (9.4, 29.6)
35.3 (17.9,
H
colour
40.6 68.1)
50.4
71.3)
41.7 (25.7,
84.0 (42.8,
H
all
13.5 (12.6,
22.0 (17.3,
20.4 (12.7,
20.2 (19.8, 30.99)
E
background
11.6 32.6)
22.2 (19.0, 34.0)
6.7 (6.1, 12.8)
44.3 (97.1,
E
viewpoint
16.4 (9.0, 80.8)
9.2 (T.2, 14.8)
7.7 (6.6, 10.3)
32.4
(22.6, 48.5)
E
colour
16.9 (9.0, 28.0)
11.2 (9.6, 21.7)
6.3 (3.6, 11.3)
43.8
(19.4, 64.4)
E
all
14,2 (9.8, 10.5)
16.6 (13.9, 25.8)
8.5 (6.0, 12:0)
29.4
(21.0, 41.2)
M
background
18.0 (12/2, 21.0)
22.9 (18-5, 34.3)
8.6 (6-7, 13.1)
34.0
(26.3, 78.5)
m- H
M
viewpoint
14.8 (7.2, 24.8)
12.7 (10.6, 13.2)
9.2 (6.7, 14.0)
27.1
(21/2)
M
colour
14.5 (9.8, 24.1)
18.0 (11.7, 24.2)
10.7 (7.0, 16.3)
41.9
(18.3)
M
all
11.6 (10.4,
15.8 (12.6,
10.8 (9.2, 19.6)
24.9
(17.7,
H
background
14.0 (11.3, 20.4)
20.5 (18.4, 28.1)
8.2 (8.1, 9.3)
27.9
(20.3, 45.5)
H
viewpoint
10.8 (7.0, 20.7)
12.3 (11.0, 17.9)
8.6 (6.7, 16.7)
32.7
(22.0, 36.3)
H
colour
15.9 (9.7, 25.7)
13.6 (10.3, 23.8)
13.1 (7.9, 18.7)
29.1
(17.4, 98.2)
H
all
12.4 (9.1, 15.7)
16.6 (18.3, 82.7)
9.8 (9.2, 14.0)
27.3
(16.7, 33.0)
13

---

Published in Transactions on Machine Learning Research (10/2024)
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offiine rein- forcement learning. In International Conference Machine Learning (ICML), 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Gaon An, Scungyong Moon, Hang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Represenlations (ICLR), 2021.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
Tom Dupuis, Jaonary Rabarisoa, Quoc Cuong PHAM, and David Filliat. Domain invariant q-learning for model-free robust continuous control under visual distractions. In NewrIPS Deep Reinforcement Learning Workshop, 2022.
Pete Florence, Corcy Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning (CoRL), 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, arXie preprint arXiv:2004.07211 2020.
Scolt Fujimoto and Shixiang Shane Gu. A minimalist approach to oflline reinforcement learning. In Con- ference on Neural Information Processing Systems (NeurIPS), 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional Conference on Machine Learning (ICML), 2015.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V. Le. DropBlock: A regularization method for convolutional networks. In Conference Neural Information Processing Systems (NewrIPS), 2018.
Kaiyang Guo, yunfeng shao, and Yanhui Geng. Model-based offline reinforcement learning with pessimism- modulated dynamics belief. In Conference Neural Information Processing Systems (NeurIPS), 2022.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations (ICLR), 2021.
Tom Haider, Felippe Schmoeller Roza, Dirk Eilers, Karsten Roscher, and Stephan Günnemann. Domain shifts in reinforcement learning: Identifying disturbances in environments. In IJCAT ATSafely Workshop, 2021.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-laarning with convnets and vision trans- formers under data augmentation. In Conference on Neural Information Processing Systems (NeurIPS), 2021a.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Nicklas Hansen, Yu Sun, Picter Abbeel, Alexci A. Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. Tn International Conference on Learning Represenlations (TCLR), 2021b.
Takuya Hiraoka, Takahisa Imagawa, Taisci Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Repre- sentations (ICLR), 2022.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottle- nock. In Conference on Neural Information Processing Systems (NeurIPS). 2019.
Riashat Islam, Hongyu Zang, Manan Tomar, Aniket Didolkar, Md Mofijul Islam, Samin Yeasar Arnob, Tariq Iqbal, Xin Li, Anirudh Goyal, Nicolas Heess, and Alex Lamb. Representation learning in deep rl via discrete information bottleneck. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. Xiv preprint 1rx:i:1907 000,.0045, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. In Conference on Neural Information Processing Systems (NewrIPS), 2020.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Conference on Neural Information Processing Systems (NeurIPS). 2020.
Bonnie Li, Vincent François- Lavet, Thang Doan, and Joelle Pineau. Domain adversarial reinforcement learning. arXio preprint orX.v:2102.000 070977 2021.
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Tch. Challenges and opportunities in offline reinforcement learning from visual observations. In Transactions on Machine Learning Research (TMLR), 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXie preprint orXiv:1312.5600 2013.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXie preprint arXiv:2006.0933 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Tlya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, In Journal of Machine Learning Research (JMLR), 2014.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite a challenging benchmark for reinforcement learning from pixels. arXio preprint arXiv:2101.0272 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Frez, Yazhe Li, Diego de Tas Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Decpmind control 2018.
suite. arXiv preprinl ar
G.E. van der Maaten, L.J.P.; Hinton. Visualizing high-dimensional data using t-sne. In Journal of Machine Learning Research (JMLR), 2008.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Xudong Wang, Long Lian, and Stella X. Yu. Unsupervised visual attention and invariance for reinforcement learning. In Conference on Compuler Vision and Poltern Recognition (CVPR), 2021.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (TCLRR: 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Im- proved anta-augmented reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
Takuma Yoneda, Ge Yang, Matthew R. Walter, and Bradly C. Stadie. Invariance through latent aligmment.
Robotics: Science and Systems (RSS), 2022.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupla, Aravind Rajeswaran, and Vikash Kumar. Real world offiine reinforcement learning with realistic data source. arXiv preprint 0r1::22200.064 2022.
16

---

Published in Transactions on Machine Learning Research (10/2024)
A
Visualisations of tasks
Brief visualisation of tasks is shown below in Figure 4
with different difficulties of the tasks.
cheetah run
walker walk
cup catch
reacher easy
reacher hard
cheetah run
walker walk
catch
reacher easy
reacher hard
cheetah
walker walk
cup catch
reacher easy
reacher hard
cheetah run
walker walk
cup catch
reacher easy
reacher hard
Figure 4: Brief visualisation of tasks showing the different difficulties of the tasks, From top bottom are normal observations, dis-casy observations, dis-medium observations and dis-hard observations.
17

---

Published in Transactions on Machine Learning Research (10/2024)
B Full experiment results with offline RL algorithms
In this section, we include the full table results train on V-D4RL chectah-run distraction datasct, our pro- posed cheetah-run distraction dataset, ball-in-cup-catch distraction dataset, reacher-easy distraction dataset, reacher-hard distraction dataset and walker-walk distraction dataset. For V-D4RL. dataset, we compare with baseline agents DrQv2+BC, AWAC+BC and IQL. For our datasets, we mainly compared with DrQV2+BC as it was the best performing for the V-D4RL dataset, with the exception of walker-wall distraction dataset where included IQL to compare.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 8: Training result with V-D4RL cheetah-run distraction dataset. The reported numbers are IQM and 95% stratified bootstrap Cls between parentheses. We mark numbers in bold when one algorithm performs statistically significantly better using t-test with I 0.05 compared to other algorithms on 1QM. Abbreviations: N: normal, E: dis-casy, M: dis-medium, H=dis- hardd. et expert. m. medium. m-c: medium- expert.
cheetah-run (V-D4RL)
train
eval
DrQv2+BC
AWAC+BC
IQL
DV2
Ours
N
108.7 (42.0, 182.9)
44.6 (26.3, 129.6)
59.1 (27.4, 97.3)
124.2 (92.1, 199.3)
233.7 (197.9, 250.8)
E
34.2 (30.5, 47.8)
31.4 (18.4, 45.2)
50.0 (41.0, 95.6)
61.6 (48.1, 106.1)
100.8 (83.6, 113.7)
E
M
13.7 (11.6, 21.7)
7.8 (6.4, 25.1)
17.3 (1.0, 22.1)
141.2 (180.4, 150.1)
42.0 (85.8, 73.8)
H
9.7 13.7)
6.6
6.8 (5.0,
61.0 (48-1, 184.7)
33.6 (22.3,
N
18.9 (4.2. 64.7)
41.1 (11.7,
252.8 (197.0, 346.7)
145.4
238.1)
76.5 (25.2, 110.3)
E
33.3 (18.8, 40.7)
15.0 (9.3, 21.2)
15.2 (6.9. 26.6)
106.4 (80.0, 133.0)
31.1 (26.0, 51.1)
M
M
28.3 (22.7, 89.3)
17.9 (8.8, 23.1)
11.4 (6.0. 13.9)
85.8 (80.4, 180.1
27.7 (24.7, 41.6)
H
11.1 (9.3, 14.89
7.1 (5.7, 13:1)
5.3 (4.5, 8.8)
145.3 (86.7, 196.8)
20.1 (15.9, 27.3)
N
1.5 (1.0, 3.4)
5.0 [1.7. 15.3)
11.2 (4.1, 24.7)
79.5 (39 163.0)
24.3 (17.8, 44.9)
E
9.1 (8.4, 14:2)
14.4 (11.9, 10.8)
8.3 (8.2, 12.0)
69.6 (52.0, 83.9)
18.8 (12.8, 22.2)
H
M
9.8 (7.9, 17.8)
12.4 (10.2, 18.9)
11.5 (8.1, 16.6)
76.0 (35.3, 123.2)
16.2 (12.9, 18.9)
H
11.1
13.6
8.7
75.6
12.4
18.5)
N
131.0 (316.0,
79.6 (31.8,
124.9 (52.2, 170.9)
186.1 (180.7,
246.9
(219.7,
m E
E
99.5 (65.8, 130.4)
56.4 (44.0, 77.73
98.8 (63.1, 118.4)
83.8 66.66, 117.3)
152.0
(122.0,
M
41.4 (32.4, 66.8)
23.2 (17.2, 42.8)
26.1 (20.0, 70.4)
182.6 (101:6, 228.5)
68.2
(49.0, 59.3)
H
18.6 (19.7, 22.6)
14.4 (8-7, 19.3)
11.2 (7.6, 18.6)
128.9 (94.1; 171.3)
34.0 (26.9, 9, $1.5)
N
125.9 (81.9, 146.0)
70.4 (49.9, 139.2)
28.3 (16.4, 72.2)
128.7 (78.9, 179:2)
116.4 (68.0, 164.3)
E
37.6 (31.0, 63:2)
18.6 (11.9, 28.8)
27.0 (17.1, 41.6)
63.3 (41.9) 80.5)
45.5 (38.0, 97.9)
M
M
40.3 (23.6, 52.0)
14.3 (8-7, 17.7)
15.6 (13.5,
131.4 (82.5, 189.3)
50.3 (27.5,
H
21.2 (14.4,
9.8 (7.2. 20.6)
8,1 (7.2, 29.1)
71.8
26.5 (19.7,
N
60.3 (43.5, 70.7)
19.3 (4.2, 77.9)
13.6 (6.3, 24.7)
194.9 (101.6, 288.6)
36.2 (27.6,
E
26.3 (19.9, 83.6)
21.8 (13.0, 31.3)
8.7 (5.5, 14.6)
112.4 (66.7, 218.1)
18.3 (16.0, 24.8)
m H
M
24.5 (20.4, 02.6)
14.0 (7.1, 21.6)
10.6 (7.6, 16.9)
172.6 (11377, 176.8)
18.3 (14.4, 21.6)
H
33.7 (21.1, 87.0)
12.8 (9.8, 16.2)
15.6 (11.7, 21:2)
123.6 (90.1, 207.7)
16.2 (15.6, 19.9)
N 115.1
(51.4, 177.0)
82.2 (86.0, 250.8)
113.9 (06.0, 185.8)
59.6 (19.8, 118.2)
229.7
(180.0, 261.8)
m--e E
E
83.6 (57.8, 86.0)
60.9 (53.3, 72.8)
67.1 (54.8, 76.9)
70.0 (13.8, 92.2)
130.6
(102.0, 161.0)
M
35.0 (38.1, 44.0)
28.5 (20.2,
41.9 (30.7,
76.2 (36.6, 186
81.0 (66.3, 100.8)
H
18.1 (14.2,
18.6
20.3
90.2 (42.4,
55.4
N
69.6 (48-1, 87.2)
35.1 (3-1,
55.8 (37.6, 115:3)
84.2 (50.3, 88.6)
84.4 (36.2, 116.0)
E
32.5 (29.3, 56.2)
18.4 (17.5, 28.4)
39.3 (26.1, 61.5)
55.3 (90.6, 112
60.5 (33.0, 72.8)
m--
M
34.4 (21.3. 49.4)
24.5 (17.2, 29.8)
36.9 (20.7, 40.1)
61.8 (34.8, 89.2)
49.2 (37.0, 66.0)
H
13.5 (12.6, 18.0)
22.0 (17.3, 32.9)
20.4 (12.7, 24.9)
65.4 (48.0, 98.7)
20.2 (13.8, 50.8)
N
17.3 (0.8, 29.3)
9.9 (3.8, 16.0)
4.9 (2:4, 7.1)
73.6 (48.8, 134.3)
39.2 (16.9, 60.8)
E
14.2 (9.8, 16.8)
16.6 (18.9, 25.8)
8.5 (8.0, 12.0)
76.5 (38.2, 104.7)
29.4 (21.0, 41.2)
m-- H
M
11.6 (10.4, 22.3)
15.8 (12.5, 21:1)
10.8 (9.2, 19.5)
105.5 (82.7, 116.9)
24.9 (17.7,
H
12.4
16.6 (13.3,
9.8 (0.2,
58.5 (87.9, 81.7)
27.3
(10.7,
IQM total
1407.0
920.6
1275.7
3628.5
2298.0
19

---

Published in Transactions on Machine Learning Research (10/2024)
Table 9: Results comparison our proposed walker-walk distraction dataset. The reported numbers are
IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal: N, dis-casy: E, dis-medium: M, dis-hard: H. expert. medium. m-e:: medium-
walker-walk (ours)
train
eval
DrQv2+BC
IQL
DV2
Ours
N
532.8 (503.7,
244.3
30.0 (26.4,
633.2 2
E
190.5 (145.3, 217.6)
186.0 (164.7, 211.7)
41.2 (27.6, 61.7)
289.5
(269.8, 398.5)
M
82.5 (71., 137.7)
76.9 (56.5, 92.8)
26.8 (21.3 32.9)
137.9
(129.2, 234.3)
H
63.5 (52.2, 73.0)
51.5 (37.2, 65.0)
28.8 (22.8, 19288,00 69.30
96.9
(67.0) 121.2)
N
91.1 (47.3, 164.8)
59.1 (28.9, 92.8)
27.3 (26.7, 64.0)
246.4
(189.5, 276.0)
M
E
99.6 (94.4, 113.8)
50.7 (45.2, 59.1)
31.7 (27-4, 44.5)
170.6
(130.9, 213.9)
M
96.5 (65.4, 104.5)
51.6 (48.1, 91.9)
24.0 (21.2, 27.0)
154.9
(145.1, 205.5)
H
65.6
74.0)
50.3 (45.1, 56.2)
31.6 (30.5, 54.1)
79.9
(03.8;
N 48.3 (24.8,
22.4 (17.6,
38.1 (38.6, 42.8)
38.9
(88.6,
E 44.3 (28.2, 46.2)
29.7 (27.2, 33.6)
42.0 (33.2, 48.01
36.8 (36.2, 63.8)
M 38.3
(32.7, 43.5)
28.8 (26.7, 32.3)
33.4 (25.6, 43-63
40.6 (35.3, 52.9)
H 43.5
(32:3, 47.0)
35.5 (32.1, 43.3)
26.3 (24.0, 57.0)
48.5
(42.3, 63.6)
N 531.0
(489.1, 656.2)
312.8 (111.1, 476.1)
30.0 (20.3, 40.9)
544.9
(533.3, 586.1)
E
F 156.6
(138.0, 170.0)
205.1 (19433, 251 9)
41.2 (27.5, 51.7)
279.1
(957. 347.0)
M
81.8
(08.9, 115.9)
96.0 (08.2. 108.9)
26.8 (21.4, 82.9)
160.1
(182.4, 203.8)
H 66.7
(50.4,
49.0 (42.8, 64.4)
28.8 (23.0, 09.3)
73.0
(08.5, 146.6)
N 247.9
(118.9,
104.0 (56.6,
27.3 (26.7) 68.4)
420.8
(988.6
175.6
(127.2, 219.7)
119.9 (106.4, 212.5)
31.7 (27.4, 44.5)
263.66
(201.4, 317.1)
M 125.8
(102.7, 145.3)
77.7 (06.7. 102.1)
24.0 (21.2, 27.0)
227.9
(155.8 268.1)
H
92.1
(83.2; 98.7)
85.6 (96.9, 96.8)
31.6 (30.8, 63.8)
117.4
(84.5, 178.4)
N
27.8
(20.3, 60.8)
32.8 (24.1, 68.4)
38.1 (38.6, 42.8)
71.2
(54.6, 182.13
H
E
36.5
(31.9, 40.2)
35.6 (24.1, 42.7)
42.0 (34.1, 48.0)
64.4
(48 90.7)
M
49.9
(00.7, 59.9)
39.2 (27.9, 45.9)
33.4 (35.0 43.6)
68.6
(46.9, 96.4)
H
40.6
39.8 (34.6, 44.3)
26.3 (24.0, 67.1)
68.0
(63.1,
N 469.8
184.7 394.2)
30.6 (27.3,
633.4
(600.4,
231.9
(184.7, 254.0)
246.4 (165.9, 284.1)
21.0 (19.4, 98.7)
293.8
(934.4,
M
118.0 (97.4, 159.
96.7 (70.0, 165.5)
26.4 (26.2, 81.0)
156.5
(124.1, 261.7)
70.1
(41.9, k5.5)
56.2 (40.4, 74.8)
29.1 (24.8, 30.3)
104.0
(85.3, 142.23
N
229.8
(123.2, 317.7)
143.6 (39.6, 345.11
31.5 (37. 41.8)
362.9
(145.6 407.9)
M
E
163.8
(113.9, 173.6)
105.6 (76-1, 170.7)
32.0 (28.9, 39.7)
209.1
(178.5, 259.2)
M
125.2
(108.4, 160.2)
130.6 (94.7, 140.8)
35.6 (31.2, 42.1)
191.1
(164.3, 249.7)
H
69.0
70.4 (57.6, 96.0)
31.4 (30.4,
112.5
N
30.0 (26.5, 63-1)
30.6 (18.8, 41.9)
45.
02.7)
42.5 (39.2,
32.3 (26.1, 87.9)
31.4 (26.6, 86.5)
47.5 (36.9, 64.1)
47.6 (39.4, 80.2)
38.9
(34.3, 42.0)
31.1 (24.2, 40.6)
40.2 (34.0, 50.8)
56.6 (42.0, 70.0)
H
37.1
(38.8, 71.00
30.9 (29.7, 45.0)
38.4 (30.4, 48.1)
51.2 (42.0, 71.3)
IQM total
4644.7
4518.2
1204.4
6594.3
20

---

Published in Transactions on Machine Learning Research (10/2024)
Table 10: Results comparison our proposed cheetah-run distraction dataset. The reported numbers are
IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal:
N, dis-casy: E,
dis-medium: M, dis-hard: H. expert. m: medium. m-e:: medium-expert.
cheetah-run (ours)
train
eval
DrQv2+BC
Ours
N
8.4 (2.5,
114.9
E
22.4 (12.2. 86.8)
54.6 (60.9, 81.2)
E
12.7 (8.8, 17.4)
39.8 (81.4, 44.5)
5.4 (4.3, 6.2)
21.9 (15.7) 30.1)
N
9.9 (5.8, 18.8)
22.0 (7.9, 47.0)
E
5.1 (3.8, 7.7)
24.2 (21.8, 27.5)
M
M
9.3 (6.0, 14.0)
23.0 (15.3, $2.0)
H
6.1 (6-7, 7-3)
9.8 (6-1, 11:7)
N
3.5 (1.7,
17.0 (11.7, 122.1)
7.0 (5.3, 9.9)
13.4 (8.9, 17.4)
e H
8.2 (7-7; 10.4)
9.3 (6.1, 10.7)
10.6 (6.7, 15.1)
7.2 (6.0, 18.0)
132.3 (48.4, 165.9)
205.6 (105.3, 296.7)
E
90.5 (50 85.4)
149.3 (106.1, 168.9)
E
M
35.3
6, 60.8)
63.9 (58.8, 81.2)
H
19.1
(17.3, 21.1)
41.6 (33.6, 02.2)
N
46.1 (19.8,
163.2 2 (144.4,
18.4 (12.6, 26.0)
72.6 (42.0, 77.7)
m M
28.5 (15.5, 42.5)
68.5 (60.8, 73.4)
13.8 (10.0. 24.1)
37.3 (30.0, 63.13
12.3 (9.4, 16.9)
25.8
(14.4, 47.3)
m H
E
11.8 (9.5, 16.7)
23.3
(17.0, 42.2)
M
16.7 (10.4, 18.0)
22.1 (13.6,
H
18.7 (12.4.
23.4
(16.7.
N
47.3
212.4
m-- E
61.6 (42 65.0)
151.3
(108.9, 176.5)
46.0 (30.0. 60.93
91.8 (60.0, 101.3)
26.1 (19.9, 37.0)
58.2 (84-7, 78.0)
N
24.1 (11.6, 69.6)
67.0 (39.1, 191.0)
E
27.7 (16.4, 37.1)
48.0 (87.7, 66.1)
m-e M
M
23.2 (14.9. 27.6)
34.3 (18.7, 06.4)
H
15.0
26.3
2.6 (1.3,
37.2 (19.2, 82.7)
9.5 (7.2, 14.7)
20.0 (13.0, 20.3)
m- H
8.2 (6.5, 10.2)
17.4 (12.2, 26.6)
10.5 (8.9, 11.0)
17.4 (14.0. 24.6)
total
853.9
2035.0
21

---

Published in Transactions on Machine Learning Research (10/2024)
Table 11: Results comparison our proposed ball-in-cup-catch distraction dataset. The reported numbers
are IQM and 95% stratified bootstrap Cls in between parentheses. Abbreviations: normal: N, dis-casy: E, dis-medium: M, dis-hard: H. expert. medium. m-e:: medium-
ball-in-cup-catch (ours)
train
eval
DrQv2+BC
Ours
N
329.7 7 (0.0,
641.7
E
329.7 7 00.0, 797.8)
640.6 (189.4, 973.1)
M
329.7 7 (0.0, 797.8)
633.5
(185.2, 967.3)
H
329.7 (0.0, 797 8)
639.8 (189.0, 972.6)
330.3 (0.0, 798
659.3
(197.6, 991.0)
E
330.3 (0.0, 0)
557.5 (163.7, 896.8)
M
M
330.5 (0.0,
581.0 (158.2, 938.3)
H
330.3 (0.0,
560.8 (14855,
N
330.0
(0.0,
656.0 (195.6, 987.0)
E
330.0 (0.0, 796
650.5
(194.6, 982.6)
M 341.0
(13-2, 802.8)
581.1
(134.5, 935.6)
H
339.3
(5.6, 800.4)
596.5
(165.4, 927.4)
N 968.3
(918.8, 985.2)
972.0
(966.0, 992.83
E
E 541.0
(308 0, 806.1)
578.4
(384-4, 894.5)
M
369.0
(111.8, 791.9)
465.8
(213.7, $25.9)
H 300.2 (81.9, 700.2)
400.7 (108.1, 706.3)
N 965.3
(805.0,
662.0
(198.2, 995.4)
m M
E 452.6
(193.7, 801.2)
430.8 (139.6, 817.5)
M 371.2
(162.3, 824.1)
452.4 (196.7, 843.3)
H 398.7
(127.9. 779.4)
375.0 (164.9, 778.2)
484.7
(119.6, 883.2)
595.3 (189.4, 962.8)
H
E 289.7
(192.5, 827.4)
242.5 (117.4, 785.3)
M
224.1
(80.6,
241.3 (75.3,
H 245.2
(69.2,
296.0 (46.9, 807.6)
N 958.3
950.0
E 432.1
(167.4, 829.2)
574.1 (371.9, 896.9)
m-
M 362.7
(88.0, 801.6)
442.9
(189.2, 830.2)
H 313.6
(67.6, 809.1)
413.7
(117.7, 814.5)
972.7
(192 6, B84.6)
563.7
(143.4, 926.6)
m- M
E 506.0
(914.7, 832.4)
366.8 (37.4, 803-1)
M 355.6
(108.5, 780.5)
399.0 (110.7, 789.1)
H
362.0
(60.0,
344.2
633.3 (218.8, 801.2)
332.3 (0.0, 798.00
203.9 (70.4, 610.9)
322.2 (69.9, 704.6)
m-
M
289.3 33 (33.5, 670.1)
331.0
(17.,, 700.4)
H
318.4
(46.0, 719.0)
315.5 (17.1, 766.1)
IQM total
15298.4
18465.9
22

---

Published in Transactions on Machine Learning Research (10/2024)
Table 12: Results comparison our proposed reacher-easy distraction dataset. The reported numbers are
IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal: N, dis-casy: E, dis-medium: M, dis-hard: H. expert. m: medium. mi-e: medium-
reacher-easy (ours)
train
eval
DrQv2+BC
Ours
N
48.3 (20.0.
46.0
E
61.6 (30.9, 91.1)
52.5 (12.2, 212.9)
M
62.2 (31.6, 120.2)
49.7 (8.8, 213.6)
H
61.3 (31.4, 100.6)
48.6 (7.5, 213.6)
85.7 (25.8, 173.4)
46.3 (7.8, 214.4)
E
71.7 (29.8, 108.2)
45.4 (7.8, 220-4)
M
M 79.8
(34.4, 182.4)
44.8 (7.8, 216.7)
H 68.4
44.7
N 41.3 (14.2. 112.5)
46.7
E
43.7 (21.0, 86.6)
46.2 (5.2, 212.6)
M 58.6 (26-6, 83.5)
46.7 (3.2, 209-7)
H 48.8 (32-7, 114.5)
46.7 (5.2, 209.7)
N
14.6 (T.0, 780.0)
315.3 (0.0, 171.2)
E
132.2 (70-1, 463-1)
199.8 (83.0, 962.4)
E
M
45.2
868.3)
80.6 (62.9, 876.8)
H 57.7 (26.3, 197.0)
87.5 (48.0, 265.7)
N 30.0
129.6 (0.0,
E 28.7 (16.22 266.0)
12.3 457.4)
m M
M 37.6 (31.7, 312.9)
102.5 (40.3, 825.3)
H 11.7 (9.8, 910.3)
45.6 (19.7, 202.7)
0.0 (0.0, 232.0)
4.3 (2.0, 756.0)
E
24.9 (14.8, 271.4)
91.2 (40.8, 207 0)
H
M
37.5 (17.9, 259.1)
49.2 (29.0,
H 18.6 (4.8,
36.5 (19.0,
N 62.3
50.4
E
43.6 (24.7, 992.4)
84.0 (29.0, 183.4)
m-
M 63.0 (29.7, 216.3)
49.3 (29.7, 186.0)
H 59.6
(24.5, 218.8)
38.3 (21.7, 179.6)
69.1
(39:1, 763.0)
47.9 (36.7, 64.1)
m- M
E
54.7
(20.7 285.2)
78.5 (30.6, 109.3)
M
28.0 (17.8, 205.4)
86.9 (48.6, 189.1)
H
68.5
66.2
N
0.0
752.8)
24.3 (7.8, 723.8)
E 93.1
(40.7, 338.4)
69.5 (41.0, 261.3)
m-e
M
42.0
(27.0, 310.6)
21.1
(10/7, 301.8)
H
38.1
(20. 4. 109.8)
57.1
(33.3, 296.3)
IQM total
1792.1
2432.6
23

---

Published in Transactions on Machine Learning Research (10/2024)
Table 13: Results comparison our proposed reacher-hard distraction dataset. The reported numbers are
IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal: N, dis-casy: E, dis-medium: M, dis-hard: H. expert. m:: medium. m-e:: mepium-
reacher-hard (ours)
train
eval
DrQv2+BC
Ours
N
0.0 (a.0.
1.0
E
0.0 (0.0,
1.0 (0.0.
e E
M
0.0 (0.0, 4.0)
1.0 (0.0.
H
0.0 (0.0, 4.0)
1.0 (0.0, 3.0)
N
0.0 (0.0, 8.2)
1.7 (0.0. 4.01
E
0.0 (0.0, 3:2)
1.7 (0.0, 4.0)
M
M
0.0 (0.0. 3:2)
1.7 (0.0. 4.0)
H
0.0 (0.0,
1.7 (0.0,
N
0.0 (0.0, 4.0)
1.3 (0.0,
E
0.0 (0.0.
1.2 (0.0. 8.3)
e H
M
0.0 (0-9, 4.03
1.3 (0.0, 6.3)
H
0.0 (0.0, 4.0)
(0.0. 8.1)
N
0.7 (0.4, 20.3)
2.3 (0.0. 8.0)
E
1.0 (0.9, 32.6)
30.2 (13.6) 52.1)
m E
M
1.1 (0.5,
8.1 (0.5, 27.0)
H
1.4 (1.1,
2.5 (0.6,
N
0.0 (0.0.
0.0 (0.0,
E
0.3 (0.1,
0.5 (0.3,
m
M
0.4 (0.3, 1.0)
15.2 (5.1, 54.13
H
12.0 (1.1, 27.0)
0.8 (0.5. 4.91
N
0.0 (0.0, 0.0)
0.0 (0.0. 9.2)
E
1.4 (0.T, 2.9)
2.7 (0.9, 8.0)
m H
M
1.5
(1-1, 3.4)
0.6 (0.3,
H
0.9 (0.0.
32.1 (18.1.
N
0.1
0.0
E
3.1 (1-1,
8.6 (4.7,
m-e
M
0.7 (0.3, 5.8)
7.0 (0.3, 28.9)
H
4.1 (3-1, 21.8)
16.6 (5.8, 26. 3)
N
0.4 (0.3, 35.2)
0.0 (0.0, 42.4)
E
3.6 (2.1,
10.7 (6-1, 36.7)
m-e M
M
3.9 5.0)
10.0 (4.8,
H
5.8 (2.5.
12.4
N
0.0 (0.0,
25.3 (0.0, 638.8)
3.6 (2.4.
11.0 (2.4, 42.1)
m-
M
1.5 (1.4, 9.4)
11.7 (1.5, 29.0)
H
4.5 (2.7, 25.0)
7.4 (3.3, 17.03
total
52.0
231.6
24

---

Published in Transactions on Machine Learning Research (10/2024)
C Full experiment results with visual generalisation methods in RL adapted to offline settings
Table 14: Results comparison with related domain adaption methods in RL. The reported numbers are IQM and 95% stratified bootstrap CIs between parentheses.
cheetah-run (V-D4RL)
train
eval
PAD
ILA
SVEA
Ours
N
55.3 (33.9, 67.8)
133.8 (1015, 158.6)
32.8 (29.0, 38-1)
233.7 (197.9, 258.8)
E
25.7 (20.5, 27.2)
45.2 (41.4, 65.7)
25.5 (18.4, 35.5)
100.8 (83.6, 113.7)
E
M
14.7 (13:4, 15.8)
53.0 (37 T. 64.33
29.0 (23.8, 37-3)
42.0 (35.6, 73.5)
H
18.4
(10.4, 19.3)
32.3 (30.0, 88.7)
46.9 (40.8, 49.9)
33.6 (22.3, 39.1)
N
24.8
115.3 (67.9)
37.7
76.5
110:3)
E
17.2 (13.2,
54.9 (45 57.9)
42.5 (34.5, 47.9)
31.1 (26.0, 51.13
e M
M
13.9 (10.0, 16.7)
30.9 (28.1, 41.9)
39.6 (36.2, 48.6)
27.7 (24.7, 41.6)
18.3 (12.1, 20.6)
30.1 (26.8, 83.9)
40.2 (38.6, 47.6)
20.1 (10.9, 27.3)
N
17.4 (13.8, 26.0)
99.1 (87.4, 120.2)
42.3 (18.4, 48.5)
24.3 (17.8. 44.8)
E.
14.6 (12.5, 15:7)
29.1 (17.9, (7.7)
39.1 (21.9, 47.8)
18.8 (12.6, 22.3)
H
M
16.9
(13.2, 20.4)
39.2 (14.3, 48.9)
37.9 (17.6, 47.9)
16.2 (12.9, 18.8)
H
16.2
23.5 (10.6,
49.6
12.4 (9.8,
N
23.3 (18.4,
102.5 (90.1,
21.5
246.9 (219-7,
m E
E 17.8
(14.6,
37.1 (29.6,
11.2
152.0 (122.0,
M
11.1 (9.2, 18:4)
42.4 (28.9, 68.7)
12.6 (a.4, 17.6)
68.2
(49.0, 83.35
H
15.4 (11.9, 18.0)
34.7 (38.7, 40.5)
45.1 (30.9, 61.3)
34.0
(26 41.5)
N
27.2 (21.5, 30.3)
93.5 (80.1, 105.1)
32.5 (25.5, 34.2)
116.4 (68.0, 154.3)
E
12.2 (9.5, 15.8)
55.0 (47.0. 0, 61.4)
21.5 (16.2, 23.1)
45.5 (36.0, 97.0)
M
M
9.2 (8.0, 12.9)
41.7 1.0, 45.5)
23.1 (18.1, 20.0)
50.3 (27.6, 52.9)
H
11.0 (9.0, 16.2)
17.7 12.2, 21.4)
39.8 (27.7, 48.4)
26.5 (19.7;
N 12.5 (8.0, 17.6)
109.9 (81.4, 148.9)
27.6 (21.6, 31.4)
36.2 (27.6, 47.7)
E
13.6 (10.5,
46.9 (39 53.5)
25.8 (17.7, 28.3)
18.5 (16.0, 24.8)
m
M
16.2 (13.9, 19.4)
33.1 (24.1. 47.0)
20.4 (17.1, 23.8)
18.3 (14.4, 21.6)
H 15.6 (13.0, 18.0)
24.3
(20.8. 37.9)
18.2 (10.0, 25.9)
16.2 (15.6, 19.9)
N 65.8
(43.1, 86.3)
100.4 (00.7, 124.0)
39.2 (28.7, $9.8)
229.7
(160.0, 251.5)
m-c E
E 28.7
(24.7, 39.8)
13.7 (8.7, 28.3)
28.1 (28.0, 40.8)
130.6
(102.0, 161.0)
M 17.5
(13.8, 19.3)
25.9 (18.6, 31.7)
36.8 (26.7, 48.4)
81.0
(66.3, 100.3)
H 22.4
(16.3, 24.8)
29.0 (20.1, 40.1)
57.9 (42.6, 62.0)
55.4 (42.2, 60.5)
N
23.6 (L7.6.
155.3 (101.6, 160.7)
51.2
84.4
116.0)
m-c M
19.2 (16.4,
60.0 (33.6, 68.8)
35.0
(25.6, 42.4)
60.5 (33.0, 72.8)
M
14.1 (11.9. 19.8)
34.2 (29.1, 39.5)
35.8 (28.4, 55.7)
49.2 (37.0, 68.9)
H
21.3 (16.7, 26.8)
26.4 (23.6, 33.0)
41.2 (95-7, 56.2)
20.2 (13.8, 30.8)
N 16.8 (14.2, 38.2)
96.4 (89.3, 108.1)
31.0 (22.6, 37.0)
39. 2 (16.9, 60.8)
m-e H
E 16.3 (11.4, 20.8)
38.3
(22 56.1)
25.9 (20.1, 28.0)
29.4 (21.0, 41.2)
M
18.8
(13.2, 22.0)
40.5 (28. 45.7)
25.3 (21.4, 82.1)
24.9 (17.7, 28.6)
H
20.3
(12.9 23.3)
23.4 (22.5,
21.5
27.3 .3
total
723.3
1968.7
1191.3
2298.0
25

---

Published in Transactions on Machine Learning Research (10/2024)
D Additional experiments on Data Usage
In the section, we show the results of an alternative formulation where the encoder of the bascline algotihm uses both normal dataset and distraction dataset during training, DrQv2+BC denoles the baseline algorithm, DrQv2+BC, both data denotes the baseline algorithm where its encoder has access to both distracting data and normal data, and Ours denotes our proposed algorithm, as shown Table 15 Furthermore, we can train another version where our proposed algorithm is trained on two distracting datasets, shown in Table 16
Table 15: Results comparison on cheetah-run distraction dataset (V-D4RL). The reported numbers are IQM and 95% stratified bootstrap Cls in between parentheses. Abbreviations: normal: N, dis-easy: E, dis-medium: M, dis-hard: Н. e: expert. mr medium. m-e: medium-expert. Our method performs better when the quality of data is mixed (medium-cxpert), while DrQv2+BC, both data performs much better normal observation.
cheetah-run (V-D4RL)
train
eval
DrQv2+BC DrQv2+BC, both data
Ours
N
108.7 (42.0, 182.9)
579.2 (542.4, 661 8)
233.7 (197.9, 256.8)
E
34.2 (80.5,
56.6 (32.2. 78.9)
100.8 (83.6, 118.7)
E
M
13.7
23.3 (18.0,
42.0
H
9.7 (7.4,
12.2 (10.4, 14.0)
33.6
18.9 (1.2, 61.7)
569.0 (432.5, 600
76.5 (28.2, 110.3)
33.3 (18.5, 40.7)
36.3 (29.7, 44.8)
31.1 (26.0, 61.13
e M
M
28.3 (22.7, 59.33
27.0 (18.9, 36.4)
27.7 (24.7, 41.51
H
11.1 (9:3, 16.9)
12.7 (8.6, 18.0)
20.1 (15.9, 97.3)
N
1.5 (1.0, 3.4)
419.0 (148.22 03
24.3 (17.8, 46.8)
e H
E
9.1
(6:4, 14:2)
21.8 (12.5,
18.8 (12.6, 28.2)
M
9.8 (7.9,
21.9 (14.4,
16.2 (12.9,
11.1
(7-8, 17.2)
16.6 (13-4, 27.5)
12.4 (9.8, 18.5)
N
131.0
(135.0, 143.2)
562.4 (565.0, 077.0)
246.9 (219.7, 279.7)
99.5
(65.6, 110.4)
96.5 (78.0, 106.2)
152.0 (122.0. 178.9)
m
M
41.4
(32:4, 56.63
66.5 (40.4, 70.6)
68.2 (59.0, 63.5)
18.6
(13.7, 22.6)
25.4 (19.3, 26.4)
34.0 (26.0, 41.5)
N
125.9
(81.9, 146.0)
548.5 4 (541.4, 680.9)
116.4 (68.0, 154.9)
m M
E
37.6
(81.0, 68.2)
63.2 (58.1, 91.8)
45,5 (86.0,
M
40.3
49.7 (37.8, 89.1)
50.3
H
21.2 (14.4,
23.0 (17.2, 37.1)
26.5 (19.7.
60.3
(43.5, 70.7)
531.6 (516.1, 649.6)
36,2 (27.6, 47.7)
26.3
(19.3, 50.6)
65.2 (45.7, 79.1)
18.5 (16.0, 24.8)
m
M
24.5
(20.4, $2.0)
40.9 (35 66.8)
18.3 (14.4, 21.61
H 33.7
(21-1, 37.6)
26.4 (21.0, 36.8)
16.2 (13.6, 18.9)
N 115.1
(51-4, 177.6)
93.3 (35. 142.8)
229.7 (160.8, 251.6)
m- E
E
83.6
(97.9,
69.6 (03.8, 129.7)
130.6 (102.0, 161.0)
M
35.0
(33.1,
28.2 (20.8,
81.0
H
18.1
(14.2,
20.6 (16.0, 20.6)
55.4 (42.2,
69.6
(48.1, 87.2)
88.2 (35.4 137.2)
84.4 (36.2, 118.0)
32.5 (22.3, 56.33
1 (27.0, 54.0)
60.5 (33.0, 72.9)
m-e M
M
34.4 (21.3, 49.4)
23.5 (20.7, 26.1)
49.2 (87.0, 66.9)
H
13.5 (12.5, 18.6)
13.5 (31.8, 20.3)
20.2 (13.8, 30.9)
N
17.3 (9.9, 20.2)
7.7 (3.0, 18.2)
39.2 (10.9, 60.8)
m-c H
E
14.2
16.5)
14.5 (11.3, 19.3)
29.4
M
11.6 (10.4,
16.3 (14.4, 20.0)
24.9 (17.7, 28.5)
12.4 (9.1. 15.7)
15.1 (10.8, 19.4)
27.3 (16.7, 33.6)
IQM total
1407.0
4318.5
2298.0
26

---

Published in Transactions on Machine Learning Research (10/2024)
Table 16: Results comparison cheetah-run distraction dataset (V-D4RL). The reported numbers are IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal: N, dis-easy: E, dis-medium: M, dis-hard: H. e: expert. m: medium. m-c: medium-expert. Two distraction data is trained on easy distraction and medium distraction, with dropblock drop rate p = 0.3. Without tuning the dropblock probability p, we can observe that even when using two different difficulties of distraction observations data for training, our proposed method can achieve similar performance compared training normal and distraction observation.
cheetah-run (V-D4RL)
train
eval
Ours, - (p=0.3) Ours, normal (dim-0))))(p-0.3))
Ours normal+dis (best p)
N
202.6 (170.5, 231.6)
233.7 (197.9, 256.8)
233.7 (197.9. 256.8)
E
85.3 (63.6, 99.9)
88.3 (72.1, 107.4)
100.8 (83.6, 113.7)
E
M
44.0 (27.0, 48.2)
44.9 (34.3, 67.1)
42.0 (a5.0, 78.6)
H
30.9 (26.9, 42.9)
21.4
33.6
39.1)
N
78.5 (58.6, 102.3)
76.5 (28.2.
76.5 110.3)
E
30.3 (25.0, 36.8)
32.8 (29.9 07.4)
31.1 (25.0, 51.13
e M
M
29.6 (28.7, 40.9)
23.0 (17.4, 27.4)
27.7 (24.7, 41.6)
H
16.3 (11.6, 23.9)
20.1 (9.7. 24.0)
20.1 (15.9, 27.3)
13.4 (7.9, 18.2)
24.3 (17.8, 44.5)
24.3 (17.8, 44.8)
E
7.7 (5.8, 11:6)
16.0 (12.8, 21.0)
18.8 (12.8, 22.2)
H
M
11.9 (10.4, 18.0)
16.2 (12.9,
16.2 (12.9, 18.8)
H
10.8
9.8 11.9)
12.4
N
222.1 (123.8, 918.5)
246.9 (219.7, 279.7)
246.9 (219.7. 279.7)
E
129.1 (106.7, 142.6)
152.0 (122.0, 178.9)
152.0 (122.0, 178.9)
m
E
M
70.8 (60.2, 83.0)
64.0 (52.2, 17.8)
68.2 (49.0, 63.3)
38.4 (30.4, 50.1)
30.7 (21.6, 35-7)
34.0 (36.9, 41.5)
86.3 (78.5, 121.8)
116.4 (68 0, 164.8)
116.4 (68.0, 164.8)
E
52.8 (43.3, 59.5)
45.5 (36 07.9)
45.5 (35.0, 87.88
m M
M
40.0 (81.1, 40.9)
30.7 (21.1, 40.4)
50.3 (27.5, 62.9)
H
21.0 (18.1,
23.9
26.5 38.0)
N
24.3 (11.3, 40.0)
36.2 (27.6. 47.7)
36.2 (27.6,
E
8.4 (6.6, 25.7)
14.8 (8.T, 21.8)
18.5 (16.0, 24.8)
m
H
M
8.7 (6.6, 23.2)
12.1 (7.6, 14.2)
18.3 (14.4, 21.6)
H
7.9 (4.8, 11:2)
11.4 (T.T. 20.4)
16.2 (15.4, 19.9)
218.5 (194.5, 285.2)
229.7 (100.6, 251.6)
229.7 (180.0, 251.5)
E
145.0 (129.7, 178.9)
110.6 (83.2, 141.2)
130.6 (102.0, 161.0)
E
M
81.9 (08.1, 95.9)
81.0 (60.3, 109.3)
81.0 (68.3, 109.3)
H
50.2 (39.5, 61-0)
55.4 00.5)
55.4 (42.2, 00:5)
N
119.5 (99.8, 142.2)
84.4 (36.2. 116.0)
84.4 (36.2, 116.0)
E
45.0 (36.1, 60.4)
60.5 (33 12.8)
60.5 (33.0, 72.6)
m-e M
M
39.1 (32.4, 51.2)
43.9 (22.0, 59.5)
49.2 (37 06.9)
H
20.5 (18.0, 30.1)
13.4 (8.0, 23.7)
20.2 (13.8, 30.8)
N
19.5 (7.8, 29.3)
39.2 (16.0. 60.5)
39.2 (16.9, 00.8)
E
29.5 (19.6, 42.8)
21.5 (14.7, 48.6)
29.4 (21.0, 41.2)
m-e H
M
16.3 (8.9,
19.0
24.9
H
24.0 (12.4, 28.8)
12.7 (10.9,
27.3 (16.7, 38.6)
IQM total
2080.1
2162.9
2298.0
27

---

Published in Transactions on Machine Learning Research (10/2024)
E
Ablation on Distractions
Table 17: Distractions ablation results on V-DARL chectah-run distraction expert benchmark. The reported
mumbers are IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations for dataset type: expert: Abbreviations for visual distractions difficulty: normal: N, easy: E. medium: M, hard: H.
train
eval
type of dis
DrQv2+BC
AWAC+BC
IQL
Ours
E
background
37.9 (22.8, 58.1)
49.8 (29.6, 72.7)
40.1 (25-4, 197.7)
135.6 (110.5, 21166
e
E
E
viewpoint
67.1 (20.4, 01:2)
32.8 (20.4, 64.9)
45.3 (23.0, 80.8)
137.0 (101., 150.4)
e E
E
colour
102.3
127.3)
49.0 127-7)
67.7
203.0 (187.8.
e E
E
all
34.2 20.5,
31.4 (18.4, 45.2)
50.0
100.8 (83.6,
e E
background
43.8 (25.8, 59.3)
37.1 (24.1, 98.8)
69.8 (34.8, 114.33
173.2 (128.1, 212.2)
e E
viewpoint
34.7 (16.1, 44.8)
17.2 (9.7, 21.0)
17.6 (9.5, 22.3)
60.3 (47.4, 72.13
E
colour
98.1 (42.0, 121.6)
55.6 (96.0, 118.4)
61.8 (47.0, 69.9)
170.9 (180.7, 207.3)
E
M
all
13.7 (11-6, 21.7)
7.8 (5.4, 25.1)
17.3 (7.6, 22.1)
42.0 (35.6, 73.5)
E
H
background
55.6 (30.1, 68.6)
20.6 (12/78, 34.2)
35.2 (18.2, 66.4)
127.3 (97.8, 182.9)
E
viewpoint
12.8 17-7)
12.4 (7 18.1)
9.9 (6:31, 17-1)
46.9 (32.9,
e E
H
colour
70.1 (38.4, 95.6)
52.1 (36.80,
53.9 (82.07, 01.7)
170.8 (187.6. 187.4)
e E
all
9.7 (7.4, 12.77
6.6 (4.5,
6.8 (6.0, 14.7)
33.6 (22.35 59.13
M
background
23.7 (13.5, 49.3)
8.4 (7.5, 19.1)
43.0 (20.0,
29.6 (16.4, 38-7)
M
viewpoint
20.8 (9.4, 36.7)
25.3 (6.0, 60.2)
118.5 (85.9, 127.9)
95.8
(28.1, 107.0)
M
colour
22.6 (13.0, 42:2)
25.8 (8.8, 82.1)
319.8 (272.1, 389.1)
84.5 (32:2, 105.8)
M
all
33.3 (18.5, 40.7)
15.0 (9.3, 91.2)
15.2 (5.9,
31.1 (25.0, 51
M
M
background
45.6 (31.6, 61.3)
23.9 (11.5, 44.3)
31.2 (15.5, 6T.1)
27.9 (18.7, 86.9)
M
M
viewpoint
23.3 (18.0, 86.0)
14.0 (3.7, 85.4)
57.2 (44.7, 69.8)
62. 1 (31.7, 83.13
M
M
colour
31.8 (11.9,
28.2 (T.0.
251.4 (184.8,
77
M
all
28.3 (28.7, 39.3)
17.9 (8-8,
11.4 (5.6, 13.9)
27.7 (24-7,
M
background viewpoint
31.1 (19.2, 40.3)
20.7 (16.2, 23.5)
23.7 7 (21.6, 41.8)
23.7 (19.7, 29.0)
M
12.5 (1.4, 19.6)
6.4 (4.4. 35.00
24.8 (13.9, 29.4)
36.6 (21.4, 39.9)
M
colour
36.2 (20.1, 42.1)
29.2 (10.3, 72.61
200.6 (192.8, 221.3)
65.1 (42.4, 91.0)
M
all
11.1 (9.3, 14.8)
7.1 (5.7, 13.1)
5.3 (4.5, 8.8)
20.1 (18.9, 27.3)
H
background
12.4 (9.4, 22.1)
14.0 (11.0, 26.1)
11.6 (10.6, 18:7)
20.6 (17.8, 28.7)
H
E.
viewpoint
4.1 (2.4,
3.1 (2.0,
8.9 (3.6, 11.9)
16.9 (13.2, 16.8)
H
E
colour
4.4 (2.8, 9.1)
3.9 (2:2,
11.8
20.6 (15.5, 39.2]
all
9.1 (8.4, 14.3)
14.4 (1..9, 16.8)
8.3 (6.2, 12.6)
18.8 (12.6,
background
12.2 (9.0, 29.6)
22.2 (14.6, 29.1)
9.9 (7.1, 11.3)
25.8 (17.9, 64.9)
viewpoint
5.9 (3.4, 7.7)
3.9 (2.0, 9.03
5.0 (2.6, 7.6)
12.6 (11/2, 22.13
colour
5.5 (2.7, 8.2)
5.3 (2.8, 8.1)
10.9 (7.4, 21.4)
24.5 (16.6, 32.9)
H
all
9.8 (7.9, 17.83
12.4 (10.9, 13.9)
11.5 (8-1, 15.5)
16.2 (12.9, 18:8)
H
background
12.4 (10.8, 20.1)
13.1 (10.2, 17.7)
11.3 (7.8, 26.9)
22.6 (16.7, 27.1)
e
H
H
vicwpoint
5.6 (3.5,
4.6 (3.5,
5.9 (3.3, 16.8)
21.3 (14.7,
e
eH
H
colour
7.0 (5.2, 10.5)
5.6 (9.3,
10.7
18.5 (15.3, 26.7)
eH
all
11.1 (7.8, 17.2)
13.6 (10.1, 19.0)
8.7 (6.0, 11.0)
12.4 (9.8, 18.5)
28

---

Published in Transactions on Machine Learning Research (10/2024)
Table 18: Distractions ablation results V-D4RL cheetah-run distraction medium benchmark. The reported
numbers are IQM and 95% stratified bootstrap Cls in between parentheses. Abbreviations for dataset type:
medium: m. Abbreviations for visual distractions difficulty: normal: N, easy: E, medium: M, hard: H.
train
eval
type of dis
DrQv2+BC
AWAC+BC
IQL
Ours
E
E
background
129.7 (11511 163.5)
101.5 (62. 193.7)
165.4 167-5)
293.1
(258.5, 317.0)
m
E
E
viewpoint
69.9 (61.1, 103.1)
72.1 (42.7, 105.3)
59.8 (26.8, 99.0)
148. 1 (119.4, 172.9)
m
colour
125.7 (115.88, 136
84.0 (35.9, 130.3)
117.5 (44.3, 186.7)
223.8
(208.3, 301.))
m
all
99.5 (63.6, 110.4)
56.4 (44.0, 77.73
98.8 (63-1, 118
152.0 (122.0, 178.0
M
background
53.9 (35.6, 66.6)
8.8 (7:2, 30.0)
10.9 (2.6, 30.5)
30.3 (20.9, 70.3)
E
M
viewpoint
105.6 (86.0, 120.0)
68.4 (49.8, 88.9)
33.4 (21.4, 52.3)
98.7 (62. (02.0, 117.7)
E
M
colour
95.6 (90.1, 108.6)
76.4 (62.2, 90.03
45.4 (82.1, 92.7)
103.1 (63.5, 121.2)
m
E.
M
all
41.4 (32.1, 56.8)
23.2 (17:2,
26.1 (20.0, 70.4)
68.2 (48.0, 63.3)
m
E
background
26.8
25.6 (15.2,
8.8
13.7
m
viewpoint
42.9 (34.2, 02.6)
14.0 (10.5,
14.5 (10.4, 33.9)
27.2 (22.8, 26.4)
m
colour
49.4 (38.3, 64.3)
14.9 (4.8, 38.3)
21.3 (9.5, 80.7)
38.0 (26.8, 48.4)
m
all
18.6 (12:7, 22.6)
14.4 (8.7, 19:5)
11.2 (7.6, 18.4)
34.0 (28 8, 41.3)
M
background
153.7 (200.4, 180.1)
111.4 (47.2. 122.0)
177.5 (96.8, 202.9)
308.7 2 (286.4, 351.5)
M
viewpoint
41.1 (30.9, 65.8)
42.5 (82.0, 69.7)
34.6 (20.4, 60.7)
84.1 (58.1, 97.0)
M
E
colour
115.7 (310.3, 148-77
94.3 (48.1, 192.3)
101.7 (37-3, 148.1)
244.4 (207. 290.8)
M
E
all
37.6 (31.0, 53.2)
18.6 (11.0,
27.0 (17.1, 41.6)
45.5 (36.0, 07.9)
M
M
background
49.5 (44.9,
20.8 (14.8.
34.6 64.0)
61.1 (40.7, 76.5)
M
viewpoint
72.4 (57.3, 86.4)
56.0 (47.4,
20.6 (18.8, 71.3)
76.3 (48.5, 90.0)
colour
94.1 (65.7, 98.7)
66.2 (83.0,
46.6 (36.9, 85.4)
88.0 (58. 88.8. 108.5)
M
all
40.3 (23.6, 52.5)
14.3 (8.7, 17:7)
15.6 (13.5, 19.8)
50.3 (27.5, 52.9)
M
background
27.8 (21.0, 80.5)
22. (12.3, 84.4)
8.4 (6.7, 17.0)
12.3 (7.3. 31.6)
M
H
viewpoint
29.2 (27.4, 60.3)
13.3 (8.8, 38.4)
25.4 (12.3, 34.6)
23.4 (18.5, 24.9)
M
H
colour
51.2 (32.7, 68.8)
12.4 (0.2, 40.4)
18.5 (9:2; 91.8)
35.4 (27.9, 48.9)
M
H
all
21.2 (11.4, 29.6)
9.8 (7.2, 20.6)
8.1 (7.2, 29.1)
26.5 (18.7, 36.0)
background
121.6 163.5)
76.5 108.0)
105.4
143.8)
274.8
viewpoint
26.0 (20.6, 6, 42.4)
28.0 (15.1,
12.1 1 (6.3.
28.7 (17.8, 39.0)
colour
120.4 (200.5, 128.9)
88.5 (87.3, 182.4)
69.5 (36.2, 124.6)
245.0 (162.3, 279.2)
all
26.3 (19.3, 88.6)
21.8 (13.0, 31.3)
8.7 (6.6, 14.5)
18.5 (16.0, 24.6)
M
background
42.7 (3T.8, 47.7)
19.8 (11.0. $1.99
38.1 (25.7, 48.3)
39.8 (25.4, 62.7)
M
viewpoint
41.6 (28.6, 60.9)
32.2 (20.0, 49.33
12.5 (6.7, 18.8)
38.3 (16.1, 64.9)
colour
85.5 00.0)
67.3 (92.9, 76.2)
46.8 (28.6, 49.9)
75.6 (64.8, 04.5)
all
24.5 (20.4, 02.0)
14.0 (7.1, 21.00
10.6 (7.6,
18.3 (14.4,
H
background
25.5 (19.3, 28.6)
20.9 (14.7.
8.9 17.83
16.8 (12.0, 20.4)
viewpoint
27.4 (25.0, 53.4)
10.6 (6.9, 18.2)
21.2 (11.4, 31.2)
23.9 (20.2, 34.6)
colour
43.1 (26.7, 56.5)
13.2 (6.8, 29.7)
19.2 (7.7, 26.5)
27.0 (24.2, 2, 29.5)
all
33.7 (21.1. 97.6)
12.8 (9.3, 10.2)
15.6 (11.7) 21:2)
16.2 (16.6, 19.9)
29

---

Published in Transactions on Machine Learning Research (10/2024)
F Ablation on Effectiveness of Dropblock
In this section, we investigate further the effect of dropblock on baseline algorithms. Firstly, the full results Secondly, an ablation
of ablation study on dropblock for our proposed algorithms is shown in Table 19 study on using only dropblock with baseline algorithm DrQv2+BC is shown in Table 20 We can note that adding dropblock on baseline algorithm can improve the performance by a fair margin, but the full proposed algorithm consisting of dropblock and gradient reversal layer the most performant.
Table 19: DropBlock ablation results on V-D4RL cheetah-run distraction dataset. The reported numbers are IQM and 95% stratified bootstrap Cls in between parentheses. Abbreviations for dataset lype: medium: m, expert: e, medium-expert, m-e. Abbreviations for visual distractions difficulty: normal: N, easy: E, medium: M, hard: H. For example, 'c H' equals 'expert Hard'.
cheetah-run (V-D4RL)
train
eval
0.0
p= 0.1
0.2
p 0.3
p 0.5
N
114.1
(93.2, 176.2)
176.0 (149.2, 216.5)
153.6 (01.0, 197.3)
233.7 (197.9, 250.8)
176.6 (82.3, 254.7)
E
40.0 (26.5, 58.1)
65.3 (52.9, 85.7)
100.8 (83.6, 113.7)
88.3 (72.1, 107.4)
71.0 (61.0, 89.8)
e
M
17.2 (16.0, 28.1)
46.1 (30.0, 60.2)
42.0 (36.6, 73.5)
44.9 (34.3, 67.1)
43.7 (30.5, 68.9)
H
15.2
(11:2, 21.7)
22.5 (18.1, 99.00
22.5 (20.4, 61.4)
21.4 (17.3, 28.1)
33.6 (22.3, 39.1)
N
81.8
(18-7, 105 ()
46.9 (34-4, 78.4)
29.9 (14.5, 70.2)
76.5 (28 3, 110.3)
33.9 (15.9, 60.7)
E
49.1 (80.3, 68.7)
30.2 (22.1, 85.5)
31.1 (26.0, 61.1)
32.8 (20.2, 87.4)
30.8 (25. 60.0)
M
M
30.2
(26.1, 47.4)
27.7 (17.4, 38.5)
22.6 (15.1, 30.6)
23.0 (17.4, 27.4)
27.7 (24.7,
H 21.6 (10.8, 29.2)
11.2 27.3)
20.1 (16.9, 27.3)
20.1 (9.7. 24.0)
17.7
N
16.1
(8.0, 29.9)
9.4 (5.9. 15.7)
11.1 (6.6,
24.3 (17.8, 44.8)
23.7 (17.7, 38.7)
E
19.0 (15.6, 21.3)
18.8 (12.6, 22.3)
10.4 (7.5, 27.3)
16.0 (12.8, 21.0)
9.2 (6.5, 13.5)
M 14.7
(11-9, 22.6)
15.6 (9.9, 17:3)
10.8 (7.5, 19:3)
16.2 (12.9, 18.8)
9.0 (6.4, 16.4)
H 16.2
(13.6, 26.0)
11.5 (9.0, 19.3)
12.4 (9.8, 18.5)
9.8 (7.5, 11.93
9.8 (G.1, 19.0)
134.1
(100.6, 174.2)
181.3 (70.5, 240 6)
101.3 (82.6, 141
246.9 (219.7, 279 m
104.4 (41.0, 280.8)
E
E
82.0
(89.0, 100.00 .))
127.2 2 (105.2, 163.1)
123.1 (107.3, 182:2)
152.0 (122.0, 178.0)
11 114.4 (91.3, 180.9)
M 66.0
(31-7, 73.8)
68.2 (19-0,
58.2 (53.6, 63-4)
64.0 (52.2,
44.1
H 28.8
(27.5, 39.3)
32.8 (28.3,
28.0 (20.4, 32.4)
30.7 35.7)
34.0
N 124.7
(59.7, 134
96.3 (82.4,
79.9 (40.2, 93.2)
116.4 (68 154.3)
62.8 (23.3,
E 49.4
(81:2, 60.4)
49.9 (31.4, 60.0)
35.5 (28.4,
45.5 (36.0, 97.9)
48.5 (38.5, 66.2)
m M
M 36.3
(27.1, 67.
31.3 (19.9, 41.7)
50.3 (27.8, 52.9)
30.7 (21.1, 40.4)
35.4 (31.2, 40.3)
H 25.5
(20.7, 29.8)
8.0 (6.0, 26.0)
26.5 (19.7, 38.0)
23.9 (15.4, 97.4)
12.8 (8.8. 16.4)
N
35.7
(25.8, 81.3)
8.6 [1.6, 26.9)
11.9 (7.5, 34.2)
36.2 (27.6, 47.7)
19.3 (10.6, 36.0)
H
E 24.5
(22.2, 31.9)
18.5 (16.0, 24.9)
11.7 (7.9, 13.1)
14.8 (8.7, 21.8)
7.9 (4.6, 20.0)
M
22.0
28.8)
18.3 [14.4,
12.7 15.3)
12.1 (7.6. 14.2)
6.9 (8.5,
H
19.1
(14.9, 23.4)
16.2 (15-6,
13.3 (8-4,
11.4 (7.7, 20.4)
9.6 (6.6, 15:5)
N 85.4
(46.4, 119.G)
185.9 (172.9, 248.9)
171.7 (94.4, 216.1)
229.7 (160.6, 251.6)
194.3 (76.8, 208.5)
E 71.7
(86.9, 89.9)
130.6 (102.0. 181.0)
121.7 (99.9, 141
110.6 (83.2, 141.2)
96.8 (91.9, 118.6)
m-e
M
42.4
(31.9, 48.8)
68.0 (56.7, 64.9)
94.0 6, 102.6)
81.0 (66 3. 109.3)
77.0 (65 5, 62.0)
H 34.3
(32.0, 36.0)
50.9 (81.8, 68.4)
49.1 (48.0, 65.4)
55.4 (42.2, 60.3)
42.3 (38.7, 40.7)
N 63.7
(86.7, 116
33.5 (14.3, 115.4)
43.0 (4.5, 108.4)
84.4 (36 3. 116.0)
60.2 (26 70.6)
m-e M
E
33.4
(27.8, 47.4)
27.3 (22.6, 88.2)
40.5 (81.1, 65.0)
60.5 (38.0, 72.8)
53.0 (39.8, 90.6)
M
45.0
(31.9, 59.8)
29.7 (21.0,
31.2 (25.1, 44.0)
43.9 (22.0, 59.5)
49.2
H
25.3
(22.1, 27.6)
13.2 (10.4,
20.2 (13.8, 30.8)
13.4 4 (9.0. 25.7)
15.3 (12.1, 39.6)
N
21.9
(15.7, 39.4)
6.0 (2.4, 7.7)
21.5 (17.1, 39.7)
39.2 (16.9, 60.8)
24.4 (11.3, 50.0)
m-c H
E
12.0
(10.1, 21.9)
16.4 (18.0, 21.0)
29.4 (21.0, 41.2)
21.5 (14.7, 48.0)
18.0 (15.0, 23.5)
M
14.9
(13.9 18.0)
18.6 (11.0, 22.5)
24.9 (17.7, 28.3)
19.0 (11.2, 26.2)
15.0 (1L.S, 19:2)
H
17.1
(11.5, 23.9)
18.1 (13.2, 22.
27.3 (16.7, 83.6)
12.7 (10.9, 22.9)
12.8 (9.5, 21.0)
total
1550.4
1736.0
1694.2
2162.9
1645.1
30

---

Published in Transactions on Machine Learning Research (10/2024)
Table 20: Results comparison on cheetah-run distraction dataset (V-D4RL). The reported numbers are IQM and 95% stratified bootstrap CIs in between parentheses. Abbreviations: normal: N, dis-casy: E, dis- medium: M, dis-hard: H. e: expert. mr medium. m-e: medium-expert. To save space, we note DrQy2+BC as DB, and DrQv2+BC with dropblock (drop probability p) noted as DB-p.
cheetah-run (V-D4RL)
train
eval
DB
DB-0.1
DB-0.2
DB-0.3
Ours
108.7 (42.0, 182.9)
70.6 (20.2. 113.4)
33.5 (19.9, 92.7)
88.6 (48.3, 198.6)
233.7
(197.9, 250.8)
E
34.2 (30.5, 47.5)
42.2 (35.9, 75.1)
52.9 (40.3, 65.5)
60.1 (51.9, 74.5)
100.8
(83.6, 113.7)
M
13.7 (11.6, 23.7)
27.7 (18.0, 34.6)
23.4 (16.3, 35.8)
26.6 (19.3, 33.4)
42.0 (35.6) 73.5)
H
9.7 (7.4, 12:7)
14.7 (8.2, 18.6)
12.9 (0.2, 16.8)
19.3 (14.6, 28.6)
33.6 (22.3, 99.1)
N
18.9
14.4 T2.33
54.3
45
76.5 (28.2,
E
33.3
24.1
14.4 (12.8,
22.6
31.1 (28.0,
M
M
28.3 (22.7, 39.3)
19.8 (14.4,
20.9 (18.8,
19.8 (14.9,
27.7 (26.7, 41.55
H
11.1 (9.3, 14.8)
9.9 (9.3, 16.0)
9.5 (0.7, 11.3)
11.1 (8.4, 13.7)
20.1 (15.9, 27.3)
N
1.5 (1.0, 3.4)
3.1 (2.1, 4.9)
8.4 (7.1, 8b.8)
8.7 (6.7. 11.8)
24.3 (17.8, 44.8)
E
9.1 (8.4, 14.2)
9.6 (6.8. 0.8)
8.6 (7.5, 13.0)
9.9 (T.1, 11.6)
18.8 (12.6, 22.2)
H
M
9.8 (7.9, 17.8)
8.0 (8.2, 12.3)
9.0 (6.5, 11.0)
5.9 (8.0, 9:33
16.2 (12.9) 18.8)
II
11.1 (7.8, 17.2)
7.0 (6.4, 8.0)
8.0 (6.0.
6.4 (5.0. 10.4)
12.4 (9.8, 18.0)
N
131.0
143.9)
49.9 (17.0, 73.9)
133.0 (58.6,
271.7 (130.1,
246.9 (218.7, 979.7)
E
99.5 (66.6. 110.4)
108.6 6 (106.a, 122.9)
111.9 (102.6, 150.0)
115.8 (97.8, 184.2)
152.0 (122.0,
m E
M
41.4 (32.4, 66.8)
42.7 (37.7,
60.1 (48
36.8 (26.0, 44.9)
68.2 (49.0, 98.8)
H
18.6 (12.7, 22.6)
19.7 (11.8, 25.1)
22.2 (16.9, 27.3)
14.9 (11.0, 40.8)
34.0 (28.9, 41.8)
125.9 (81.9, 145.0)
113.3 (35.3, 1888.)
100.7 (59.1, 138 2)
119.1 (11.1, 163.23
116.4 (65.0, 154.3)
E
37.6 (31.0, 68.2)
34.6 (30.4, 88.8)
54.3 (42.7, 74.1)
45.5 (28.8, 69.8)
45.5 (36.0, 97.9)
M
M
40.3 (23.6, 52.0)
30.2 (24.8, 39.0)
47.3 (19.7, 51.1)
25.8 (92.6, 31.3)
50.3 (27.5; 52.9)
H
21.2 (14.4, 29.0)
21.0 (16.6, 24.6)
19.7 (15.6, 27.1)
9.9 (8.0. 46.1)
26.5 (19.7, 88.0)
N
60.3 (43.5, 70-7)
6.1 (4.6,
15.2 (11.7,
12.3
36.2 (27.8,
E
26.3 (19.3,
15.5 (9.3, 18.3)
11.4 (5.2, 16.00
4.7 (3:6, 7.07
18.5 (16.0,
i HH
M
24.5 (20.4, 82.0)
10.1 (8.2. 19.4)
8.1 (9.1, 10.9)
7.5 (6.1, 8.9)
18.3 (14.4, 21.6)
H 33.7 (2L1, 87.0)
14.5 (8.3, 21.6)
13.3 (9.7, 18.9)
4.5 (8.7, 6.2)
16.2 (15.5, 19.9)
N
115.1
(01.4, 177.6)
119.6 (40.4, 154.5)
184.3 (85.2, 228 1)
171.7 (86.8, 223.03
229.7 (160.6, 251.5)
E
83.6 (87.9, 96.0)
114.4 (81.3, 184.0)
141.4
(106.7, 162.6)
93.6 (84.6, 117.3)
130.6 (102.0, 161.0)
m-e
M
35.0 (33-1, 44.5)
45.8 (37.8, 58.1)
71.6 (56.3, 82.9)
67.2 (35.9, 92.2)
81.0 (68.3, 109.3)
H
18.1 (14.2, 28.0)
30.7 (24.1, BO.1)
31.8 (27 45.2)
38.9 (35.4, 47.7)
55.4 (42.2, 80.8)
N
69.6 87:2)
11.4
41.4 104.4)
86.1 133.1)
84.4
116.0)
E
32.5 (22.3, 66.2)
27.6 (16.4,
40.8 (29
35.8
60.5 (33.0,
m- M
M
34.4 (21.3, 49.4)
24.1 (17.9, 30.1)
24.3 (19.3, 36.2)
39.6 (24.0, 45.8)
49.2 (37.0, 86.9)
H
13.5 (12.5, 18.6)
15.3 (9.1, 22.1)
12.4 (9.9, 25.1)
9.6 (5.2 13.7)
20.2 (13.8, 30.8)
N
17.3 (9.9, 29.2)
2.4 (1-7: 6.7)
21.1 (11.0, 39.0)
22.5 (10.4, 43.1)
39.2 (16.9, 60.8)
E
14.2 (B.5, 16.09
9.8 (7.5, 18.7)
17.6 (14.2, 44.9)
12.3 (9.2, 18.6)
29.4 (21.0, 41.2)
H
M
11.6 (10.4, 22.3)
7.0 (8.5, 11.0)
12.3 (9.7. 38.9)
12.3 (9.1, 14.8)
24.9
(17.7, 28.5)
H
12.4
7.5 (5-4,
12.1 (6.8,
10.0
27.3
IQM total
1407.0
1132.9
1464.1
1592.3
2298.0
31

---

Published in Transactions on Machine Learning Research (10/2024)
G Robustness analysis
To show the robustness of the proposed method over bascline, we perform two additional analysis, all based on training with cheetah-run medium-expert easy dataset. Firstly, We calculated the Wasserstein distance between the encoded features of from different level of distractions. The Wasserstein distance can be seen as a similarity metric between the two encoded features. Specifically, we have two scenarios. One is sampling batch of images from the V-D4RL chectah-run medium-cxpert dataset, and another is collected via online environment (cheetah-run) by using a random policy, to collect online distractions not. existing on the dataset. The values are averaged over the batch of observations (batch size 256).
Table 21:
Wasserstein distance calculated using different data sources. In the "distance between" column, all distance are compared between distraction observation and normal observation.
cheetah-run medium-expert (V-DARL)
data source
distance between DrQv2+BC
Ours
oflline normal dis-easy
0.5577
0.1413
offline normal dis-medium
0.5346
0.1709
offline normal dis-hard
0.2832
0.0887
online
normal dis-easy
0.3732
0.1093
online
normal- dis-medium
0.3311
0.0483
online
normal dis-hard
0.3312
0.1156
Secondly, plot the t-SNE (van der Maaten 2008 results of the trained policy distribution using different distraction observations. There are also two scenario of data source; from the dataset and one from online enviromment. As we can observe, in both scenarios, our proposed method brings the policy distribution closer for different distraction observations.
DrOx2+8C
Ours
to Manlter war Esheeleer bodiddaration
lurl distrdiio
octice
eagy distackk rad ditraciion
Figure t-SNE plot on the policy distribution. Blue dots are normal observation. Red dots are easy distraction observations. Green dots medium distraction. Purple dots hard distraction.
32

---

Published in Transactions on Machine Learning Research (10/2024)
H Additional Benchmark Comparison with V-D4RL
V-D4RL was collected by training an online SAC agent and using the saved checkpoints to collect visual observations while running the agent in proprioceplive states As we could not access the SAC checkpoints of the original V-D4RL authors, we followed their guidelines as closely as possible to reproduce data collection. The checkpoint used to collect the medium dataset is described as the first stable checkpoint, resulting in consistent 500 rewards. Simultancously, the expert agent is the stable checkpoint resulting in nearly the maximum possible reward, which translates to nearly 1000 rewards in DM Control tasks.
Figure presents the histogram of collected cheetah-run medium-exper data, plotted by rewards per episode versus frequencies (counts). The figure shows that the original data and broad data are comparable in most parts; the main difference is that the medium difficulty of the broad dataset is a bit lower compared to the original, and the expert difficulty is significantly concentrated in the final rewards (around 900). The difference is because of the checkpoint used to gather data.
cheetah-run www.mm/dddddeeeeeeeee distracting easy_ reward histogram
700
original
000
500
Cour
400
300
259
200
1867
217
100
10000002001 00000000010000000 11000000000000000001000000000000 10000000000000000000000000000000
10000000000000000
01
Poward per opicode
Figure 6: The original dataset is in orange, while the broad dataset is in blue.
Table 22: Full summary statistics of per-episode return the V-D4RL vs our proposed datasct.
Dataset
task cheetab-rum
difficulty
Timesteps
Mean
Std.
Min
P25
Median
P75
Max
100k
medexp
200k
707.5
186.0
253.8
527.7 7
710.6
894.1
905.7
medium
100k
523.3
32.9
253.8
511.1
527.6
539.2
578.3
V-DARL
expert
100k
891.6
843.0
888.3
894.1
899.2
905.7
random
100k
10.1
7.9
28.
33.8
38.6
41.2
66.3
walker-walk
medexp
990.6
medium
100k
436.8
57.4
65.6
419.2
444.2
469.0
538.6
expert
100k
970.6
940.6
964.2
909.7
979.2
990.6
random
100k
1.7
6.4
8.6
16.9
cheetah-run
medexp
200k
7U5.9
192.8
447.3
514.6
695.6
899.4 4
905. 1
medium
100k
447.3
582.1
expert
100k
897.
8.7
809.0
896.9
899.3
901.2
905.1
random
100k
40.0
28.3
33.8
38
44.4
66.1
walker-walk
medexp
200k
723.6
198.0
39.1
547.9
710.1
915.3
972.6
medium
100k
5355.7
81.2
39
522.3
547.8
572.7
661.1
expert
100k
911.5
32.1
759.0
896.4
915.3
935.2
972.6
random
100k
52.2
81.9
0.0
0.0
4.0
81.0
378.0
Ours
reacher-rasy
modexp
200k
medium
100k
502.5
430.6
0.0
10.2
512.5
952.2
1000.0
expert
100k
974.5
13.7
920.0
964.00
975.0
985.0
1000.0
random
100k
10.4
22.2
0.0
0.0
0.0
12.0
167.0
rearher-hard
medium
medexp
200k
100k
845.1
735.0
296.1
374.5
0.0
0.0
946.0
583.2
949.0
961.0
967.2
976.0
999.0
999.0
expert
100k
955.2
105.7
4.0
958.0
969.5
980.0
999.0
random
190.7
846.0
medexp
200k
931.1
91.0
431.0
937.0
960.5
986.0
1000.0
cup-catch
medium
100k
891.3
114.4
431.0
859.8
937.5
962.0
1000.0
expert
100k
970.9
18.4
937.0
953.0
964.0
990.0
1000.0
33

---

Published in Transactions on Machine Learning Research (10/2024)
Implementation Details
The implementation details and hyperparameters settings used for our experiments are presented below. Similar to DrQv2, we employed clipped double Q-learning (Fujimoto et al. 2018) to reduce the overestimation bias in the target value. We also provide schematic of how the overall network architecture is implemented as in Figure 8 as well as the network architecture used in Table 28 The Actor and critic networks follow DrQv2 exactly; thercfore, they are omitted. Additionally, we show the hyperparameters used in the experiments.
Table 23 lists the common hyperparameters for all methods. Table 24 lists the additional hyperparameten used in our proposed method. Table 25 lists the additional hyperparameterss used in the experiments for the baseline DrQv2+BC. Table 26 lists the additional hyperparameters used in the experiments for the baseline AWAC+BC. Table 27 lists the additional hyperparameters used in the experiments for the baseline IQL.
For PAD (Hansen et al, 2021b), ILA Yoneda al. 2022 and SVEA Hansen et al. (2021a), we use their default hyperparameters. We include some brief explanation on some of the baseline algorithms complement the hyperparameters listed in Appendix L.4
1.1 DrQv2+BC
DrQv2+BC (Lu et al. 2023) modified version of DrQv2 (arrass et al. 2022) by adding BC term into the policy network training. The term is regulated by hyperparameter Xbe. The usage Abe as:
π (saa (Q 0₂) Abr a+)
(Q(f(s;)))))c
1.2 AWAC
AWAC updates its policy by weighted maximum likelihood, where the targets are obtained by reweight- ing the state-action pairs observed in the dataset versus the predicted advantages from the critic, ( - ((((s;8)) a)): 9g)), where at ~ ((:)ff();))).. By adding a BC term into the policy training, the policy objective calculated as:
π argmaxxlB (ea
log a exp( CONn_LQQF (((1==1)))(4))---((0::)))))0)010 \a a,)²
Anwac
1.3 IQL
IQL uses the same wevintege regression policy training as AWAC while renaming the hyperparam- eter to a more straightforward scale B. In our experiments, we did not include an additional BC term for IQL. The policy objective for IQL is:
exp(3(
V(((s;()):0.)))
For the expectile regression loss, IQL minimises the following two losses, where TIQ is given as a hyperpa- rameter for expectile Gog.
min T.g.. Bee ( - Q
qn LYg.. BE (Q(() (s);(((();))))))) V(F(ss) where Ligs. TQL 1 0)]2²
34

---

Published in Transactions on Machine Learning Research (10/2024)
1.4 Hyperparameters
Table 23: Common hyperparameters.
Parameters
Value
Parameters
Value
optimiser (all networks)
Adam
batch size
256
encoder learning rate
3e 04
critic learning rate
3e - 04
actor learning rate
3e -
04
feature dim
50
activation function
ReLU
hidden dim
1024
frame stacking (all tasks)
3
action repeat (all tasks)
nslep (all lasks)
3
weight decay
critic target tau (7)
0.01
Table 24: Additional hyperparameters for our proposed algorithm. Parameters
Value
Parameters
Value
behaviour cloning (BC) weight Abe dropblock drop rate
2.5
gradient reversal μ
Table 29
discriminator learning rate le 05
Table 25: Additional hyperparameters for baseline DrQv2+BC.
Parameters
Value
behaviour cloning (BC) weight Abe
2.5
Table 26: Additional hyperparameters for baseline AWAC+BC.
Parameters Value
AWAC lambda Anwac 0.3
Table 27: Additional hyperparameters for baseline IQL.
Parameters
Value
Parameters
Value
IQL scale IQL expectile nor.
0.7
1.5
Revelant Network Structures
Table 28: Network architectures
Network name
Operations
Kernel
Strides
Ch 1/O
Conv + ReLU
3x3
2
9/32
DropBlock
Conv + ReLU
3x3
1
32/32
Encoder
DropBlock
Conv + ReLU
3x3
32/32
DropBlock
Conv ReLU
3x3
1
32/32
Linear ReLU
39200/ 1024
Discriminator
Linear + ReLU
1024/50
Linear ReLU
50/1
35

---

Published in Transactions on Machine Learning Research (10/2024)
1.6 DropBlock hyperparameters
Table 29:
DropBlock rate p for all layers in the encoder for all tasks of the results in Table Table [0]
Table
11
Table 12
Table 13 and Table 9 Abbreviations for dataset type: medium: m, expert: e, medium-
expert: m-c.
Abbreviations for visual distractions difficulty: normal: N, easy: E, medium: M, hard: H.
Task
eval
cheetah-run cheetah-run (add.)
ball-catch
reacher-easy easy reacher-hare walker-wale
E
N
0.3
0.5
0.5
0.5
0.5
0.2
E
E
0.2
0.3
0.5
0.5
0.5
0.1
E
M
0.2
0.3
0.5
0.5
0.5
0.2
e
0.5
0.3
0.5
0.5
0.5
0.2
0.3
0.1
0.3
0.3
0.5
0.1
M
0.2
0.3
0.5
0.5
0.2
M
M
0.5
0.1
0.5
0.5
0.5
0.3
M
H
0.2
0.1
0.5
0.5
0.5
0.1
H
0.3
0.5
0.5
0.3
0.2
0.1
H
E
0.1
0.5
0.5
0.5
0.2
0.1
M
0.3
0.5
0.5
0.3
0.2
0.1
H
0.2
0.2
0.5
0.3
0.2
0.1
0.3
0.2
0.2
0.3
0.1
0.3
0.3
0.1
0.2
0.1
0.5
0.5
0.2
0.5
0.2
E
0.5
0.5
0.3
0.2
0.5
0.3
M
0.1
0.1
0.2
0.3
0.1
0.5
M
0.3
0.3
0.1
0.3
0.3
0.5
M
M
0.2
0.5
0.2
0.3
0.2
0.5
m
M
0.2
0.5
0.1
0.1
0.2
0.2
0.5
0.2
H
0.1
0.5
0.1
0.3
0.2
0.1
H
0.1
0.5
0.1
0.5
0.1
0.2
H
H
0.1
0.2
0.2
0.2
0.2
0.1
m-e E
0.3
0.5
0.3
0.1
0.3
0.5
mee E
E
0.1
0.5
0.2
0.3
0.5
0.2
mee E
M
0.3
0.1
0.3
0.3
0.5
0.3
m-c
0.3
0.2
0.2
0.2
0.1
0.5
me
0.3
0.5
0.1
0.3
0.1
me
0.3
0.5
0.2
0.5
0.5
0.1
M
M
0.5
0.3
0.2
0.1
0.5
0.2
m-e M
H
0.2
0.5
0.3
0.1
0.5
0.2
m-e H
N
0.3
0.5
0.5
0.2
0.1
0.1
me
E
0.2
0.3
0.1
0.2
0.5
0.2
m-c
0.2
0.3
0.5
0.3
0.1
0.1
m-
0.3
0.1
0.3
0.3
36


Reviews:
[{"review_id": "GCwIQI1PBq", "reviewer": "Reviewer_aSdA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The paper is mostly clear (see questions below), and the ideas presented are well motivated.\n- The new dataset extensively expands the previous visual D4RL dataset in several domains with several different policies.\n- The method is reasonable, with some reasonable flaws (see below).\n\n# Weaknesses\n\nI am fairly happy with the paper as presented, but have a few recommendations and questions.\n\n- **W-1**. I don't think you ever disclose what the discriminator is discriminating in the dataset.  I believe it is whether the data has visual distractions or not, but it is unclear given the text in section 4.1. Just clarifying what the classes are would be sufficient.\n- **W-2**. A major weakness of this method is the need for non-distracting data. It is very likely this will be difficult to get in any real-world domain. While I don't expect the authors to provide a solution in this paper, a section going over the weaknesses/limitations of the method might be warranted.\n- **W-3**. You should disclose how the hyperparameters were tuned. While they are listed in detail (which I'm grateful for), I don't think the paper discloses how they were tuned. I think this should be in the main paper, and the main text should also point to the appendix with the hyperparameters to make sure readers know it exists or to go looking for it. If hyperparameters are re-used from old work, I believe this is another major weakness in the results shown. I highly recommend [this paper](https://arxiv.org/abs/2304.01315) for guiding your future empirical study design. This is a large weakness in the field in general.\n- **W-4**. I think a missed opportunity in the ablation study was to ask whether the reverse gradient idea is additive in the RL tasks tested here. While I think it still would be (the reasoning is sound) and (Ganin and Lempitsky, 2015) paper does this in the supervised setting, I think it is low hanging fruit to see how much of a difference it makes here as well. I don't think it is necessary to include on all the domains and maybe just a single domain to show it is additive.\n- **W-5**. I would have preferred to see [InAC](https://openreview.net/forum?id=u-RuvyDYqCM) tested as it has shown to be a strong baseline in several domains. I understand it might be difficult as the method hasn't been tested thoroughly in image domains so hyperparameters likely need to be tuned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper follows up from (Lu et al, 2023) to study visual distractions in offline reinforcement learning using v-D4RL. Their contributions include\n- Substantially extending the dataset from (Lu et al, 2023) \n- Proposing a method to learn better policies in the face of visual distractions. Their method employs an adversarial approach which learns a discriminator on whether the environment contains distractions or not. The visual encoder is then also used to learn RL policies from offline data. \n- Several experiments comparing with DrQv2+BC, IQL, and AWAC+BC. The paper also includes a parameter study of their new hyperparameter.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend weaknesses 1-3 be addressed, while weakness 4 & 5 are something to think about if the authors have compute available.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5e4DQD9r5I", "reviewer": "Reviewer_2QTB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nNovel Adversarial Training Framework: The paper introduces an innovative use of adversarial training to tackle visual distractions in offline reinforcement learning (RL). This approach focuses on learning domain-invariant features, which is a great advancement in making RL agents more robust to environmental variations.\n\nComprehensive Dataset Extension: The extension of the V-D4RL dataset to include more locomotion tasks with diverse visual distractions is a valuable contribution. This enhanced dataset provides a more robust benchmarking tool for evaluating visual-based RL methods.\n\nEmpirical Validation: The empirical results demonstrate that the proposed method consistently outperforms state-of-the-art baselines on tasks involving visual distractions. This strong performance across different datasets highlights the effectiveness of the approach.\n\nIntegration of DropBlock: The use of DropBlock in the visual encoder to prevent overfitting to specific features is a practical enhancement. This technique forces the encoder to rely on a broader set of features, improving the general robustness of the model.\n\nWeaknesses\n\nEvaluation Scope: While the paper extends the V-D4RL dataset, the evaluation is primarily focused on locomotion tasks. Evaluating the method on a broader range of RL tasks would strengthen the claims regarding its general robustness.\n\nLack of Comparison with Third-Person Imitation Learning: The paper does not compare the proposed method with third-person imitation learning approaches, nor does it mention relevant papers in this area. As these methods share very similar spirit, therefore, including such comparisons would provide a more comprehensive evaluation of the proposed method's effectiveness.\n\nLack of real world data: though this approach uses mostly synthetic environments, it remains unclear whether this approach would apply for real world problems where the visual distraction may appear in more complicated format.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"Offline Deep Reinforcement Learning for Visual Distractions via Domain Adversarial Training\" presents a novel approach to enhance the robustness of reinforcement learning (RL) agents to visual distractions in offline settings. By introducing an adversarial training framework, the authors aim to train visual encoders to focus on domain-invariant features, thereby reducing the impact of visual distractions. The method involves using a domain discriminator to classify latent features into normal and distracted domains, and training the visual encoder to minimize the discrepancy between these features. Additionally, the paper extends the V-D4RL dataset to include more tasks with visual distractions and demonstrates the effectiveness of the proposed approach through extensive empirical evaluations, showing improvements over state-of-the-art baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Include a comparison of the proposed method with third-person imitation learning approaches.\n\nEvaluate the proposed method on a wider range of RL tasks beyond locomotion, such as manipulation or navigation tasks. This would strengthen the claims of the method's general robustness and applicability across different RL scenarios.\n\nProvide a more detailed analysis of how the domain discriminator impacts feature extraction and the resulting policy robustness. Discuss the instability of this adversarial training, does techniques from generative AI domain such as those training GAN and stabilize GAN training approaches help for this work?", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uA47ItoKfM", "reviewer": "Reviewer_B56R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is well-written and clear. The motivation for the proposed method is well explained and the method itself seems to make sense.\n\n2. The reported experimental results are in general strong. It does seem that the full method is outperforming the baselines. There are some reasonable ablations presented to rationalize design decisions. \n\n### Weaknesses\n\n1. The original V-D4RL paper finds more robust performance to distractors by using a model-based method (their version of offline Dreamer V2). However, this paper does not compare to this method, or any model based method. It would make the paper more comprehensive to compare to this sort of baseline.\n\n2. To my understanding, the baseline methods are trained only on the distraction dataset while the visual encoder for the proposed method is trained using the distraction and the normal dataset. This seems to be an asymmetry that is not very well controlled for in the experimental methodology. Perhaps I misunderstand something, but more clarity is needed here and perhaps another ablation where the baselines all have access to the same data is warranted.\n\n3. The use of drop-block somewhat confounds the results of the adversarial loss. There is an ablation in table 6, but there it does seem that if we remove drop-block, then the proposed method may not actually be much different from the baselines in table 2. A more direct comparison here would be good. Or even better, since drop-block is just an architectural change, it could easily be applied to the standard baselines (e.g. DrQv2+BC) to see if just adding drop-block alone makes these algorithms competitive with the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an adversarial approach for dealing with visual distractors in offline RL from images (and an architectural change to add drop-block to the encoder). Using an expanded version of the V-D4RL benchmark, the method is compared favorably with prior work of model-free offline RL from images (e.g. using image augmentations as in DrQv2+BC) and some methods for domain adaptation from online RL.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a model based baseline as in the V-D4RL paper.\n\n2. Clarify the data used for each of the baselines. \n\n3. Add more ablations of dropblock on top of the baselines.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE recommends acceptance with a minor revision. Authors have already posted a revision with red text indicating the changes from responding to reviewer concerns. Please integrate these changes for the camera ready and make the data requirements a more visible assumption of the proposed method -- e.g., by extending the 2nd paragraph on page 2 to be clearer about the "two domains" described there.
Decision: Accept (poster)