Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation
Authors: Zheyu Zhuang
RUIYU WANG
Nils Ingelhag
Ville Kyrki
Danica Kragic
Venue: CoRL 2024 (2024)
Keywords: Behaviour Cloning
Visuomotor Policy
Data Augmentation
TL;DR: Enhance policies' robustness against visual domain shifts (distractors, background variations) by leveraging saliency maps generated from policy to perform data augmentation.

Abstract: In vision-based behaviour cloning (BC), traditional image-level augmentation methods such as pixel shifting enhance in-domain performance but often struggle with visual domain shifts, including distractors, occlusion, and changes in lighting and backgrounds. Conversely, superimposition-based augmentation, proven effective in computer vision, improves model generalisability by blending training images and out-of-domain images. Despite its potential, the applicability of these methods to vision-based BC remains unclear due to the unique challenges posed by BC demonstrations; specifically, preserving task-critical scene semantics, spatial-temporal relationships, and agent-target interactions is crucial. To address this, we introduce RoboSaGA, a context-aware approach that dynamically adjusts augmentation intensity per pixel based on input saliency derived from the policy. This method ensures aggressive augmentation within task-trivial areas without compromising task-critical information. Furthermore, RoboSaGA seamlessly integrates into existing network architectures without the need for structural changes or additional learning objectives. Our empirical evaluations across both simulated and real-world settings demonstrate that RoboSaGA not only maintains in-domain performance but significantly improves resilience to distractors and background variations.

Enhancing Visual Domain Robustness in Behaviour Cloning via Saliency-Guided Augmentation
Zheyu Zhuang', Ruiyu Wang', Nils Ingelhag', Ville Kyrki", Danica Kragic KTH Royal Institute of Technology Aalto University? zheyuzh@kth.se
Abstract: In vision-based behaviour cloning (BC), conventional image augmen- tations like Random Crop and Colour Jitter often fall short when addressing substantial visual domain shifts, such as variations in shadow, distractors and backgrounds. Superinpossition-assee augmentations, which blend in-domain and out-of-domain images, have shown promise for improving model generalisation in the computer vision community, but their suitability for BC remains uncer- tain due the need to preserve task-critical semantics, spatial-temporal relation- ships, and agent-target interactions. To address this, we introduce RoboSaGA- Saliency-Guided Augmentation method within the superimposition family, tai- lored for dision-based BC. RoboSaGA dynamically adjusts augmentation inten- sity per pixel based on policy-driven saliency, enabling aggressive augmentation task-trivial areas while preserving task-critical information. Moreover, it inte- grates seamlessly into existing architectures without requiring structural changes additional learning objectives. Empirical evaluations in both simulated and real-worlds settings show that RoboSaGA maintains in-domain performance while significantly enhancing robustness to visual domain shifts, including distractors and background variations, as well as handling lighting and shadow variations. Code available at: https://git thub. cwm conm/2hhyuu-Zhuang/Rooooo Zhuang/RoooSaa....
Keywords: Behaviour Cloning, Visual Generalisation, Data Augmentation
Introduction
Vision-based behaviour cloning (BC) has made significant strides in transferring behaviour from expert demonstrations to robot skills [1, 2]. These demonstrations involve intricate spatial-temporal interactions between the agent and its environment and often require integration among multiple sensory modalities. However, given the high costs associated with collecting demonstration data, especially in real-world settings [3, 4], datasets often prioritise task-related variability over visual diversity [5, 6]. This prioritisation presents considcrable challenges for generalisation under visual domain shifts, such as variations in lighting, shadows, distractors, or backgrounds.
While image-level augmentation techniques such Random Crop and Colour Jitter preserve the in- tricacy of BC demonstrations and enhance in in-domain performance [7, 8), they fall short in address- ing challenges across the broader visual domain (Sec. 4.2). Superimposstiin--asee augmentation techniques, such as selectively removing parts of an image [9, 10] or overlaying images [11], have proven effective in the computer vision community. However, their application in vision-based be- haviour cloning remains limited due to the need to preserve task-critical semanties, spatial-temporal relationships, and agent-target interactions. To this end, we introduce RoboSaGA, a Saliency- Guided Augmentation designed for vision-based behaviour cloning. It utilises the input saliency back-prod peokppropggtted from visual features per-pixel blending factors, allowing the preservation of task-critical regions while superimposing the in-domain images onto off-oin (OOD) images. Crucially, RoboSaGA does not require additional learnable modules or specific learning objectives, making it readily applicable to multi-view inputs and compatible with various BC policies.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

fo(x²)
Training
Testing
gradient
fo
Saliency Extractor
2O
Backgrounds
(a) Building Block of RoboSaGA
(b) Examples of Visual Domain Shifts
Figure I: RoboSaGA's core components and the broadened visual domains. (a) Saliency map 8, derived from visual feature output (x). guides the overlaying of in-domain and the OOD image zo to create the augmented image x" for training. (b) Examples of training and tested VDS scenarios.
To highlight RoboSaGA's strengths in handling visual domain shifts (VDS), we compare its per- formance against traditional augmentation methods and existing superimpositiin-basse techniques from the computer vision field within the BC context. Specifically, we evaluate their performance in addressing variations in lighting, shadows, distractors, and backgrounds (with the texture of the op- erational surface considered part of the background). Performance is measured by the performance gap (Eq. (3)), defined as the difference in mean success rate between the in-domain baseline and the model trained with the augmentation method under visual domain shifts, Experiments span both simulation and real-world settings, focusing on key policy variants: BC-MLP (multi-layer percep- tron), BC-RNN (recurrent neural network) [7], and diffusion policy [2]. We summarise insights:
Simulated experiments show that both Random Crop and Colour Jitter struggle with dis- tractors and background changes. While Colour Jitter improves performance under light intensity changes, remains less effective under shadows (gap of 0.33). contrast, all superimposition--aan based methods keep gaps below 0.1 under lighting and shadow variations.
In simulated experiments, the erase-based method effectively handles distractors by replac- ing task-trivial regions with OOD images but struggles with background variations.
Random Overlay, applying constant blending factor for OOD image overlay, reduces the performance gap from Random Crop's s 0.60 to 0.24 in simulations and from 0.71 to 0.18 in real-world tests, demonstrating its effectiveness as an augmentation method against VDS.
RoboSaGA dynamically adjusts augmentation intensity based on saliency maps and further improves performance by reducing the gap to 0.14 in simulations and 0.05 in real-world scenarios, consistently outperforming Random Overlay.
2 Related Work
Vision-based Behaviour Cloning. A typical vision- based BC policy मH ((z)))=(-7) parame- terised by &, derives actions from temporal sequence of observation embeddings over window of length T' [7]. At time step t, the observation embedding z, is defined as:
z1 : 9v ([fe, ()= ho(.))
where xt and p, represent the visual input from the 7th visual modality or camera view, and the proprioceptive state. The visual encoders fe.. parameterised by 0,, process the visual inputs, while the proprioceptive encoder hes parameterised by Φ. handles the proprioceptive data. The func- tion Pw, parameterised by vb, integrates these inputs into the latent variable zt. BC policies TT(-) can be categorised into explicit and implicit types. Explicit policies form the simplest version of BC, directly regressing actions from observation embeddings [12, 13), but they struggle to handle the multi-modal nature of human demonstrations. To address this, Florence et al. [1] reformulate imitation learning as a conditional energy-based modelling problem. More recently, Chi al. [2] introduce diffusion- fiffus policy that manages multi-modality by leveraging stochasticity in both the initialisation and sampling processes of diffusion models [14].

---

Data Augmentation in Computer Vision. Data augmentation techniques, traditionally involving randomisations in colour space and image geometry, such as colour jittering, cropping, and rota- tion, have been expanded with modern superimposition augmentation techniques. Formally, the augmented image x" is a weighted addition of the in-domain image æ and out-of-domain (OOD) images zo. both of the same size h wt
(I) where M Rixw is the blending matrix and @ denotes element-wise multiplication. MM can be uniformly filled with constant value between zero and one [11], binary with randomly located and sized patches [9, 10], or binary patches guided by input saliency maps [15, 16]. To can be all-zero matrices [9], random noise [10], or inter-class samples [11, 10]. KeepAugment [15], conceptually closest to RoboSaGA, uses class activation maps to apply a fixed-size rectangular binary mask that covers the most salient areas, preventing the corruption of task-critical regions during augmentation.
x = M o + M) + xo.
Data Augmentations for Robotic Manipulation. With collected datasets, mild augmentations like Random Crop essential for improving in-domain generalisability of models [2, 7], though they do not expand the semantic range to cover broader visual domains. Additionally, incorporating language descriptions, Abolghasemi et al. [17] learns language--onnditonee masks to exclude task- irrelevant areas. Yu et al. [18] utilise off-the-shelf foundational modules to alter scene attributes, thereby facilitating skill acquisition and enhancing robustness. Superimposition augmentation tech- niques without task prior have received little attention BC settings, in contrast to the image-based reinforcement learning (RL) community, where such techniques are more commonly used to en- hance policy generalisability 20, 21].
3 Saliency-Guided Augmentation for Robotic Behaviour Cloning
RoboSaGA distinguishes itself from other saliency-based superimposition augmentation meth- ods [16, 15] by employing adaptive per-pixel blending factors and extracting saliency directly at the visual encoder level. Unlike KeepAugment [15]. which relies on classification logits and fixed- size rectangular crops, RoboSaGA is specifically designed for vision-based manipulation tasks. Its encoder-level saliency extraction ensures modularity and scalability across multiple camera views and policy types, including BC-MLP, BC-RNN [7], and Diffusion Policy [2]. The per- per-pixel blending approach more effectively handles spatially separated, variable-sized objects and provides greater robustness to visual domain shifts, such as background and lighting changes (Sec. 4.1).
Saliency Extraction. RoboSaGA leverages FullGrad [22], which back-propagates the full gradi- ent from the output feature vector back to the full-resolution input, rather than being restricted to gradients from classification logits specifie layers, in methods like Grad-CAM [23] or Smooth- Grad [24]. FullGrad also incorporates learned biases, in addition to network weights, during saliency computation, resulting in sharper and more accurate saliency maps.
Saliency Clipping. RoboSaGA implements a clipping mechanism that caps the normalised saliency scores to a pre-defined threshold À E (0, 1). By applying a slightly clipped saliency score, Ro- boSaGA enhances the augmentation of task-critical regions through subtle overlay, crucial for improving robustness against background changes. See Section 4.1 for details. Formally, given an input image x and its corresponding encoder fe, (-), we define the normalised salieney derived from Fullgrad as g" : Fullgrad fo, (). The saliency map s" is then formulated as:
Sh min(g" A)
Data Augmentation. Within training batch containing m trajectories (considering single image as a trajectory of length one), RoboSaGA selectively augments a out of m trajectories, where a serves as a hyper-parameter This selective augmentation strategy is designed to mitigate potential negative impacts on convergence that can arise from overly aggressive augmentation across the entire batch. Each sample within trajectory augmented independently, as no performance gap has been observed when using trajectory- wise augmentation, where all samples in a trajectory share the same augmentation parameters as described in [25].
3

---

Algorithm RoboSaGA Saliency-Guided Augmentation Process
1: a, B # number of trajectories for augmentation and saliency update.
2: A (0,1) # saliency clipping factor
3: Define class RoboSaGA with methods:
4: set(saliency maps. global id) update the buffer with new saliency maps
5: get(global id) # retrieve saliency maps from the buffer
6: set-globall slbbal.idbbatch indices) convert batch indices to global IDs
7: get_ood_images() # #gtt random nut-of-domain images
8: Initialize RoboSaGA instance with all-one saliency maps.
9: for each batch M with trajectories do
10:
for each visual encoder e in the policy network do Define M, as the images processed by encoder
11:
12:
Update steps
13:
Sort batch indices I by saliency buffer update frequency and select bottom 3 as Lg Compute new saliency maps: new maps min (Fullgrad (v(M,[Is)) A)
14:
15:
Update the buffer S.set (new maps, S.get global id(1;))
16:
Augmentation steps
17:
Sample batch indices I, uniformly from f1, ................................ n), targeting trajectories
18:
Retrieve saliency maps for sampled indices: S, = S. S.get(S Des.ge-gglbae id (I,))
19:
Augment images: M, [I.] + S. @ M, [T]] (1 S.) @ S.get_ immages end for
20:
21: Optimise the policy with augmented batch M
22:
end for
Global Saliency Buffer. While Fullgrad produces high-quality saliency maps [22], it incurs higher computational cost due to the inclusion of both network weights and biases. RoboSaGA mitigates this by employing a global buffer that stores and monitors past saliency maps for each image, significantly reducing computational demands. The buffer is initially filled with ones and updates the saliency for the least recently updated 33 trajectories in each training batch, where /3 is a hyper-parameter. During the augmentation phase, a saliency maps are retrieved from the buffer. To conserve memory, these maps are stored as 8-bit, single-channel images a reduced resolution. For example, storing maps for 10,000 samples at resolution of 84 x 84 consumes approximately 67 MB of graphic memory. Through experiments, we found this delayed saliency update does not cause observable performance degradation. Analysis of the saliency buffer's performance-computatii trade-off can be found in Appendix A. The comprehensive algorithm is detailed in Algorithm 1.
4 Experiments
Manipulation Tasks and Out-of-Domain Settings. As illustrated in Fig.2, in simulations, we use expert human demonstrations from the Robomimic environment [7] to evaluate four tasks of varying complexity and visual diversity: Lift (pick), Square (pick and insert), Can (pick and place), and Transport (dual-arm pick, handover, place). Real-world experiments focus on Toy (pick and place). Most tasks use a second-person and an eye-in-hand camera, except Transport with four cameras, all at 84 x 84 resolution. simulations, lighting is varied by adjusting intensity and colour, shadows are toggled on or off, and distractors are randomised by changing category, quantity, and location. Surface textures and backgrounds are shuffled using materials such as textiles, wood, and ceramics. In real-world settings, distractors randomly selected from set of items, and surfaces are altered using textile combinations to change backgrounds. Scenes randomised per trajectory.
Tested Policies and Evaluation Protocol. We evaluated three seminal policies: BC-MLP (Multi- Layer Perceptron), BC-RNN (Recurrent Neural Network), and the sfate-of-the- Diffusion Policy [2]. All policies incorporate robot proprioceptive states. BC-RNN and Diffusion Policy also use past observations, while BC-MLP relies on the current state only. Following [7]. BC-MLP and BC-RNN employ Gaussian Mixture Model to capture the multi-modal distribution in human demonstrations. Policies are trained from scratch by default. In simulation, the average task success rate is based on

---

00D Image
Random Overlay
Goided Frase
Robo- SaGA
FallGrad
Simulation
Real
Salieney
Barkground Diiractos In-Doomain
Square
Transguett
Ed0
Square
Can
Transport
Zay
The trinsport task unitses four contera view: only two are dieplayed here
Figure 2: Experiment environment setups.
Figure 3: Augmentation with BC-MLP.
the top three checkpoints from 600 training epochs, saved every 50 epochs, and tested over 50 simulation rollouts per checkpoint. In the real world, the final checkpoint after 500 epochs is used, with task success rates averaged over 20 trials per data point.
Implementation Details. For RoboSaGA, the clipping threshold (Eq. 2) is set to 0.8 across all ex- periments. The saliency buffer updates 10% of the training batch, and RoboSaGA augments half of the training batch. For OOD images, we 5000 images from subset of MSCOCO [26] combined with 1000 synthetic images including plain colours, grids and gradients. The BC-MLP and BC-RNN implementations are identical to the default configurations found in [7]. The Diffusion Policy aligns with the hybrid-CNN, in [2]. See Appendix B for experimental setup and implementation details.
Random Crop as the Default Augmentation. Random Crop significantly improves both in-domain and out-of-domain generalisation in robotic manipulation [7, 8]. making the baseline augmentation in all experiments. Methods prefixed with "+" denote combination with Random Crop. All networks use crop size of 90% of the input, resized back to the original resolution.
Colour Jitter. Colour Jitter baselines are obtained by fine-tuning the Random Crop baselines for 50 additional epochs, evaluating checkpoints every 10 epochs, and selecting the top three for compar- ison. Colour Jitter is implemented with PyTorch's built-in ColorJitter function with brightness, contrast, saturation, and hue to 0.2, as [8].
Evaluation Metric. We assess performance degradation using the performance gap AP+Aug +Augo de- fined as the difference between the in-domain mean success rate of the Random Crop baseline and the mean success rate under visual domain shift scenarios with the applied augmentation method. Specifically, the performance gap is expressed as:
AP+Aug := pin Baseline
VDS -Aug'
(3)
This formulation allows for a direct comparison of two augmentation methods by subtraction, effec- tively cancelling out the baseline performance. A smaller gap indicates greater robustness against the tested visual domain shifts (VDS). Metrics such standard error are computed using pooled observations across checkpoints, tasks, and policies.
success rate, performance gap, and
4.1 Dissecting RoboSaGA: The Interplay of Overlay and Erase in Behaviour Cloning
RoboSaGA can conceptualised as a hybrid method that synergistically integrates two distinct augmentation techniques: overlay and erase. Before expanding the scope our experiments, we undertake ablation study dissect the intertwined roles of these techniques within RoboSaGA. The erasing and overlaying variants of RoboSaGA are:
Guided Erase: RoboSaGA binarises saliency maps at pre-defined threshold for superim- position. This method, designed as variation of KeepAugment [15], uses variable-sized regions instead of fixed-sized patches and derives saliency from visual features rather than classification logits, enabling more adaptive augmentation.
Random Overlay: The context-aware per-pixel blending matrix from RoboSaGA is placed by global constant. This method also appears baseline in [20].
5

---

Average over All Tested Policies
Cop
+Jittrr
+SODA
+Overlay
+SoGA
BC-MLP
Lighting Lighting & Shadow
0.39
0.10
0.05
0.02
0.01
0.56
0.33
0.08
0.09
0.05
Random Crop
Guided Erare
Rondom "verlay
Robo- SoGA
meam
0.48
0.21
0.07
0.05
0.03
Background
0.84
0.55
0.21
0.13
Background
0.72
0.61
0.35
0.26
0.19
Distractor 0.52
0.08
0.31
0.06
Distractor
0.48
0.46
0.20
0.22
0.10
monm
0.68
0.31
0.26
0.10
0.60
0.55
0.28
0.24
0.14
Table I: Gap (1) of BC-MLP under varia- tions in background and distractors.
Table 2: Gap (L) across All Policies (BC-MLP, BC- RNN, Diffusion Policy) under visual domain shifts.
BC-MLP
BC-RNN
Diffusion Policy
Crup
+Jitter
+SODA
+Over
+SaGA
Crop
+itter
+SODA
+Over
+SaGA
Crop
+Jitter
+50DA
+Over
+SoGA
BG.
0.75
0.38
0.27
0.21
0.74
0.40
0.21
0.60
0.52
0.29
0.24
0.16
Dist.
0.46
0.46
0.22
0.31
0.09
0.53
0.48
0.22
0.19
0.10
0.45
0.44
0.16
0.15
0.11
0.60
0.56
0.30
0.29
0.15
0.66
0.61
0.31
0.23
0.15
0.52
0.48
0.23
0.19
0.13
Table 3: Performance Gap (L) of BC-MLP, BC-RNN, and Diffusion Policy with Random Crop, +Random Jitter, +Random Overlay, +SODA, and +RoboSaGA under visual domain shifts.
Both the binarisation threshold for Guided Erase and the blending factor for Random Overlay are set to 0.5. These implementations differ from RoboSaGA only in their approach to superimposition, while the overarching augmentation strategy remains consistent. Specifically, we randomly sample and augment 50% of the trajectories from each training batch for all experiments. Fig. 3 illustrates the augmented images from the above methods. We evaluated the efficacy of RoboSaGA and its two variants across three simulation tasks (Lift, Can, Square) with BC-MLP.
As shown in Tab.1, Guided Erase fully replaces task-trivial regions, which enhances resilience to distractors, but it leaves task-critical areas untouched, limiting its performance against background variations. On the other hand, Random Overlay uses a universal blending factor, maintaining con- sistent performance across visual domain shifts but being less effective against distractors due to its milder modification of task-trivial areas. RoboSaGA combines the strengths of both methods, reducing Random Overlay's performance gap under distractors from 0.31 0.06, and improving Guided-Erasee performance gap under background variations from 0.55 to 0.13.
4.2 The Efficacy of RoboSaGA and Main Observations
In this section, we evaluate each selected augmentation method under visual domain shift scenarios across various behaviour cloning policies in both simulated and real-world settings. Full tables with task success rates and the standard error of the mean are provided in Appendix C. This evaluation aims to answer the following questions:
Q1: How well do traditional augmentation methods handle visual domain shifts (VDS)?
We evaluated two conventional augmentation methods, Random Crop and Random Crop Colour Jitter, under four VDS scenarios. Tab. 2 presents the performance gap averaged across four tasks and three policies. The results show that lighting significantly degrades baseline performance. While Colour Jitter effectively handles plain lighting changes (intensity and colour), reducing the average gap to 0.1, its improvement is limited under shadows, with gap of 0.33, compared to less than 0.1 for overlay methods. For distractors and background variation, colour jitter reduces the gap from 0.6 0.55, offering only an 8% relative improvement. Given the strong performance of tested superimposition methods in addressing lighting changes, the remainder of this section and the real- world experiments focus on VDS scenarios involving distractor and background variations.
Q2: Is representation learning more effective than direct training with augmented images?
In vision-based reinforcement learning, aggressive data augmentation is frequently implemented with representation learning techniques for improved policy stability [19, 20]. Within this frame-

---

BC-MLP
BC-RNN
Diffusion Policy
All Policies
Cop
+Overlay
+SaGA
Crop
+Overley
+SaGA
Crop
+Overley
+SaGA
Cmp
+Overlay
+SaGA
Background 0.70
0.20
0.05
0.85
0.25
0.00
1.00
0.35
0.25
0.85
0.27
0.07
Distractor
0.65
0.05
0.100
0.70
0.20
0.15
0.35
0.05
0.05
0.57
0.10
0.08
mecm 0.68
0.12
0.08
0.78
0.22
0.07
0.68
0.20
0.15
|0.71
0.18
0.05
Table 4: weal-world Performance Gap (1) across Each Policy (and Average) for Random Crop, Random Overlay, and RoboSaGA on the Toy task.
Front Camera
Eye-in-Haue Hand
Front Camera
Eye-in-Hand Hand
External View
(a) Severe ail-liggtiig and shadow
(b) Full occlusion from front view
(c) Clutter
Figure 4: Examples of RoboSaGA against Real-World Visual Domain Shifts, including lighting changes, occlusion, object clutter, and background variations.
work, we aim to determine the necessity and effectiveness of representation learning in behaviour cloning (BC), where action labels are readily available. To this end, we adopted SODA [20] as our baseline, which utilises student-teacher architecture with a consistency loss to encourage Euclidean similarity between features derived from images augmented by Random Overlay and their original counterparts. We assessed the performance RoboSaGA and Random Overlay in comparison to SODA using the three policies across four simulation tasks. Experimental results in Tab. 2 show that direct training with augmented samples delivers comparable yet marginally improved performance, reducing the performance gap by an average of 0.04 under distractor and background variations, and by 0.02 under lighting and shadows.
Q3: Does RoboSaGA's effectiveness against VDS extend policies beyond BC-MLP?
Tab. and Tab. 3 tabulate the mean performance gap under visual domain shift (VDS). The results suggest that while performance varies across different policies, all three tested augmentation meth- ods (Random Overlay, RoboSaGA, and SODA) consistently achieve translatable improvements. No- tably, the Diffusion Policy exhibits the least performance degradation. These findings are confirmed by real-world experiments, where both Random Overlay and RoboSaGA significantly enhance per- formance against VDS compared their Random Crop counterparts (Tab. 4).
Q4: Does RoboSaGA maintain its performance lead against the computation-free Random Overlay?
Random Overlay an efficient and effective augmentation method against VDS, reducing the per- formance gap Random Crop + Jitter from 0.55 to 0.24 for distractor and background variations, and from 0.21 to 0.06 for lighting and shadow variations. RoboSaGA further improves performance, lowering Random Overlay's ' gaps to 0.14 and 0.03, with relative improvements of 41.6% and 50.0%, respectively. In real-world experiments (Tab. 4), Random Overlay reduces the gap (averaged over three policies) from 0.71 to 0.18 for distractor and background variations. RoboSaGA further duces it to 0.05, yielding a 72% improvement over Random Overlay.
Q5: How effective RoboSaGA is in the real world?
Instead of exploring wide range of task variants, focused on relatively simple 3 degrees-of- freedom pick-and-place task to rigorously evaluate the real-world performance of Random Overlay and RoboSaGA. These methods were tested using BC-MLP, BC-RNN, and Diffusson Policy in chal- lenging real-world visual domain shift scenarios, where distractors appeared as clutter. addition to the quantitative results reported Q4 and Tab. 4. we observed highly desirable behaviours from RoboSaGA during testing under severe visual domain shifts across three policies. These behaviours included successful task completion despite aggressive lighting variations, which introduced strong shadows from distractors, and maintaining task execution even with significant occlusions in one view and background changes with added clutter (Fig. 4).

---

Eye-in-Hand
Eye-in-Hand
Eye-in-TTand
(a) Reaching Nut
(b) Lifting Nut
(c) Inserting Nut
Figure 5: Saliency Maps across Two Views during different stages of task execution (BC-RNN).
Q6: How closely do saliency maps correspond to human expectations of task relevance?
Saliency maps do not always align with the robot parts or targets that humans consider critical. As shown in Fig. 5a, while reaching for the nut, saliency highlights the general area around the gripper and nut, but the eye-in-hand view focuses only the handle. When lifting the nut (Fig. 5b), the saliency shifts away from the object. During insertion (Fig. 5c), the front camera overlooks the target peg and nut, while the eye-in-hand view focuses on the peg and hole. We hypothesise that this salient misalignment with human intuition occurs because vision-ban based BC integrates multi multi-sensory inputs (e.g., proprioception, observation history, multi-views) and accounts for task progression. Notably, history-independent BC-MLP produces more inter-iitterree saliency maps than its history-dependent counterparts. Examples are provided in Appendix D.
5 Limitations
Computation Overhead. The implementation of saliency buffer improves efficiency, but the computational demands remain considerable. For example, saliency computation with ResNet18 encoder on 84 x 84 inputs takes about 1.5 times longer than training. To improve efficiency, approx- imating saliency at mid-feature layers or exploring alternative saliency extractors could help. For policies with larger encoders, knowledge distillation may offer a solution, where a smaller encoder trained without augmentation, and saliency computed in later epochs to create an offline buffer.
Performance Degradation of BC-MLP and BC-RNN in Transport. As shown in Tab. 10, 11, and 12 in Appendix C, while the Diffusion Policy exhibits the expected improvements under the presence of distractors and changes in backgrounds, BC-MLP and BC-RNN demonstrate comparatively lower gains across all tested methods. This discrepancy may arise from the policies' differing abilities to distil task-relevant information from images with high task-related pixel density. As depicted in Fig. 3, the second-pers persoon view in the Transport task is largely dominated by two manipulators, offering significantly less information about the targets.
Focused Task Exploration. Our real-world experiment showcased our approach through a concen- trated evaluation of 3-DoF pick-and-place task, offering valuable insights under specific condi- tions. Future work could extend these findings to a broader range tasks.
Conclusion
In this work, we aimed to enhance the robustness of vision-based behaviour cloning against visual domain shifts, particularly with background changes and object distractors. using superimposition- based image augmentation. Our empirical results from both simulated and real-world experiments demonstrate that overlaying random images onto training data is a cost-effective method that sig- nificantly improves performance under visual domain shifts, reducing the performance gap from 0.60 to 0.24 in simulations and from 0.71 to 0.18 in real-world tasks. We introduced RoboSaGA, which uses input saliency maps to adjust augmentation intensity dynamically. This method further reduces the performance gap to 0.14 in simulations and 0.05 in real-world settings, delivering rel- ative improvements of 41.6% and 72%, respectively, compared to Random Overlay. RoboSaGA has revealed the extent of task-relevant semantics hidden within visual features, and the application of it for improving robustness against visual domain shifts. We excited to explore how these task-specific insights can further advance vision-based behaviour cloning.

---

Acknowledgments
This work has been supported through WASP Wallenberg AI, Autonomous systems and Soft- Program. Zheyu would also like to thank Michael Welle for his valuable feedback the manuscript.
References
[1] P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong. J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning (CoRL), 2021.
[2] C. Chi, Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023.
[3] Levine, Pastor, Krizhevsky, Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421-436 2018.
[4] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. Science and Systems (RSS), 2023.
Robotics:
[5] A. Mandlekar, S. Nasiriany. B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. Conference on Robot Learning (CoRL), 2023.
[6] Mitrano and D. Berenson. Data Augmentation for Manipulation. Proceedingg of Robolics; Science and Systems, New York City, NY, USA, June 2022. doi: 1021. 031.
[7] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[8] A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint arXiv:2307.03659 2023.
[9] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, 2020.
[10] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /tt::100..040 04552, 148.52200.04522 2017.
[11] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. arXiv preprint arXiv:
2017.
[12] D. A. Pomerleau. Alvinn: An autonomous land vehicle neural network. Conference on Advances in Neural Information Processing Systems (NeurIPS). 1988.
[13] R. Rahmatizadeh, Abolghasemi, I.. Bölöni, and Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. International Conference Robotics and Automation (ICRA), 2018.
[14] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Conference on Neural Information Processing Systems (NeurIPS), 2020.

---

[15] C. Gong, D. Wang, M. Li, V. Chandra, and Q. Liu. Keepaugment: A simple information- preserving data augmentation approach. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 1055-1064, 2021.
[16] A. F. M. S. Uddin, M. S. Monira, W. Shin, T. Chung, and S.-H. Bae. Saliencymix: A saliency guided data augmentation strategy for better regularization. In International Con- ference Learning Representattinss 2021. URL https://openrevieet -MOQkvBGTTq.
[17] Abolghasemi, A. Mazaheri, M. Shah, and L.. Boloni. Pay attentiion-rrouustiiinng deep visuomotor policy through task-focused visual attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4254 4254-4262, 2019.
[18] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang. J. Singh. C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. Robotics: Science and Systems (RSS), 2023.
[19] K. Lee, K. Lee, J. Shin, and H. Lee. Network randomization: simple technique for general- ization in deep reinforcement learning. International Conference on Learning Representa- tions, 2020. URL https: bttps://openreviess http://rrreeeeeooooooooottttt///
[20] N. Hansen and X. Wang. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1361 1- 13617. IEEE, 2021.
[21] Z. Yuan, G. Ma, Mu, B. Xia, Yuan, Wang, P. Luo, and H. Xu. Don't touch what mat- ters: Task-aware lipschitz data augmentation for visual reinforcement learning. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI-22, 2022.
[22] S. Srinivas and F. Fleuret. Full-grade Full-gradient representation for neural network visualization. Ad- vances in neural information processing systems, 32, 2019.
[23] R. R. Selvaraju, M. Cogswell, Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pages 618 2--26,,221 2017.
[24] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: / 706. 001777060.322 03825, 2017.
[25] M. Laskin, Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pages 5639-5650. PMLR, 2020.
[26] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[27] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519. IEEE, 2016.
[29] S. loffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448 456456. pmir, 2015.
10

---

[30] Y. Wu and K. He. Group normalization. In Proceedings of the European conference on com-- puter vision (ECCV), pages 2.18.
2018.
[31] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690 2014.

---

Appendix A: Performance and Computation Trade-off of Saliency Buffer
Saliency Buffer Implementation Details
Image Global Index:
Each image sample is assigned a global index, which is included as additional data returned by the data loader.
Random Crop: When random cropping (pixel shift) is enabled, cropping randomisers return the cropping parameters, which include the pixel coordinates of the top-left comerr and the cropping size. Assuming no additional down-sampling when saving, the buffer maintains the saliency map at its original size before cropping. saliency map derived from cropped image padded with zeros the regions cropped out to match the full image size. Upon retrieval, saliency maps are adjusted the specified cropping parameters.
Each saliency map stored in the buffer is kept as a single-channel 8-bit unsigned integer (UINT8) image to optimise memory use. For augmentation, saliency maps are normalised with pixel values scaled to [0, 1]. These values are converted to UINT8 when saved and back float32 (FLOAT32) within [0. upon retrieval.
Pixel Range Conversion:
Saliency Warm-up: The buffer does not update and returns all-one saliency maps dur- ing the first epochs (default set to 10), allowing the encoders to develop task-specific knowledge before strong data augmentation is applied. This warm-up strategy is also im- plemented the RoboSaGA variants: Random Overlay and Guided Erase.
Performance and Computation Trade-off
Lift
Can
Square
Buffer
who Buffer
Buffer
who Buffer
Buffer
w/o Buffer
In-domain
0.98
0.99
0.97
0.95
0.68 20.04
0.69
Background
0.89
0.9210.02
0.87
0.79
0.39
0.43...00
Distractor
0.95
0.98 20.01
0.71
20.04
0.69 00.94
0.67 20.04
meun
0.92 10
0.95
0.79_0.02 0.73 oz
0.54
0.55 g0.0
Table 5: Per-task Success Rate (f) of RoboSaGA and the standard error of the mean with and without Saliency Buffer under visual domain shifts (BC-MLP).
Buffer
w/o Buffer
domain
0.8820.02
0.88.10.2
Background 0. 0.7222.02
0.71+0.02
Butfer No Buffer
Distractor
0. 0. 0.77
0.22s
0.70s
01
0.74
Table 7: Average Processing Time for Per-batch Augmentation
Table 6: Average Task Success Rate (1) of RoboSaGA with and without Saliency Buffer under VDS (BC-MLP).
Here, we compare the performance and computational differences between utilising a saliency buffer and directly computing the saliency. In the RoboSaGA experiments, detailed in Sec. 4.1, 10% of the current batch is sampled for saliency updates and saved in the saliency buffer; the augmentation ratio is maintained at 50% of the current batch. By contrast, when the buffer is disabled, saliency maps are directly computed for 50% of the batch. The experiments are conducted on the Lift, Can, and Square tasks in simulation using the BC-MLP policy.
As shown in Tab. 5, 6, the average success rates over three tasks for RoboSaGA, with and without the saliency buffer under visual domain shifts, are 0.75 and 0.74, respectively. No significant perfor- mance difference is observed. While the use of the saliency buffer does not enhance performance, significantly reduces the computation time required for saliency extraction to one-third (Tab. 7).
12

---

Appendix B: Experimental Details
Task Descriptions
Action Dimension
Observation Dimension
Proprio. Dimension
Max. Steps
Long Horizon
Num. Subtaska
Task
Simulation
Lift
2 x
84
x
84
7
200
No
X
84
200
No
Square
2 84
84
200
No
Transport
x
84
x
200
Yes
Real-world
Toy
2 x 84 84
12
160
No
Table 8: Task Summary. Action Dimension: Robot action dimension, including end-effector 6- DOF velocity and parallel gripper width. Observation Dimension: Number of views x Image ob- servation dimension. Proprio. Dimension: the dimension of robot proprioceptive states. Num. of Demos: Number of demonstrations provided. Maximum Steps: Maximum rollout steps in simula- tion. Long-Hore Hong-Horizon: Indicates whether the task requires learning multiple behaviours together. Num. of Subtasks: Number of sub-tasks within each task.
In this work, we utilise four simulation tasks from RoboMimic [7] with provided proficient human (PH) demonstrations, and one real-world pick-and -eck-and- task collected via proficient human tele- operation. All tasks employ Franka Panda as the manipulator within 7-dimensional action space, which includes the 6 degrees of freedom for end-effector pose and gripper width. In the simulation, the proprioceptive states include the end-effector pose (rotation is represented by quaternion) and gripper width. The real task utilises a 12- 12-dimensional robot state represented by a flattened SE(3) matrix (last row omitted). Except for the Transport task, all tasks use two camera views: a second- person camera and a first-person camera. The Transport task features two camera views associated with each manipulator. Task descriptions and details are summarised below and in Tab.
Lift: Grasp and lift red cube from the table.
Can: Grasp Coke Can from the left bin (sub-task 1), and place it into the corresponding target bin on its right (sub-task 2).
Square: Grasp the square nut by the handle (sub-task 1) and insert it into matching square peg (sub-task 2).
Transport:
Left arm: Pick the handle of the source bin's 's lid (sub-task 1), place the lid in the empty space (sub-task 2), grasp the hammer within the source bin (sub-task 3), and deliver the workspace within the right arm's 's reach (sub-task 4).
Right arm: Pick up a red cube from the target bin (sub-task 5), place it into the trash bin (sub-task 6), take the hammer delivered by the left arm (sub-task 7), and place into the target bin (sub-task 8).
Toy: Pick up green squashy toy (sub-task 1) and place it into yellow cup (sub-task 2).
The location of the toy and the cup varies within a 10 x 10 cm2 area.
Policy and Training Details
BC-MLP and BC-RNN utilise the default network configurations as specified in RoboMimic [7]. Specifically, each image observation is processed by a ResNet18 [27] followed by a spatial-softmax layer [28]. All observation features are then concatenated with the robot's proprioceptive states, forming single feature vector. The flattened states are fed into either MLP RNN. The MIP employs ReLU activations, while the RNN consists of 2-layer LSTM. The final layer's
13

---

Hyperparameter
BC-MLP
BC-RNN
Learning Rate (LR)
1 10
1 10
Actor MLP Dimensions
[1024, 1024]
RNN Hidden Dimension
1000
RNN Sequence Length
10
GMM Number of Modes
3
Image Encoder
ResNet18
ResNet18
SpatialSoftmax (num-KP)
64
64
Table 9: Hyper-parameters for BC-MLP and BC-RNN.
hidden states are fed into the downstream Gaussian Mixture Model (GMM). As described in RoboMimic [7], during the rollout with GMM policies, the learned standard deviations of each mode are replaced with 1 x 10-4. arper-parameter settings are detailed in Tab. 9.
Diffusion Policy employs the hybrid-CNN architecture as detailed in [2]. utilises a similar image encoder as described above but replaces BatchNorm layers [29] with GroupNorm [30]. The input horizon, action horizon, and action prediction horizon are set 2, and 16, respectively. The learning rate to 1 10-4
Networks are defaulted to train from scratch with Adam [31] optimiser (learning rate set to 1 x10* 1x10-*) +) Except for the simple Lift task is trained for 200 epochs, all tasks are trained for 600 epochs.
Out-of-domain Images for RoboSaGA
MSCOCO
Plain
Gradient
Chess
Perlin
Figure 6: Examples of out-of-domain images for data augmentation
RoboSaGA utilises approximately 6,000 out-of-domain images for augmentation, comprising 5,000 real-world images from MSCOCO [26) and 1,000 synthetic images featuring plain, gradient, grid, chess, and Perlin patterns (Fig. 6). All images are subject random rotations and brightness adjust- ments.
Lighting and Shadow Evaluation Setup
Lighting Changes Shadow Rendering Off
Lighting Changes Shadow Rendering On
Figure 7: Examples of lighting and shadow variations simulation evaluation.
14

---

We conducted two sets of simulated experiments on various policies, including BC-MLP, BC-RNN, and diffusion policies, across the four simulated tasks, examining the impact lighting changes (varying intensity and colour by adjusting the diffuse parameter between 0.2 and 0.8) with shadow rendering both enabled and disabled (shadows turned off in the training data). The experimental setup, tasks, and evaluation protocol are identical to those in the submitted version, with random cropping applied by default.
Backgrounds and Distractors for Evaluation
Floor Texture
Table Texture
Wall Texture
Distractors
Figure 8: Examples of Lighting and distractors in simulation evaluation.
Textile Set
Distractor Set
Background Shuffle
Distractor Shuffle
Figure 9: Examples of textiles and distractors in real-world evaluation.
In the simulation, as illustrated in Fig. 8, table textures are derived from images of textiles and patterns. Floor textures utilise common materials such as tiles and wooden flooring, while wall textures are selected from both indoor and outdoor scenes. Distractors, including items like a Coke can, bottle, cereal, bread, and lemon, form the default prevent duplicating task-specific objects. For example, the Coke can is excluded from the Can task. Distractors are strategically placed to minimise collisions with the manipulator and targets, although collisions are still possible. Each selected distractor has 50% chance of appearing.
Some distractors may be removed to
In the real-world setting, as shown in Fig. 9, background shuffling is achieved by varying the combinations of textiles drawn from the default textile set. These textiles sufficiently cover the field of view for both the second-person camera and the eye-in-hand camera, with examples of the respective fields of view illustrated in Fig. 4. Distractors are also drawn from the default distractor set, forming cluttered arrangements.
15

---

Appendix C: Full Tables
All data points in the tables presented mean + the standard error of the mean.
6.1 Simulated Experiments
Crop
+Jiter
+S0DA
+Ovex
+SaGA
In-domain
0.97+0.01
+o.00
1.00
0.99+
0.98++..11
Lighting
0.27.20.04
0.97
0.97
0.99
0.99
Lighting & Shadow
0.21 e3
0.7510.04
0.9710.01 0.94
0.9710.01
Background
0.00 oo
0.0320.02
0.80
0.87 e3
0. 0.8940.03
Distractor
0.2110003
0.31g0.01
0.96_o.02
0.6620.04
0.95
In-domain
0.95
0.91
0.93g0.02
0.92+
0.97+
Lighting 0.422...4 04
0.69
0.93.o.02
0.93 o2
0.981
Lightnng & Shadow
0.19 os
0.5320.04
0.9320.02
0.87 os
0.95
Background
0.03+0.01
0.3840.04
0.65=0004
0.79+0.08
0.87+0.08
Distractor
0.4340.04
0.33
0.57
0.53+0.04
0.7140..0
In-domain
0.62+0.04
0,53+0.04
0.47=0.04
0.51+
0.68++o.04
Lighting 0.02 01
0.43
0.59
0.59 04
0.62
Lighting & Shadow
0.00
0.09
0.46.0.04
0.33+0.04
0..1
Background
0.00+0.00
0.0140.01
0.07
0.25+0.04
0.39.+0.04
Distractor
0.37:20.4
0.37
0.41
0.42g0.04
0.69
In-domain
0.49+0.04
0.47+0.04
0.41+0.04
0.37+00.04
0.46++0.04
Tamqprrt
Lighting
0.07+0.02
0.33
0.20
0.36 04
0.49
Lighting & Shadow
0.00+0.00
0.07+0.02
0.25==004
0.35++0.04
0.33++0.04
Background
0.00
0. 0.00
0.01
0 0.05
0.05
Distractor
0.21 03
0.
0.23gn.03
0.21
0.3410.04
Table 10: Success Rate (f) of BC-MLP with Random Crop, Colour Jitter, Random Overlay, SODA and RoboSaGA under visual domain shifts.
Crop
+Jitter
+S0DA
+Over
+SoGA
Indommiin
0.97
0.97
0.95
1.00
0.99
Lighting
0.43
0.93
0.95
1.00 00
1.00
Lighting Shadow
0.17 os
0.6640.04
0.95
0.99
0.99
Background
0.00+0.00
0.0140.01
0.65+0.04
0.92+a.02
0.98++..02
Distractor
0.21 as
0.1940.03
0.9220.02
0.9610.02
0000
In-domain
0.97+a.01
0.9740..1
0.96+0.02
0.97+a..3
0.97+0.01
Cán
Lighting
0.65 64
0.8540.03
0.94
0.95 e2
0.94
Lighting Shadow
0.4040.04
0.814o.aa
0.93
0.85+0.03
0.92+40.02
Background
0.0220.01
0.21400.0
0.69
0.714a..4
0. 00.00
0.32 64
0.
0.64go.04
0.5510.04
04
In-domain
0.66 04
0.68
0.74
0.692..04
0.7540.04
Square
Lightng
0.03 o1
0.0610.01
0.68
0.73
0. 0.8210.00 03
Lighting Shadow
0.00 no
0.2544.04
0.61.2.04
0.62+a..4
0.
Background
0.00
0.0210.01
0.22g0.03
0.3810.0s
0.47
0.33 04
0.40
0.38
0.5940.04
0.
In-domain
0.61 04
0.61
0.65.o.os
0.59
0.
Transport
Lightnng
0.35 04
0.39
0.5920.04
0.62
0. 0.
Lighting
0.01 o1
0.2240.03
0.53
0.5210.04
04
Background
0.00
0.0010:00
0.05go.02
0.14
0.2011..3
0.24+0.03 03
0.30
0.30+0.04
0.35
0.41
Table 11: Success Rate of BC-RNN with Random Crop, Colour Jitter, Random Overlay, SODA and RoboSaGA under visual domain shifts.
16

---

Crop
Jitter
+SODA
+Over
+SaGA
In-domain
1.00
1.00
10000
1.00
0.00
Lightnng
1.00 so
1.00
1.00
1.00 00
1.00
Lighting & Shadow
1. 1.00 oo
1.00+0.a0
1.00=0.00
1.00=0.00
1.0040.00
Background
0.861 o3
0.8610.03
0.98
0.98
0.93
Distractor
0.55
0.5540.04
1.00=0.00
1.00
1.00+0.00
In In-domain
1.00
0.96
0.97
0.98go.01
0.9710001
Lighting Lighting & Shadow
0.93 02
0.97
0.93
0.98
0.94+0.00
0.80+0.03
0.82+0.03
0.93+0.02
0.97+0.01
0,92+0.02
Background
0.55:20.04
0.72:10.4
0.75.20.0
0.87:2..3
0.91:0.02
Districtor
0.56+0.04
0.38++0.04
0.77+0.08
0.75+0.04
0.80+
In domain
0.93 02
0.00
0.87
0.91
0.91
Square
Lighting Lighting & Shadow
0.45+0.04
0.87+0.00
0.86+0.09
0.89
0.85++0.0
0.12:g0.3
0.37
0.83 20.00
0.78
0.80+20.0
Background
0.01
05
0.6320.0s
0.59
03
Distractor
0.48
0.63
0.81
0.89
0. 0..02
In-domain
0.002
0.94
0.90
0.90
0.89+0.03
Thamsport
Lighting Lighting & Shadow
0.81
0.87 0..78
0.80
0.85
0.81+o.0s
0.4810 04
0. 0.63100.4
0.7720.03
0.85
0.8210.00
Background
0.00 oo
0.00
0.33
0.45
0.5811.04
Distractor
0.41+0.04
0.32+0.04
0.61#0.04
0.58+0.04
0.65+0.04
Table 12: Success Rate (t) of Diffusion Policy with Random Crop, Colour Jitter, Random Overlay,
SODA and RoboSaGA under visual domain shifts.
6.2 Real Experiments
BC-MLP
BC-RNN
Diffusion Policy
Crop
+Over
SaGA
Crup
+Over
+SaGA
Crop
+Over
+aGGA
In-domain
0,70+0.11
0.85
0.85
0.85
0.95
0.90=0.07
1.00
1.00+0.90
1.00
Backgroand
0.00
0.50
0.75 0.75
0.00 so
0.60 0.85 0.85_o.0s
0.004 oo 0.65
0.75
Distractor
0.05
0.65
0.80
0. 0.15
0.6540.11
0.70
0.65
0.95
0.95
Table 13: Real-world Success Rate (1) for Each Policy: Random Crop, Random Overlay, and RoboSaGA on the Toy task against distractor and background variations.
Appendix D: Interpretability Misalignment in Saliency Maps
Here, we select three representative examples from the Transport task to illustrate the potential misalignment between input saliency from the policies and human interpretation.
OOD
Random
Guided
Rebe-
FullGraad Salieney
Image
Overlay
Frase
SuGA
Left
Right 2nd
Figure 10: Examples of Saliency Visualisations and Data Augmentations of Transport Task from BC-MLP. EiH: eye-in-hand camera. 2nd: accon-pperson-camera
As observed in Fig. 10, the saliency maps produced by the -istor--indeeendentt BC-MLP align more closely with human intuition compared to those from BC-RNN (Fig. 1la) and the Diffusion
17

---

000
00D
Random Overlay
Gaided
Robe-
FullGrad
Image
Overlay
SaGA
Suliency
EIE!
Right EIEE
Left 2nd
Left 2nd
Right Ind
Right Znd
(a) BC-RNN
(b) Diffusion Policy
Figure 11: Examples of Saliency Visualisations and Data Augmentations of Transport Task from BC-RNN and Diffusion Policy. KiH: eye-in-hand camera. 2nd: second-person-cameraa
Policy (Fig. 11b). these visualisations, the Left Eye-in- Syy-in-Hann Hand (EiH) camera focuses on the box handle, while the Right EiH camera does provide task-critical information, exhibiting random focus instead. The left second-person camera focuses the right arm, and the right second-person camera focuses on the right hand. Notably, the left second- person camera does not focus on the lid handle, as this piece of information is already provided by the left EiH camera. However, despite the alignment of the saliency maps with human intuition, experimental results indicate that this alignment does not necessarily translate into improved robustness against background variations. Indeed, none of the tested augmentation methods enhanced the performance of BC-MLP in the Transport task (see Tab. 10, 11, 12).
Although achieving higher robustness against background variations, the saliency maps produced by the history-dependent BC-RNN and Diffusion Policy, in contrast to the history-independent BC- MLP, are less interpretable from a human perspective. These maps can focus on elements considered trivial by humans. Given that the task relies on historical observations, the robot's states, multi- views, and exhibits varying levels of visual dependency throughout its execution, we argue that what humans perceive visually important may not always align with the network's focus.
18


Reviews:
[{"review_id": "OnhSUAZE8Q", "reviewer": "Reviewer_Kp3m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The author propose an image augmentation scheme called RoboSaGA for policy learning. The augmentation is $I = M \\odot x + (1-M) \\odot x_{\\text{ood}}$ where the mask, $M$ comes from a saliency map. This augmentation decreases performance degradation in the presence of background changes and the presence of distractors. They test their augmentation with BC-MLP, BC-RNN, and Diffusion Policy across 3 Robomimic tasks and one real-world task.", "summary_of_recommendation": "Leaning reject unless more answers can be provided about lighting shifts and baseline data augmentations.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. What data augmentations are used in the baseline model and with what parameters? How well does a policy with RandomCrop, ColorJitter, and RandomRotation perform?\n\n2. How does RoboSaGA perform under lighting distribution shifts?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K6aNJ4u29s", "reviewer": "Reviewer_Nf11", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors proposes to guide superimposition-based augmentation with model saliency with validation on simulation/real-world manipulation tasks", "summary_of_recommendation": "Recommending weak accept -- happy to raise my scores if there would be clarifications/experiments to the questions/comments above.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I think the paper in its current form may have a limited impact due to the potential problems with saliency quality. It would be interesting to extend the scope of the paper to more general task-relevance notions which this reviewer belives would greatly improve the applicability/significance of the paper.\n\nFor the simulated environment task, would it be possible to extract the ground-truth task-relevant pixels? This would be an interesting baseline to consider and somewhat establish an upper bound of task-relevance-guided overlay. In practice, such maps could potentially be obtained through image segmentation models in certain domains.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bJGRNXVzy9", "reviewer": "Reviewer_wTGw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors' primary assertion is that traditional augmentations like pixel shift are effective in boosting in-domain performance; however, these methods are not as effective for visual domain shifts (i.e. shifts where there are other objects/background is different). To improve performance for visual domain shifts, it is crucial to superimpose additional objects or images onto the training image. However, this approach can result in information loss for behavior cloning. Therefore, the authors propose a saliency-guided approach to augmenting existing data.", "summary_of_recommendation": "The paper is very clear and easy to follow. The performance improvement is fairly significant and the experiments performed are thorough. Some additional analysis could be provided to improve the overall quality. In terms of originality though such methods have already been explored or published in the field of computer vision, but not in the context of behavior cloning. Hence my recommendation for a weak accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "A few major points:\n1. A discussion on fullgrad in the related work section would help the reader understand the motivation behind using this over other saliency methods.\n2. As per my understanding, consecutive images within a trajectory could get very different augmentations, i.e. different backgrounds which is unlikely to happen in real-world scenarios. Wouldn’t this mean that data could be out of distribution for consecutive images making saliency maps for images later in the trajectory a little arbitrary, especially for BC-RNN and Diffusion Policy?\n3. It would be interesting to see how the saliency maps look for time dependent networks, especially BC-RNN and Diffusion Policy in the main paper as well\n4. Does it make sense to use this method in the full training cycle? The saliency map might not make sense in the very beginning - would this cause slower convergence?\n5. The abstract mentions that pixel shift is not effective for visual domain shifts, but would like to see this backed up by citations or experiments. All the experiments compare only overlay based methods with each other\n\nMore minor changes/typos:\n1. Line 211 - \"Another unspecified method\" is a typo\n2. Line 255 - \"Distil\" -> \"Distill\"\n3. Line 166 - \"Thus\" should be \"this\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: AC summarizes the strengths and weaknesses of the paper based on the reviewers' comments as follows:

**Strength:**

- The paper is overall well-written
- The idea of using saliency for data augmentation is reasonable
- The results reported in the paper are promising.

**Weakness:**

- The idea is similar to that of KeepAugment
- There are some unclear points regarding experiments

=== comments after the rebuttal ===

In the rebuttal, the authors provided additional experiments on lighting variations, which demonstrate the advantage of the proposed method. The reviewers appreciate the authors' response and agree that the paper meets the criteria of CoRL.
Decision: Accept