Simple Masked Training Strategies Yield Control Policies That Are Robust to Sensor Failure
Authors: Skand Skand
Bikram Pandit
Chanho Kim
Li Fuxin
Stefan Lee
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Robustness
Sensorimotor Learning
TL;DR: We use a multimodal encoder with modality dropout to train policies that are robust to sensory failures during deployment.

Abstract: Sensor failure is common when robots are deployed in the real world, as sensors naturally wear out over time. Such failures can lead to catastrophic outcomes, including damage to the robot from unexpected robot behaviors such as falling during walking. Previous work has tried to address this problem by recovering missing sensor values from the history of states or by adapting learned control policies to handle corrupted sensors through fine-tuning during deployment.
In this work, we propose training reinforcement learning (RL) policies that are robust to sensory failures. We use a multimodal encoder designed to account for these failures and a training strategy that randomly drops a subset of sensor modalities, similar to missing observations caused by failed sensors. We conduct evaluations across multiple tasks (bipedal locomotion and robotic manipulation) with varying robot embodiments in both simulation and the real world to demonstrate the effectiveness of our approach. Our results show that the proposed method produces robust RL policies that handle failures in both low-dimensional proprioceptive and high-dimensional visual modalities without a significant increase in training time or decrease in sample efficiency, making it a promising solution for learning RL policies robust to sensory failures.

Simple Masked Training Strategies Yield Control Policies That Are Robust to Sensor Failure
Skand Peri, Bikram Pandit, Chanho Kim, Li Fuxin, Stefan Lee
Oregon State University
Project Page: https///posaaad..ghhub..///prers
Abstract: Sensor failure is common when robots are deployed in the real world, sensors naturally wear out time. Such failures can lead to catastrophic outcomes, including damage to the robot from unexpected robot behaviors such as falling during walking. Previous work attempted to address this problem by recovering missing sensor values from the history of states or by adapting learned control policies to handle corrupted sensors through fine-tuning during deploy- ment. In this work, we propose training reinforcement learning (RL) policies that are robust sensory failures. We multimodal encoder designed ac- count for these failures and training strategy that randomly drops subset of sensor modalities, similar to missing observations caused by failed sensors. We conduct evaluations across multiple tasks (bipedal locomotion and robotic manip- ulation) with varying robot embodiments in both simulation and the real world to demonstrate the effectiveness of our approach. Our results show that the proposed method produces robust RL policies that handle failures both dimdeeesiional proprioceptive and high-dimensional visual modalities without significant in- crease training time decrease in sample efficiency, making it promising solution for learning RL policies that are robust to sensory failures.
Keywords: Reinforcement Learning. Robustness, Sensorimotor Learning
Original
Shut Down
(No sensor fallure)
RME
Time
Figure 1: Motivating Example: When the left-shin sensor of the bipedal robot Cassie fails, simple shut down operation leads it to topple down (middle) as opposed to our RME trained policy (right) which continues to walk similarly to the scenario with no sensory failure (left).
Introduction
Sensory failure in robots is both pervasive and unavoidable. What should robot next when has detected a sensory failure or corruption? A shut down action is one option that an agent could afford; however, doing safely for dynamical systems with complex robot morphology in uncertain terrains is challenging control task in its own right. A straightforward option that cuts power to the motors could result in hazardous behavior, causing harm to the robot itself and other entities in the environment. For instance, the bipedal robot shown in Figure (middle) immediately falls down when power is cut during a squatting motion.
In such scenario where the robot detects sensory failure, one would like the robot function to the best of its abilities and not collapse. For instance robot failing in warchouse would
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

cause disruptions to the operations. Additionally, consequences of such failures can be exacerbated for robots deployed for critical rescue operations [1] or in scenarios where immediate repair is unavailable [2]. Hence, is crucial to have robots that are robust to sensor failures to help mitigate the risk of catastrophic outcomes and provide best-effort control until repairs can be made.
Several works have attempted address the lack of sensory robustness from various perspectives. From a classical control viewpoint, robust model predictive control (MPC) has been thoroughly examined [3, 4]. For example, common approach to managing sensor corruptions as demonstrated in [5], involves incorporating an additional feedback controller to address the deviations caused by the corruption. On the learning end, [6. 7, 8] adopt a continual learning setup where the control policy of the robot is adapted explicitly to the missing sensor during deployment, and [9, 10, 11] impute the missing values of the sensors from the history of states. The above proposed works try to address this issue after the control policy is trained by either adapting the policy training another model to correct the values of the missing sensors (imputation networks).
Instead of addressing this issue in a post hoc manner, this work aims to train reinforcement learning (RL) control policies that are robust to sensory failures, enabling the robot to function without further fine-tuning or adaptation during deployment. As an example, we demonstrate this phenomenon in an experiment with Cassie the bipedal robot for the task of blind walking in Figure 1. When standard walking policy trained with RL [12] is deployed on the robot with a broken left-shin, native shut down operation leads to Cassie falling down (middle). However, when the policy takes into account such a failure we barely observe a difference in gaits of the walking policy (right). In this work, we focus on the setting where sensor failures are presumed to already be detected and develop best-effort policies that are robust to this loss of information.
To achieve this, propose straightforward multimodal encoder and incorporate simple modal- ity dropout schema where we randomly drop subset of sensors during training so that the resulting policy is robust to missing sensors during test time. Further, we also extend this idea to multiple high-dimensional modalities such as multiview RGB and depth images in robotic manipulation and demonstrate its effectiveness for model- based and model-free reinforcement learners. In contrast to prior work on learning masked multimodal visual features in model-based RL. [13, 14], our approach includes both low- and nigh-dimensional sensor modalities, does not require modality reconstruc- tion, and results in policy that can operate on arbitrary subsets of modalities during deployment.
Our experiments demonstrate that the proposed approach is effective across multiple robot embod- iments, diverse tasks (tabletop pick & place, mobile manipulation, and bipedal locomotion), and even in sim2real deployment of a highly-dynamic bipedal robot. For most tasks, our approach in- curs only a minimal increase of wall-clock training time and decrease in sample-efficiency making it easy-to-implement,, efficient, and effective strategy to produce policies robust to sensor failure.
2 Related Works
Sensor value imputation. One common technique to deal with missing modalities is to predict them. This is well studied problem in databases [9], machine learning [15, 16, 17, 18, 19] and model predictive control (MPC) [5, 3, 4]. In RL from high-dimensional sensory inputs such as RGB images, InformedDreamer [11] proposed to predict privilege modalities from latent state however, one major drawback of this approach as noted in [20], that not all privilege information predictable from a set of input modalities. On similar lines [21] also proposed to predict linear and angular velocity for autonomous vehicles in the context of off-road robot driving. They also observe that predicting dinenssinnii modalities in multi-modal world models can be difficult and lead to blurry outputs in RGB images and lidar maps.
Handling missing modalities: Another line of works attempted to learn to generate state rep- resentations robust to missing modalities without recovering them explicitly. For instance, [22] introduces CCM address the sensor corruption problem in RL for images and force sensors. CCM trains multimodal encoder that is flexible with respect to missing modalities by representing latent state distribution as product of per-modality distributions. However, proprioceptive
2

---

observations are never corrupted in [22] and their experiments were limited to single peg-in-hole setting with only single-modality failures.
[23] proposed a Sensor Dropout (SD) strategy for training an RL agent for autonomous driving. in which, instead of randomly dropping neurons in standard dropout, they randomly select modali- ties and zero out their features. They employ an MLP to fuse the modality features. While MLP is suitable architecture for lower numbers of modalities, we find that its performance can suffer for higher numbers of modalities, as shown in our experiments in Section 8.11. Additionally, this method was demonstrated with only three modalities on a single task with a 2-d action space. In con- trast, we perform a wide range of experiments ranging from four high dimensional modalities (im- ages and depth maps) and up to seven owwdimensional sensors with multiple complex action spaces (6-d for Walker, 10-d for bi-pedal walking, 7-d for manipulation, 13-d for ioco-manipulation tasks).
Multimodal policies. Another way to address the availability of multiple sensors is using mut- limodal policies - policies trained on multiple inputs. Different observations may provide comple- mentary information that is helpful for the agent to achieve its task for example robotic manip- ulators are more sample-efficient when provided with first-person RGB inputs via a wrist camera [24]. Cross-modal policies that operate on multiple modalities [25, 26, 27] have been successful in achieving fine grained manipulation, but they end up fusing the representations from each modality, making it infeasible to function when subset of modalities is missing.
RPT [28] has investigated learning pre-trained representations for robot learning from a dataset of trajectories. In particular, they employ a masking mechanism where a subset of modalities are masked out and are predicted from the rest. However, this work differs from ours in two aspects: (a) their primary goal was to learn transferable pretrained representations for robotic tasks, and (b) RPT includes losses that encourage explicit reconstruction of missing modalities which is shown to be detrimental to performance [20] and we find the same. MUTEX [29] uses multiple modalities like text instructions, video demos, speech and image goals along with robot states to train policy that is robust to missing modalities. However, all multi-modal transformer based models such as MUTEX are limited to imitation learning where access to high fidelity video demonstrations are assumed. Our work focuses primarily on reinforcement learning policies where the agent has to learn the behavior from trial-and-en and-error where transformer based encoders have been empirically shown be unstable [30]. For more related works, please refer to Section 8.1.
3 Preliminaries
We pose our problem as an infinite-horizon Partially Observable Markov Decision Process (POMDP) [31] defined (S,A, O). represents the complete state space, which is usually not accessible to the robot. The observation space, O O x O²... x O is set of sensory ob- servations or modalities' such as proprioceptive state, multi-view RGB or depth images. A E Rd is a d-dimensional continuous action space, T O x A + O is the transition function, and R O + IR is the reward function. The goal of the agent is to learn a policy x O > AA that maximizes the ex- pected sum of discounted rewards; max, Ex, 12 R'R(s;))
)], where 7 E [0. 1) is the discount factor. While in principle, our proposed encoder design be applied any model-basee or model-free method, we show experiments on model-based TD-MPC2 [32] and model-free PPO [33] agents.
TD-MPC2. TD-MPC2 has shown strong results across diverse robotic environments. Like many other model-based RL algorithms, consists of (a) world model training and (b) policy learning.
World model learning. While most existing model-based methods such as Dreamer [34, 35, 36] learn generative model of the world through supervision that includes state reconstruction loss, TD- MPC2 learns an transition dynamic using supervision from rewards and TD-leaming [37] alone.
In this work, we interchangeably sensors and modalities.
3

---

Specifically, the TD-MPC2 world model consists of the following components:
Representation:
2t ho(or. 01-1,00-21
Dynamics:
2. fe(2--1 0(-1)
Reward:
F. R2(21)
(1)
Value:
4: Qa(2t,
Policy prior:
ā =Tp(=)) (x((1)
where o and a are the observations and actions and z is the learned latent which further used to
obtain a policy. Denoting the cross-entropy loss CE and the stop-gst soop-gradient operator as sg, the
world model is trained using the following objective:
( (8,0.r,
x saa-ssghhhh
sg(hs O 212))
CE(f.r) CE at
latent
roward
valuc
consistency
prediction
prediction
where the expectation is over samples from reply buffer collected under Tp. The latent consis- tency term ensures the predicted latent & is similar to the embedding obtained by the representation network h for the real observation time t. The remaining terms encourage the predicted reward i; and value d match discrete regression encoding [38, 36] of the true reward r, and TD-estimated value 4r-
((2,((Ee)) where Q is the exponential moving average of Q [39].
Policy learning. TD-MPC2 trains the policy prior with the maximum entropy RL [40] objective:
H
(6)
=
E
X (0Q(2,
mp(2))) - (H((= 2x))
(5
where 2, is given by equation (1). While training with the objective Lp, only network parameters of Tp are updated. TD-MPC2 employs gradient-free model predictive control via Model Predictive Path Intergral (MPPI) [41] that samples actions using rollouts from the learned world model. For more details, refer the reader to the TD-MPC2 paper [32].
PPO. For model-free agents trained with PPO [33], a policy network T(2) is optimized by using a clipped objective function to limit the magnitude of policy updates to maximize the expected discounted return directly, which is predicted by learned critic c(zz) that is trained to estimate value function simultaneously.
4 Learning Policies Robust to Sensor Failure
To support learning policies that perform well the face of sensor failures, we focus on design- ing representation encoder networks that are flexible to the number of input modalities and corre- sponding modality dropout training strategies that encourage strong policy performance even when individual modalities are unavailable. Our resulting Robust Multi modal Encoder (RME) is sim- ple to integrate with existing model-based and model-free algorithms, accommodates failures in both high-level (visual) and low-level (proprioceptive) sensors, and demonstrates similar (or better) sample-efficiency and wall-clock training time compared to single-modality baselines. The design of RME is straightforward and shows strong robustness to sensor failures when paired with modality dropout during training - in both simulated and real robotic deployments.
Robust Multimodal Encoder (RME). Our RME model consists of multimodal transformer and individual modality encoders (as shown in Figure 2).
Multimodal Transformer. A common choice when combining multiple modalities in other domains is to use transformers [42] as used in [43, 44, 45]. Though widely used as sequence models, trans- formers without positional encoders operate as order-invariant set functions and offer flexible way retain or remove modalities simply by excluding them from the input set. Considering M modal-
ities (1)) (M) and corresponding modality encoders f()), ...., f...... ((A) which map each modality to a d-dimensional observation encoding, we can write the multimodal transformer as the function TF,
(2) ..... (M)) (as) TF (1 + ...,,............. (mm) (mm) (m) + (m)) ((is))
(4)

---

2
Muttimodal Transformer
Worldmood Cortro Policy
MLF
[CLS]
CNN
GNNN
GNN
CNN
13
12
Input Unevailable
Figure 2: Robust Multimodal Encoder (RME): We propose a simple, yet effective modification to learn robust representations within existing RL frameworks. We consider each robot sensor as modality that can be potentially missing during deployment. sists of both iow-dimensional proprioceptive inputs as well as high dimensional image inputs that are encoded using appropriate encoders (MLP, CNNs) and fed into transformer. We employ different dropout mechanisms for low and high dimensional modalities. other low-d (ww-dimensional) input, we append a mask vector where 1/0 indicate the presence/absence of the sensor respectively. For high-dimensional modalities, we simply drop the modality shaded in gray while training.
The proposed RME architecture con-
Given the proprioceptive (or any
where e(m) is a d-dimensional learned modality type embedding akin those used in multimodal models [46] and (CCLS) is a learned input that always present to aggregate information across modalities [47]. When using the RME in model-based or model-free methods, use the output vector (CLS) as the representation encoding of the input modalities. In practice, individual modali- ties might be represented via multiple vectors, for instance representing visual modalities via many patches as done in vision transformer-base encodings [48, 49]. However, in this work, we con- sider only the single vector case to reduce computational costs from processing long sequences with transformers and to provide uniform interface for considered sensor modalities.
We note that the transformer-base based image autoencoder from [14] has similar structure for multi-view visual representation learning in MBRL- taking patch-based encodings of either single- or multi-view observations. However, [14] trains this model independently and then uses the feature encoder as representation for separate multi- or single-view policies; not producing a policy that is capable of adapting to sensor failure at runtime. In practice, many combinations of failed sensors are possible and learning independent policies for each is costly solution.
Encoding Visual Sensors. For high-dimensional sensors such as RGB or depth images, we can accommodate any encoder f that summarizes the image as a vector. For ease of comparison, we retain the image encoding network from TD-MPC2 [32] consisting of four conv layers with ReLU activations. The final spatial feature volume flattened into 512-dim vector. Each visual modality instantiates different parameterization of this architecture.
Accommodating Low-Dine Now-Dimensional Sensors. Proprioceptive states ofp) typically comprised of combination of K low-dimensional sensors such as IMUs and joint encoders where each sensor (r), og k K a (2.7)-dimensional real valued vector. In early experiments, we found treating each sensor as an independent input to the multimodal transformer (analogous to the visual features) was highly unstable for policy training. To overcome this while still allowing for individual sensors to fail, we adopt a different strategy for proprioception inputs. Specifically, for every sensor of we have corresponding binary mask M(()) which is set to 0 if sensor is dropped out, otherwise to 1. These masks are concatenated fole),,(f _MM()) 3 ...., M..,), that is fed to single modality encoder (f to obtain our representation for proprioception.
Modality Dropout, When sampling a training batch, we randomly drop a subset of the M sensor modalities. Specifically, we select random subset of modalities with random size I to M-1. We then drop out sensors from the subset individually with probability This scheme ensures that
5

---

least one modality always remains. Implicit in this method is the assumption that the remaining modalities are sufficient for the policy to reason about correct actions.
Experiments
The following experiments study three key questions: How robust is RME to proprioceptive sensory failures for locomotion tasks (Sec. 5.1)? Can RME scale with multiple high-dimensional modalities such as RGB & depth images and maintain its robustness sensory failures during deployment (Sec. 5.2y? And can control policy trained with RME transfer to the real world (Sec. 5.3)?
5.1 Sensor failures in robotic proprioceptive states
Experimental setting. We first consider tasks where individual sensors in the proprioceptive states of the robot encounter a sensory failure. We consider two different embodiments shown to the left of Figure 3 Walker from the DMControl suite [50] and a bipedal robot, Cassie For Walker, we show results on four different tasks (Walk, Run, Walk-back and Run Run-back) and for Cassie, we show results on walking. For walker, we consider sensors (torso, thigh (x/1), leg (x/1) and foot (r/1)) where and r indicate left or right. For Cassie, consider 4 sensors {tarsus (1/r) and shin (1/r) encoders}. In all our experiments, we do not drop any sensors that are part of the action space or are required by the underlying controller - in other words, the sensors that are dropped are non-actuated sensors. Baseline Walker policies are trained with TD-MPC2 whereas Cassic uses the model-free PPO algorithm [33] as in prior work [51]. For each, we present results when replacing the representation encoder with our RME model and training with modality dropout. For baseline policies, we failed sensors values of 0 during inference essentially performing fixed in-distribution imputation. We also show results from an imputation baseline where failed sensor values are predicted from history of 10 previous states using trained 3 layer MLP with LayerNorm [52] and Mish activation [53]. The final layer is linear transformation without any activations. For fairness, the imputation network is trained on transitions from the replay buffer collected during (TD-MCCC PPO} training. For Walker, we consider 200k transitions and for Cassie we train the imputation network on IB transitions in simulation.
Results. We consider single sensor failures and report the sensor failure that causes the minimum (largest impact) and maximum (smallest impact) return for the task well as the mean return over sensor failure conditions. For context, baseline policies without sensor failures achieve returns of 900-950 for Walker and 320-350 for Cassie. From results in Figure 3 (left), we observe that RME maintains comparably high average returns across different tasks and sensor failures. The simple imputation by zeros strategy sometimes leads to performant policies when the failed sensor does seem relevant to the learned policy (e.g., left-foot in Walker (Walk) max), but is catastrophic in the worst cases (e.g., min returns 5-20x lower than max for Walker). Using learned imputation narrows this variance while raising mean performance achieving lower max and higher min returns but still underperforms. For the more complex Cassie robot, zeros imputation fails to produce a viable policy entirely and learned imputation suffers significant reductions as well compared to RME. We hypothesize that errors in learned imputation may have compounding effects if predicted sensor values drive the policy to aat-of-distribution states where the imputation network is less well trained.
5.2 Sensor failures in high-dimensional modalities
Experimental setting. We also consider sensor failures in the context of robotic manipulation for high dimensional modalities such as images. Specifically, we conduct robotic manipulation experiments in simulation on the Maniskill2 [54] benchmark and consider first- and third person RGB and depth images as additional modalities. We consider three pick & place tasks (PickCube, StackCube, PickYCB) and two mobile manipulation tasks (OpenDrawer, OpenCabinet).
We consider both Easy settings where only single modalities (e.g., first-person depth) are dropped and Hard settings where all but one modalities are dropped. We report results for Easy in the

---

Robustness towards modality dropout during
Robustness of RME sensory failures in proprioception
deployment
Figure 3: (left) Robusmess of RME to sensory failures in proprioception: We compare policies trained via pro- prioceptive masking mechanism against imputation baselines. We find that RME based policy significantly robust to sensory failures. (right) Robustness towards modality dropout during deployment (Hard:) test. MMWM against others with only one of (RGB/Depth images) } along with robot's proprioception. We report the mean, min, max success rates averaged over 3 seeds. The sensor is of the format (RGB Color (CVDepth (D) Third Person (TP)/First Person (FP)) as marked below the numbers. indicates the mean and indicates the max/min values.
supplementary and Hard in Figure (right). For these experiments, consider the following baselines:
TD-MPC2: As an upper bound, we consider a TD-MPC2 trained policy with all modalities for each task and report performance without any sensor failure in the table.
Informed TD-MPC2: Inspired by InformedDreamer [11], we augment TD-MPC2 to reconstruct missing modality observations. Specifically,
have decoders for each modalities that take in the latent z decode back the missing observations. The decoders jointly trained using mean-squared error loss. To be consistent with RME, we apply modality dropout during training. RME Modality Dropout: We also consider using our RME encoder without applying modality dropout-trann training with all modalities and then simply removing failed sensors at inference time.
Results. Figure (right) shows results for the Hard scenario of only one high dimensional modality (along with proprioception). Note that (sensor) annotations for max and min refer to the retained sensor. In all tasks, RME significantly outperforms the Informed TD-MPC2 baseline (nearly 2x mean success rates across tasks) and the ablation without modality dropout. We find that RME retains approximately 70-80% of the policy performance without sensor failure on average. For pick & place tasks (first three rows), we observe that RME w/o Modality Dropout performs uniformly poorly resulting in policies that rely on cross-modal information that generalize poorly to sensor failure. For Informed TD-MPC2 and RME, we find consistent trends that retaining third third-person cameras leads to stronger performance than first-person. We provide detailed results for each modality in the supplementary.
5.3 Sensor failure in Sim2Real policy transfer
Experimental setting. We show Sim2Real transfer of RME based policy Cassie in blind lo- comotion. We test the robustness of our policy against failure of four joint encoders {left shin, right shin, left tarsus, right tarsus). Each of these sensors measures the angular position and velocity of their respective joint links. We train a robust model-free RME policy with PPO [33] and RME (denoted as RME-PPO), where we use proprioceptive masking as described above (Fig. 2). The RME takes in a 38 38-dimensional proprioceptive state as input and the policy outputs a 10-dim action. We train the policy with domain randomization [51] and single contact reward as proposed in [12]. We train the policy in two stages of 500M steps each. In the first stage, train with dynamics randomization and in the second stage we introduce random external forces to train a robust policy for Sim2Real transfer. Please find more details in the supplementary.
For evaluation of the policy the real world, we test the commandability of Cassie which measures how well the policy follows a given the user command. Concretely, evaluate two metrics commandability in linear direction (linear) which measures how much time it take for Cassie to cover distance of feet (2.43 meters) at 0.25 m/sec speed, and
Evaluation Protocol:
7

---

Commandabilit
Commandabilty
Cassie Walking
Stack Cube
Sensor failed
x-Iirecrion (sec)
Angular drift (sec)
300
ne Sewe hower
Base Policy
RME-PPO
(Ours)
Base Policy
No malen
RNE-PPO
(Curs)
Raturn
2000
None
7.96
1007
25.04
25.1
100
left-shin
25.21
1*ft-taraux
12,340
30,00
right ligh-ttaeus
xight-ahiis
13
x
8:10 2012
9.21 al4
32.46 2131
X
25. 1004
26.85 2005
RAD
PPC
16
10%6
1
both-shin
x
8.73 1175
x
25. 1077
all
7.65
25.37
Environment Stages
Table I: Commandability of Cassie: We evaluate how well Cassie responds to input linear velocity (x- direction) and angular velocity (turn-rate). Values averaged over 2 trials for each entry of the table.
Figure 4: Efficieeif of RME We observe either comparable (bipedal locomotion) sample-efficiency when compared to certain modalities (eg. Stack Cube; perr-p cam). Shaded areas are 95% bootstrapped CIs over seeds.
increase in
commandability in angular direction (angular) that measures how much time takes Cassie to rotate 360 (at fixed position) 15 degrees/sec. For well trained policy, the time for linear travel should be around 9.72 sec and time for rotation should be around 24 sec.
Results. Based on the results in Table 1, we observe that in most instances where a sensor has failed, the original base policy with fixed value imputation O, fails to walk and topples. In contrast, RME not only consistently walks without falling but the linear and angular commandability close the optimal time even when all the four sensors fail, showcasing the robustness that RME provides. We provide additional videos in the supplementary material.
6 Discussion and Limitations
Sample-efficieney, wall-clock time, and trainable parameters of RME: The additional parameters that RME adds on top TD-MPC2 are those for the multimodal transformer (2-layer, 4-head) an increase from a base 5M trainable parameters to 7.2M. Averaging over all RME+TD-MPC2 models presented in this work, adding RME increases wall-clock runtime of training by 72%; however, Walker skews this substantially and the average drops 27% if it is excluded. In terms of sample ef- ficiency, we find little difference in the model-free experiment Cassie, (Fig. 4). While in the case of robotic manipulation, we observe that the sample-efficiency of RME-TDMPCS is either compara- ble (third-person cam) or even better than individual modalities (first-person-cam) for StackCube. We note that having all modalities in the case of StackCube not as sample-efficient as RME, poten- tially due to state over-specification problem [55]. These increases in wall-clock time and reductions in
are a trade-off off for significantly increased robustness to missing modalities.
I Limitations: A key assumption of our work is that the agent knows which sensors have failed or are corrupted, which in itself is challenging problem [56] that is beyond the scope of this work. Another limitation of our work is that the proposed dropout mechanism in RME assumes that the agent can correctly predict actions with the information from remaining modalities after dropout during training- which may not always be true and could cause difficulties in policy learning.
Conclusion
In this work, we investigate the effectiveness of RL agents in the presence of sensor failure during robot manipulation and locomotion tasks. Our findings indicate that RI. policies trained with standard multimodal encoder do not generalize well to various sensor failure scenarios. To address this issue, we propose RME, a multimodal encoder design incorporating additional information about missing modalities as input and a training strategy based on modality dropout. Our encoder design is compatible with both low- and high-dimensional input modalities. Experimental results, conducted in both simulation and on a real-world bipedal robot Cassie, demonstrate that RL policies trained with the proposed encoder and training strategy maintain effectiveness even under severe sensor failure scenarios, significantly outperforming the baseline RL policies.
8

---

Acknowledgments
The authors would like thank the anonymous reviewers for their insightful feedback that helped improve the quality of the paper. SP would like to thank Mohitvishnu Gadde and Pranay Dugar with their help conducting real world experiments on Cassie and DMV & ViRL labmates for their feedback on the draft. This work is supported in part by NSF Award No. 2321851 and DARPA TIAMAT Agreement No. HR001 100011-244 1224-990423. The views and conclusions contained herein are those the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Govermment, or any sponsor.
References
[1] T. Miki, J. Lee, L.. Wellhausen, and M. Hutter. Learning walk in confined spaces using 3d representation. arXiv preprint wxxvx:403.0018/7 2024.
[2] N. M. S. Laboratory. Curiosity rover: Mars science laboratory. https://d2pnkkiw22221 cl Zloudfront. www.duoott.eee/eooooooo mat.://boccumettssrrr--ciiiaaaie wttp://ttttestrttiieeneeeeeeeeee laboorator pdf,, 2024. Accessed: 2024- 05-22.
[3] J. Campo and M. Morari. Robust model predictive control. American control conference, 1987.
[4] A. Bemporad and M. Morari. Robust model predictive control: A survey. In Robustness in identification and control. 2007.
[5] D. Q. Mayne, M. M. Seron, and S. V. Raković. Robust model predictive control constrained linear systems with bounded disturbances. Automatica, 2005.
[6] K. Chatzilygeroudis, Vassiliades, and J.-B. Mouret. Reset-free trial-and-error learning for robot damage recovery. Robotics and Autonomous Systems, 2018.
[7] M. Allard, S. C. Smith, Chatzilygeroudis, B. Lim, and A. Cully, Online damage recovery for physical robots with hierarchical quality-diversity ACM Transactions on Evolutionary Learning, 2023.
[8] R. Kaushik, T. Anne, and J.-B. Mouret. Fast online adaptation in robotics through meta- learning embeddings of simulated priors. In 2020 IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS). IEEE, 2020.
[9] X. Li, H. attribute sensor data streams via message propagation. Proceedings of the VLDB Endowment, 2023.
H. Lu, C. S. Jensen, V. Pandey, and V. Markl. Missing value imputation for multi-
[10] M. Le Morvan, Josse, E. Scoree,, and G. Varoquaux. What's sa good imputation to predict with missing values? Advances in Neural Information Processing Systems, 2021.
[11] G. Lambrechts, A. Bolland, and D. Ernst. Informed pomdp: Leveraging additional information model-based rl. arXiv preprint arXiv:2306. /488, 2023.
[12] B. van Marum, A. Shrestha, H. Duan, Dugar, Dao, and Fern. Revisiting reward design and evaluation for robust humanoid standing and walking. arXiv, 2024.
[13] Y. Seo, D. Hafner, H. Liu, F. Liu, S. James, K. Lee, and Abbeel. Masked world models for visual control. In Conference Robot Learning, 2022.
[14] Y. Seo, J. Kim, S. James, K. Lee, Shin, and P. Abbeel. Multi-view masked world models for visual robotic manipulation. ArXiv, abs/2302.024088, 2023.

---

[15] K. Mohan, F. Thoemmes, and J. Pearl. Estimation with incomplete data: The linear case. In Proceedings of the International Joint Conferences on Artificial Intelligence Organization, 2018.
[16] K. Mohan and J. Pearl. On the Testability of Models with Missing Data. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, 2014.
[17] K. Mohan and Pearl. Graphical models for processing missing data. Journal of the American Statistical Association, 2021.
[18] B. Muzellec, J. Josse, C. Boyer, and M. Cuturi. Missing data imputation using optimal trans- port. In International Conference on Muchine Learning. PMLR, 2020.
[19] P.-. Mattei and J. Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International conference on machine learning. PMLR, 2019.
[20] S.S. S. Hu, Springer, O. Rybkin, and D. Jayaraman. Privileged sensing scaffolds reinforcement learning. In The Twelfth International Conference Learning Representations, 2024.
[21] J.-F. Tremblay, T. Manderson, A. Noca, G. Dudck, and D. Meger. Multimodal dynamics mod- eling for off-road autonomous vehicles. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[22] M. A. Lee, M. Tan, Y. Zhu, and J. Bohg. Detect, reject, correct: Crossmodal compensation of corrupted sensors. In 2021 IEEE international conference on robotics and automation (ICRA), 2021.
[23] G.-H. Liu, A. Siravuru, S. Prabhakar, M. Veloso, and G. Kantor. Learning end-to-end multi- modal sensor policies for autonomous navigation. In Conference on Robot Learning, 2017.
[24] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also from their hands. ICLR, 2022.
[25] R. Yang, M. Zhang, N. Hansen, Xu, and X. Wang. Learning vision-guided quadrupedal lo- comotion end-to-end with cross-modal transformers. In International Conference on Learning Representations, 2022.
[26] V. Dave, Lygerakis, and E. Rueckert. Multimodal visual-tactile representation learning through self-supervised contrastive pre-training. arXiv preprint arXiv:2401.12024 2024.
[27] J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-Il: Learning multimodal manipulation policies with deep reinforcement learning. In 2022 International Conference on Robotics and Automation (ICRA), 2022.
[28] I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and J. Malik. Robot learning with sensorimotor pre-training. 7th Annual Conference Robot Learning, 2023.
[29] R. Shah, R. Martin-Martin, and Zhu. Mutex: Learning unified policies from multimodal
task specifications. In 7th Annual Conference on Robot Learning, 2023. URL https:// openreview wwwp:/rrneeeeeeeeeeeeeeeee htt.rrrummmiddppiaaaa.......
[30] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, 2020.
[31] A. R. Cassandra, L. P. Kaelbling, and M. L. Littman. Acting optimally in partially observable stochastic domains. In AAAI Conference on Artificial Intelligence, 1994.
[32] N. Hansen, H. Su, and X. Wang. Td-mpe2: Scalable, robust world models for continuous control. International Conference Learning Representations (ICLR), 2024.
10

---

[33] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347 2017. URL https: http///ppp.aaattcccecc/ semanticschollat . org/
:
[34] D. Ha and J. Schmidhuber. World models. arXiv preprint 1803 10/22, 2018.
[35] D. Hafner, T. Lillicrap. M. Norouzi, and J. Ba. Mastering atari with discrete world models. Internation Conference Learning Representations. 2021.
[36] D. Hafner, Pasukonis, J. Ba, and Lillicrap. Mastering diverse domains through world models. arXiv preprint MXiv:2301.04100 2023.
[37] R. Sutton. Learning to predict by the methods temporal differences. Machine learning, 1988.
[38] E. Imani and M. White. Improving regression performance with distributional losses. In International Conference on Machine Learning. 2018
[39] T. P. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (10000099771 2015.
[40] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18120508,, 2018.
[41] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 2017.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 2017.
[43] Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for vision-and-lange and-uagg tasks. Advances in neural information processing systems, 2019.
[44] Huang, Z. Zeng, B. Liu, Fu, and J. Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849) 2020.
[45] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong. Vatt: Trans- formers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 2021.
[46] S. Chen, P.-L. Guhur, Schmid, and Laptev. History aware multimodal transformer for vision-and-language navigation. NeurlPS, 2021.
[47] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional 2018.
transformers for language understanding. arXiv preprint
[48] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 2021.
[49] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
[50] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y. Tassa. dm imm_ontrol: Software and tasks for continuous control. Software Impacts, 2020.
[51] J. Siekmann, S. Valluri, J. Dao, L. Bermillo, H. Duan, A. Fern, and J. Hurst. Learning memory- based control for human-scall bipedal locomotion. arXiv preprint arXiv:2006.02400 2020.
11

---

[52] J. J., R. Kiros, and G. E. Hinton. Layer normalization. ArXiv, abs/1607.064500 2016. URL https://api.... ttttp:://ap..mmmatiic.o semanti semanticscholar
cscholar. wwaccholar.or/Corppuuie org/CorpuusD:8823311, 8236317.
[53] D. Misra, Mish: A self regularized non-monotonic neural activation function. arXiv preprint
2019.
[54] J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang. S. Tao, X. Wei, Y. Yao, X. Yuan,
P. Xie, Z. Huang, R. Chen, and H. Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023.
[55] M. Tomar, A. Zhang, R. Calandra, M. Taylor, and J. Pineau. Model-invariant state abstrac- tions for model- reinforcement learning. arXiv preprint
2021.
[56] D. Li, Y. Wang, J. Wang, C. Wang, and Y. Duan. Recent advances in sensor fault diagnosis: A review. Sensors and Actuators A: Physical, 2020.
[57] H. Ichiwara, H. Ito, K. Yamamoto, H. Mori, and Ogata. Modality attention for prediction- based robot motion generation: Improving interpretability and robustness of using multi- modality. IEEE Robotics and Automation Letters, 2023.
[58] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint ar0iv:2405. 12213, 2024.
[59] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.08864, 2023.
[60] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 1991.
[61] T. Walsh, S. Goschin, and M. Littman. Integrating sasple-based planning and model-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2010.
[62] Pineau, G. Gordon, S. Thrun, et al. Point-based value iteration: An anytime algorithm for pomdps. In Ijcai, 2003.
[63] J. Oh, S. Singh, and H. Lee. Value prediction network. In NeurlPS, 2017. URL https: sapiissmanticacholla wrg/ /Corporrsss ang/Corrpu.ID:2645505 26464771.
[64] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, 2019.
[65] R. Veerapaneni, J. D. Co-Reyes, M. Chang, M. Janner, C. Finn, J. Wu, J. Tenenbaum, and
S. Levine. Entity abstraction in visual model-based reinforcement learning. In Conference Robot Learning, 2020.
[66] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[67] J. Yamada, M. Rigter, J. Collins, and I. Posner. Twist: Teacher-student world model distillation for efficient sim-to-reat transfer. International Conference on Robotics and Automation, 2023.
[68] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 173: 1780, 1997.
[69] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690 2014.
12

---

[70] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 (1):1929-1958, 2014.
[71] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 648-656, 2015.
[72] G. Ghiasi, T.-.. Lin, and Q V. Le. Dropblock: A regularization method for convolutional networks. Advances in neural information processing systems, 31, 2018.
13

---

Appendix
8.1 Additional Related Works
Handlind Missing modalities. Another work [57] trains hierarchical RNNs, where the low level RNN handles individual modalities and the higher level RNN handles the fused representation from the individual modality representations. Although their approach, in theory, can work with more than 2 modalities, they restrict all their experiments to two modalities, and non-trivial whether the extension to multiple modalities would work well with a large number of sensors because multiple modality representations are fused via concatenation an RNN, similarly as in an MLP network used in [23].
Large Multimodal policies: Octo [58] and RT-X [59] models train a large multi-modal policies that are capable of handling multiple robotic embodiments, action spaces. These large transformer based models are trained using the causal masking. However, all such transformer based models are limited imitation learning where access to high fidelity video demonstrations are assumed.
Model based RL (MBRL). A sample-efficient way train learnable control policies is via model- based RL, which learns the model of the environment [60] and uses it for policy learning [61, 62]. In the case of high-dimensions vironment dynamics in a compact latent space that is supervised using rewards [63, 32] and image reconstruction [64, 65, 34, 66]. Recently, Scaffolder [20] proposed learning two dynamics mod- els one for privileged inputs and another for the target modalities and distilling representations from the former. Similarly, TWIST [67] also proposed distilling the teacher world model to the student world model as means for efficient Sim2Real transfer. For multiple modalities, both these methods require the number of dynamics models to increase linearly with the number of modali- ties, in contrast to our work, which aims to learn a compact abstraction from various modalities and develop a single dynamics model. More importantly, unlike our method, these models would not generalize to situations where some the input modalities fail.
modalities such as images, popular approach is to learn the en-
8.2 Details of robotic environments
consider 5 tasks (four from DM Control suite [50] and one task on Cassie-walking [51]) for low dimensional proprioceptive sensory failure, 5 tasks for high-dimensional modality failures from the ManiSkil12 [54] suite as shown in Figure 5. The dimensionality of the observation space and action space is mentioned in Table 2.
Bipedal Locomation
WaBkerun
Waller
Robotic
Maniulltion
tasks
Pick Dute
Stack Cube
Pick Door
Open Cabencc
Figure 5: Tasks We consider 5 bi-pedal locomotion tasks (top) and 5 robotic manipulation tasks (bottom). We test the low-dimensional sensory failures (e.g., the robot's proprioception) in the locomotion task suite and the high-dimensional sensory failures (e.g., cameras) in the manipulation task suite.
8.3
Details on Cassie bipedal walking experiment
Setting: We consider sensory failures in 4 joint links of Cassie, a bipedal robot. Specifically, we simulate the failure in shin and tarsus links both the legs. We train the policy following the reward function design in [12] and train the model in stages. The first stage considers dynamics and environment state variable randomization. and the second stage involves learning robust policy
14

---

Task
Observation dim
Observation type
Action dim
Walker-Warl
24
Proprioception
Walker-Run
24
Proprioception
Talker-WalkBarn
24
Proprioception
Walker-RunBack
24
Proprioception
Cassie-Walk
Proprioception
10
Pick Cube
(64 x 64 x 3)
RGB Depth (TP, FP)
Pick YCB
(64 64 x x
RGB & Depth (TP, FP)
Stack Cube
(64 x 64 x 3)
RGB & Depth (TP, FP)
Open Door
(128 128 3)
RGB & Depth (TP, FP)
11
Open Drawer
(128 x 128 3)
RGB & Depth (TP, FP)
Table 2: State and Action spaces of each task in the locomotion and manipulation suite.
(2 malmme))))
Figure 6: Measuring Commandability of locomotion policy on Cassie (left) commadability in the x-direction and (right) angular drift
under random external forces (of 30N) with probability of 0.1 acting on Cassie. We follow this training procedure from [12]. Each of the stages is trained for 500M steps each.
Metrics: It is important to measure a set of quantifiable metrics to measure the progress in real-world robot deployments. For this we adopt
metrics from [12] that measure the commandability of the robot in x-direction and angular drift. The evaluation setup shown in Figure 6.
Commandability in x-direction. We consider a distance of 8 feet (or 2.43 meters) shown in Figure 6 and command Cassie to travel at a speed of 0.25 m/s. For a near-optimal policy, it takes about 9.72 seconds. We then measure how long it takes for an RME based policy to travel the distance and report it in Table 1.
Commandability to measure angular drift. We consider a circular region with a diameter of 50cm and initialize Cassie at the center of the circle (center of Cassie defined as the mid-point of line joining the feet of Cassie shown by the in Figure 6). We command angular speed of 15 degree/s and measure the time taken to complete single rotation (360%). We consider the trial successful if the center of Cassie is withing 10 cm from the circumference of the circle. For an optimal policy it takes 24 seconds and we compare this against RME based policy with sensory failures to see how much additional time takes for our policy (Table 1).
8.4 Architecture Details
RME multimodal transformer: For the base TD-MPC2 model (5M parameters), the multimodal transformer consists of 2 encoder blocks each consisting of multi-head attention (MHA) and feedforward network [42]. We consider 16 heads for MHA module. The input to the transformer is the encodings from each modality and the [CLS] token which are all of 512 dimensions each. We also generate fixed 512 dimensional modality type embeddings and add them to the corresponding modality encodings. The hidden dimension of the feedforward network in each encoder block of transformer is 256.
15

---

The output of the RME is the output of the attention layer on the [CLS] token attending to the other input tokens of the transformer, which produces a 512-dimensional latent vector. This is the input to the policy. The policy network for TD-MPC2 is implemented as a 3-layer MLP network with SiLU activation, and for model-free PPO training of Cassie it is implemented as a 2-layer LSTM[68] network.
Imputation Network: For the locomotion tasks, we consider 3 layer MLP with LayerNorm [52] and Mish activation function [53] with the final output layer as linear layer. We train one imputation network for predicting each proprioceptive sensor value for 200 epochs with a learning rate chosen from (le-5, 5e-5, 1c-4]. The network parameters are optimized by Adam [69] and the transitions from the replay buffer of RME are used to train the network with a 80-10-10 10 (train-val-tese split.
Decoder architecture for Informed TD-MPC2 baseline. We use the decoder structure from [36]. We show the architecture in Table 3.
Layer
Channels
Kernel Size
Norm/AActivation
Conv ConvTranspose2d
512
2
LayerNorm/SiLLU0
Conv ConyTrranpoos2d
256
2
LayerNorm/SiLLU0
ConyTranspose2d
128
4
LayerNorm/SiLUU
Conv TonyTranspose2d
32
4
LayerNorm/SiLUO
ConvTranspose2d
2
LayerNorm/SiLUO
SigmoidO)
Table 3: Decoder Architecture for Informer-TDMPC. baseline.
8.5 Additional results for sensor failures in high-dimensional modalities (Easy)
In this section, we present the results for sensory failures in high-dimensional modalities where only one of the modalities is dropped during evaluation. Specifically, for all tasks either the RGB or the depth modalities from the first-person (FP) or the third-person (TP) cameras are dropped out. Since only one of the modality is dropped, we denote this setting by Easy as compared to only operating on a single modality (Hard) as shown in Table ?? of the main paper.
Base Policy
Informed
RME Modality Dropout)
ANE (Ours)
Task
Sease Palooo
TD MPC2
max
min
mean
max
min
mean
max
min
mean
0.79
0.69
0.72 1006
0.30 10.10
0.10 1oot (6-19)
0.17 10:1
0.91 100s 01-99
0.80
0.86 10:6
PickCahe
1.0 an
(0-99)
(2-P9)
(9-99)
(5-F9)
0.65 =00s
0.60 $0.19
0.62 +0.0
0.40 20.11
0.07 4001
0.18 +0.14
0.75 #0,0
0.81 #006
0.79 #0004
PickYCB
0.83 20.30
0.74 20.13 60 201
0,51 #0.1
0.72 #0.10
0.31 +0.15 (0 m91
0.02 40.03 021
0.15 +0.14
0.89 20009 10 m7)
0.71 +0,20 (0 TD1
0.82 #007
StackCube
0.96 40.0
< 271
0.73 2011
0.69 so.ss
0.70 #0.00
0.27 200s
0.09 4001
0.19 =0:00
0.80 #0.11
0.77 1008
0.79 =001
OpenDrawer
0.81 60.20
0.66 #0.10
0.61
#005
0.65
#0.0
0.40 2000
0.11 +0.17
0.28 #0:33
0.81 +0.23
0.73 +0.05
0.77 #004
OpenDoor
0.76
175 m
6c
291
LC T91
10 m99
6C TD1
Table 4: Robustness towards modality dropout during deployment (Easy). We test MMWM against others by removing one of (RGB/Depth images]. We report the mean, min, max success rates averaged over 3 seeds. The sensor is of the format (RGB Color (CYDeptth (D) Third Person (TP)/irst Person (FP)) as marked below the numbers.
Results: We show the results for the Easy scenario in Table 4. Not surprisingly, we observe that removing only one modality during evaluation leads to better performance across all basclines. How- ever, there still remains a considerable gap between Informed TD-MPC2 and RME. Additionally, for the case of RME w/o Modality Dropout, we observe that even removal of a single modality can have huge impacts with a relative 357 % drop in performance compared to RME. For the Easy scenario, the RME has a relative increase of 18.1 % compared to Informed TD-MPC2 baseline.
16

---

Ablation with Transformer for low dimensional proprioceptive inputs
1000
750
500
250
0.0
0.5
1.0
1.5
2.0
le5
RME
16 Heads
8 Heads
4 Heads
Figure 7: Ablation study instability of using low-dimensional sensors direct input to trans- former
8.6 Wall-clock time
In Table 5, present the wall-clock running time single NVIDIA Quadro RTX 8000 GPU. The increase in relative wall-clock time because of addition of RME is 27% when we don't include the Walker tasks. Adding the walker tasks skews the results to a relative 72% increase in the wall- clock time. However, the Walker tasks take the least amount time to learn (3 hours), so the impact of such an increase is not significant in practice.
Task
TD-MPC2
Walker
1
3.1
Cassie
70
70.1
PickCuhe
13
20
StackCube
25
32
Table 5: Wall-clock training time in (hrs).
8.7 Ablation: Stability of Transformer as opposed to Masking MLP for low-dimensional proprioceptive inputs
As mentioned in Section 4, we observed that, to accommodate low-dimensional sensors such as proprioception, treating each low-dim sensor as an independent input after encoding via MLP led to instability in training. We show one such result Table on Walker Walk, where only seed for a transformer with 16 heads in the multi-head attention led to a successful policy. Remaining 4 seeds well as all seeds for 8 and 4-head transformer layer led no successful training. TThis could be potentially due to the difficulty of converting small-dimensional vector, such the one in proprioceptive input, into meaningful 512 dimensional modality encoding vector via MLP. This also can add redundant dimensions in the latent space, slowing down the transformer's training
8.8 Compounding errors in imputation
We additionally analyze how the error of the imputation network behaves function of time steps. Specifically, for the imputation baseline in Cassie simulation, we plot the mean-squared error between the predicted missing sensor values and the ground truth value. The imputation network is trained to predict the current timestep's value based on a history of the previous 10 timesteps.
Observation: We observe that the initial 5-10 timestep predictions of the imputation network are accurate. However, the prediction quickly worsens around t=15-20. Visually, this corresponds to Cassie finding hard balance its legs on the ground and after this point, tries maintain its balance before falling onto the ground in about 250 timesteps. We hypothesize that states that correspond to this peak in the plot are potentially underexplored in the replay buffer on which the
17

---

imputation network was trained - hence causing compounding effect in the imputation prediction values, and thus causing the rohot to fail.
imputation Error
20
10
0
0.0
0.2
0.4
1.0
1e2
Left Tarsus
Shin
Right Shin
Figure 8: Imputation error as a function of time steps (on horizontal axis).
8.9 Network dropout
We performed three additional experiments to evaluate the robustness of network dropout towards missing modalities. For this, we experiment with Dropout [70] for low-dimensional modalities and for high dimensional modalities we perform SpatialDropout [71] as well as DropBlock[72], During inference, for all the experiments, dropout is turned off. We show the results for these experiments in Figure 9.
Network dropout for low-dimensional sensors: We perform experiments with dropout rates of 0.2 and 0.5. With a dropout rate of 0.2, we find that the robustness of the policy to missing sensors did increase compared to the base policy with zero-imputation (Figure ??), however with a rate of 0.5, we found that the models struggled to learn good policy. Hence, we observe a significant drop in the performance as shown Figure 9 (left). We hypothesize that 0.5 is too high of a dropout rate for the encoder to obtain meaningful representation to the policy.
Network dropout for high-dimensional modalities: We perform
kinds of network dropout for the CNN layers Spatial [71] which drops entire features by zeroing out them and DropBlock [72] that drops a contiguous block of features. While in general, we find that BlockDrop performs slightly better than SpatialDropoutt performance of both is still lags far behing RME as shown in Figure (right).
8.10 Including imputation baselines during training
In Section 5.1 we presented results for imputation where the imputation model trained post- policy training using the replay buffer collected by the RL algorithm during training. In this section, we present two additional results on Walker environment where the (a) zeros-imputation and (b) leamed-imputation models are part of the policy training loop.
Specifically, for (a), we perform zero-imputation and feed the representation obtained by the encoder the policy and for (b), we trained the imputation network simultaneously and used the predictions of the imputation network to feed to the policy. We show the results of this experiment in Figure 10. We observe that while compared to post hoc accommodation.. imputation models trained in the RL policy training improves the robustness of the policy to sensor failures, it still lags significantly behind an RME type architecture.
One important thing to note is that the maximum return for all the baselines, especially for Walk and Walk backwards on par even better than RME and we consistently notice that this is because of the Right Foot sensor which when dropped in testing leads to decent policy -making infer that the right foot sensor for the simulated Walker task is potentially redundant.
18

---

Desoue
Spatil Drepet 12
Bock Drgeest
Alfelicn mi
rntwork dlogunni
Wellan Looks
Allation siudy agrini DryBBick Spttie
Menipduuiao lonks
1
CTT
C.FP
64
6--11
cm
03:1P
280
1-11
1 PM
1230
StusCClle Thos
OperDrenc
Figure 9: Comparison against network dropout: We perform Dropou[[7]] for encoder networks in Walker tasks and SpatialDropout [71] and DropBlock[72] for manipulation tasks and observe con- sistently that all these network dropout strategies lead to sub-optimal performance when compared to RME. indicates the mean and indicates the max/min values.
ttaring
Larred Imputation (reeap
Ablation auaist larsal Irpucatior bandfres Wilker taoka
Dripeal Living MLP
reys Moculiy Urook!)
1993
(Herd) Companser
36
0.04
PP
TP
114
PTT
0.2
P.TP
Walke
Stadl Cabe
Figure 10: Imputation during truining: We perform both ceros-imputationn and learned imputation part of the RL policy training and observe that while the performance of including them training loop is better than imputation post policy training, the impu- tation methods still lag behind in their performance compared to RME. indicates the and indi- cates the max/min values.
Figure 11: Ablation of RME without transformer We observe an average improvement of 93% across three robotic manipulation tasks of RME when compared to policy that is trained without transformer encoder. indicates the mean and indicates the max/min values.
8.11 Ablation without RME with sensor dropout
test the effectiveness of the proposed RME' transformer structure. Instead of using transformer done in RME, used MLP network with modality feature masking to indicate missing modalities for robotic manipulation experiments in Figure 11. As expected, this is clearly better than training RME without any modality dropout during training; however, we find that this is still less performant compared to the proposed RME encoder+dropout (with an average increase 93% success rate), indicating the benefit of using transformer based architecture.
Another experiment performed
encoder, similar to [23]. We present our results
8.12 Effect of history and memory in imputation baseline
In our imputation baselines, our imputation network takes in past 10 timesteps the state of the robot and predicts the next state. The imputation network was trained by randomly sampling 10 consecutive states from the replay buffer and using that predict the next (11 state. During the inference, have access to the history of ground truth state information that get from the environment (except for the masked sensor) which is the input to the imputation network. We
19

---

imputation Error
Zored lepateiss
Ingutation (fhtory 30g
Iepuiston Halery- 59)
Irpxzation (LS IM]
RMF
Adleiic agiins attwork deopoce Walker troks
Woliig
20
I
0.0
0.2
0.4
0.6
0.8
10
162
Lottirruu (History 10)
100
(histeryn30)
200
LeftTersus (LSTM)
Figure 13: Imputation error as function of time steps (on horizon- tal axis). We find that the LSTM archi- tecture for imputation network does incur a high error slightly later than an MLP with 10/50 timesteps history.
106
Dax
10 1Pas DarDas
Case Reiieg
Figure 12: Effect of history length and memory for imputation model: We compare RME against different models of imputation network with different lengths of history (10, well as LSTM network. We do not observe significant diffcrences between different Icarned imputation baselines, al- beit the average performance of the LSTM network is slightly better. indicates the mean and indicates the max/min values.
further experiment with (a) higher history length of 50 and (b) an LSTM [68] architecture this extended history leads better robustness under sensor failures. As we showed in Section 8.10 that including the imputation in the RL policy loop performs better than post hoc imputation, perform these experiments under that configuration.
As shown in Figure 12, we find no significant difference between different levels of history for imputation networks. The LSTT architecture the imputation model performs slightly better on average terms of returns, however its overall performance is still behind RME. As mentioned in Section 5.1, hypothesize that this may be because the errors in learned imputation may have compounding effects if predicted sensor values drive the policy to out-of-distribution states where the imputation network is less well trained.
Further, we also extend the imputation error analysis on the new imputation models (History=50 steps and LSTM) as shown in Figure 13. We find that the LSTM model does incur high error albeit slightly later timestep than the MLP model trained on history of 10 or 50 steps history.
20


Reviews:
[{"review_id": "uFJrk3PL7Q", "reviewer": "Reviewer_iBv4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Provide a masking strategy to mitigate sensor failures affecting robotic control.", "summary_of_recommendation": "Well written paper studying the affect of masking on sensor failures. However, have some concerns about novelty given prior work.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How do other alternate masking approaches perform (e.g adding dropout in the decoder/encoder/both, blockwise) compare to modality dropout? Are these approaches also able to recover the performance?\n2. Are there sensor failures in the training dataset or are these filtered out? If they are filtered, wouldn't this prevent modality dropout from naturally arising?\n3. Does a pre-trained encoder ameliorate some of this issues if for example they were trained with some masking (e.g MAE/VIT with patch dropout)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqhxVjKuBM", "reviewer": "Reviewer_X6ey", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes two strategies for improving the robustness of learned robot control policies to sensor failures. The first is a training scheme that includes random sensor dropout, and the second is a multi-modal encoder architecture that is robust to sensor dropout. The paper evaluates these components on multiple tasks in simulation and hardware.", "summary_of_recommendation": "My concerns about the evaluation and novelty of this paper's contributions are substantial enough that I recommend rejection, but I would be open to upgrading if my concerns were addressed. [UPDATE] The authors have addressed my concerns and I am in favor of accepting the revised paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- For the learned imputation baseline: is the policy trained with the learned imputation mechanism present? And if so, is the learned imputation network trained or fine-tuned simultaneously with the policy network?\n- Similarly for the zero imputation baseline, is the control policy trained with the zero-imputation mechanism and sensor failures?\n- Would Table 1 be better presented as a plot? e.g. a scatterplot or boxplot to show the distribution of returns. Similar for Table 2.\n- Why do the results for Walker skew the wall-clock time increase?\n- Fig 7 in the supplementary materials: It's not surprising that the imputation network fails after 10 timesteps if it only has 10 timesteps of memory. Would you see the same behavior from an imputation network that has memory or estimates some sort of latent state? Fig. 7 makes me think that the baseline you have chosen for the imputation network is not particularly strong. Did you recreate any of the sensor imputation baselines mentioned in the related work section?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ur12EYtyKF", "reviewer": "Reviewer_SZc2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method to learn robot policies that are robust against sensor failure. During training sensor failure is simulated using modality dropout and a transformer-based module called Robust Multimodal Encoder (RME) learns to attend to different input modalities to robustly create a latent representation. The method is evaluated in locomotion and evaluation tasks and the results show that the method is more robust against missing sensor modalities than comparable baselines.", "summary_of_recommendation": "It's a great paper that proposes a promising method for generating behavior robust to sensor failure.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The authors need to clarify what the action space is in each experiment and deliver a realistic explanation of why the actions are still executable with failed sensors. If this is not possible, then the method needs to tune down its claim* or adapt the method such that it can cope with impeded action capabilities. (*the contribution is more theoretical than practical in this case. the method does not actually yield behavior robust against corrupted sensors, but against cases where sensor information is available to low-level control but not somehow does not arrive at the policy level) \n\n- The related work should refer to the two works I mentioned above, but ideally also search for other works of related similarity level.\n\n- The paper should cleary and explicitly provide information how the RME is part of the policy and what the full policy looks like.\n\n- Clarify what is sg() in equation 2.\n\n- 129 - typo - Intergral\n\n- 164 - typo - multi=\n\n- 179 grammar? concatenated to the vector of observations", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers and area chair have identified the following strength and weaknesses with the initial submission:

Strengths:
- the problem the paper addresses is very relevant
- the paper is well-written and easy to follow 
- the paper provides useful experiments in simulation and the real world on both locomotion and manipulation 

Weaknesses:
- Novelty: the idea of drop-out has been explored in prior work and the authors should revisit their novelty claims and related work section
- Scope & assumptions: the authors only consider complete sensor breakdown’s while more subtle sensor failures are possible; the assumption that actions are not affected by sensor failure seems strong. 
- Baselines: The authors should revisit their baselines to ensure that the comparisons to the proposed approach are fair and support the claims

Post-Rebuttal, the authors have made a substantial effort to clarify their contribution and added multiple baselines as required by the reviewers. The results are compelling leading to an accept.
Decision: Accept