Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation
Authors: Ruoxuan Feng
Di Hu
Wenke Ma
Xuelong Li
Venue: CoRL 2024 (2024)
Keywords: Multi-Sensory
Robotic Manipulation
Multi-Stage
TL;DR: We propose MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine-grained state within the predicted current stage.

Abstract: Humans possess a remarkable talent for flexibly alternating to different senses when interacting with the environment. Picture a chef skillfully gauging the timing of ingredient additions and controlling the heat according to the colors, sounds, and aromas, seamlessly navigating through every stage of the complex cooking process. This ability is founded upon a thorough comprehension of task stages, as achieving the sub-goal within each stage can necessitate the utilization of different senses. In order to endow robots with similar ability, we incorporate the task stages divided by sub-goals into the imitation learning process to accordingly guide dynamic multi-sensory fusion. We propose MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine-grained state within the predicted current stage. We train a robot system equipped with visual, auditory, and tactile sensors to accomplish challenging robotic manipulation tasks: pouring and peg insertion with keyway. Experimental results indicate that our approach enables more effective and explainable dynamic fusion, aligning more closely with the human fusion process than existing methods.

Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation
Ruoxuan Feng'
Di Hu Wenke Ma²
Xuelong Li¹
'Gaoling School of Artificial Intelligence. Renmin University of China
Shenzhen Taobotics Co., Ltd.
Institute of Artificial Intelligence (TeleAD, China Telecom com,
{ (fengruoxuan, dihu @ruc. edu. cn, wenkeStaobotice cs. com, xuelong.11@ieee..oo
Abstract: Humans possess remarkable talent for flexibly alternating different senses when interacting with the environment. Picture chef skillfully gauging the timing of ingredient additions and controlling the heat according the colors, sounds, and aromas, seamlessly navigating through every stage of the complex cooking process. This ability is founded upon a thorough comprehension of task stages, as achieving the sub-goal within each stage can necessitate the utilization of different senses. In order to endow robots with similar ability, we incorporate the task stages divided by sub-goals into the imitation learning process to accord- ingly guide dynamic multi-sensory fusion. We propose MS-Bot, stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine- une-grained state within the predicted current stage. We train robot system equipped with visual, auditory, and tactile sensors to accomplish challenging robotic manipula- tion tasks: pouring and peg insertion with keyway. Experimental results indicate that our approach enables more effective and explainable dynamic fusion, align- ing more closely with the human fusion process than existing methods. The code and demos can be found at https://gewu-lab.giiuht www.iooo/////BBBM Bot/.
Keywords: Multi-Sensory, Robotic Manipulation, Multi-Stage
Introduction
Humans are blessed with the ability flexibly use various sensors to perceive and interact with the world. Over the years, endowing robots with this potent capability has remained dream for humanity. Fortunately, with the advancement of computing devices and sensors [1, 21, building multi-sensory robot system to assist humans has gradually become achievable. We have witnessed the emergence of some exciting multi-sensory robots, such as Optimus [3] and Figure 01 [4].
One challenge in tasks that robots assist humans with is the complex object manipulation task, which requires achieving a series of sub-goals. These sub-goals naturally divide a complex task into multiple stages. Many efforts attempt make models comprehend these task stages, whether through hierarchical learning [5, 6, 7] employing LLMs [8, 9, 10]. However, when incorporated into multi-sensory robots, this challenge becomes more profound. We need to not only understand the stages themselves but also rethink multi-sensory fusion from the fresh perspective of task stages.
Completing sub-goals within a complex task may require different senses, with varying modality importance at each stage. Thus, stage transitions may entail changes in modality importance. To illustrate this, build multi-sensory robot system (Fig. 1, left). We then use imitation learning to train a MULSA [11] model with self-attention fusion complete pouring task and record the test- ing confidence (Fig. 1, right). We observe correlations between the confidence and task stages after
Corresponding author. "Play to the score" is slang for adjusting actions to fit current circumstances.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Multi-Sensory System
Confidence Variation in Pouring Task
Canfilence
Low Contidonee Hastentiend
Cer'i 10
Codl d
Alkotes
Stari hearin
Sugg Hokding SaR
Ind Poarizz
Time
Viska
Vakaa (NGB),
Aiale
Teuch
Figure 1: An illustration for our multi-sensory robot system and a visualization of Modality Temporality in multi-stage task: pouring. We show confidence in action prediction (maximum softmax score) when using the inputs of all modalities and selectively masking uni-modal features. Due to the changing importance of modalities, both evident inter-stage (coarse-grained) and minor intra-stage (fine-grained) changes in confidence are observed when masking uni-model features. The low confidence fluctuations near stage boundaries also reflect insufficient task stage understanding.
masking one modality. For example, vision is most important the aligning stage, and masking causes a notable confidence drop. We also note minor modality confidence changes inside stages. These indicate that modality importance undergoes both inter-stage changes and nttra-stage adjust- ments. We summarize this as a challenge in multi-sensory imitation learning: Modality Temporality.
The key to addressing this challenge is understanding the current task stage and its concrete internal state, i.e., coarse-to-fine stage comprehension. Multi-sensory features should be dynamically fused to meet the needs of the coarse-grained stage and adjust to the fine-grained internal state. However, most of the existing works that utilize multiple sensors [12, 13, 14, 15] essentially use static fusion like concatenating. They lack the adaptability to dynamic changes of modality importance in com- plex manipulation tasks. Some recent works [16, 11] attempt to attention for dynamic fusion, but they still fail to break out the paradigm of fusion based solely on current observations. They neglect the importance of stage understanding in multi-sensory fusion.
In light of this key insight, we incorporate the stages divided by sub-goals into multi-sensory fusion. To perform coarse-to- to-fine stage understanding, we first encode current observations and historical actions into a state token and identify the current stage based it. We then inject the stage infor- mation into the state token by weighting learnalle stage tokens. For fusion, we propose dynamic Multi-Sensory fusion method for roBot (MS-Bot) to allocate modality weights based on the state within current stage. Using cross-attention [17] with stage-injected state token as query, we dynam- ically select multi-sensory features of interest based on the fine-grained internal state. The dynamic allocation of attention score to feature tokens ensures effective fusion aligned with requirements.
To evaluate our MS-Bot method in the real world, we build robot system that can perceive the environment with audio, visual and tactile sensors. We test it on two challenging manipulation tasks: pouring, where the robot needs to pour tiny beads of specific quality, and peg insertion with keyway, where robot needs to rotate the peg with a key to align the keyway during the insertion. To summarize, our contributions are as follows:
We summarize Modality Temporality as a key challenge and identify the coarse-to-fine stage comprehension a crucial point to handle the changing modality importance in complex tasks.
Based on the above insight, propose dynamic multi-sensory fusion method guided by the prediction of fine-grained state within the identified task stage.
We build multi-sensory robot system and evaluate our method on two challenging robotic ma- nipulation tasks: pouring and peg insertion with keyway.
Experimental results show that MS-Bot better comprehends the fine-grained state in current task stage and dynamically adjusts modality attention needed, outperforming prior methods in both tasks. We hope this work opens up new perspective multi-sensor fusion, inspiring future works.

---

2
Related Work
Multi-Sensory Robot Learning, Recent works have extensively integrated visual, auditory, and tactile sensors into robot systems. The two common visual modalities, RGB and depth, are widely combined and applied in grasping [18, 19, 20, 21, 22], navigation [23, 24), and pose estimation [25, 26, 27] tasks. In recent years, the development of tactile sensors has sparked interest in the study of tactile modality. Fine- Fine-grained tactile data assist in tasks such grasping [28, 29, 12, 13], shape reconstruction [30, 31] and peg insertion [32. 15]. Audio modalities are typically applied in tasks such as pouring [11, 14], object tracking [33, 34], and navigation [35, 36, 37]. Some works even attempt to jointly use visual, audio, and tactile modalities to manipulate objects [38, 39, 11J. Based on these efforts, we further explore effective multi-sensory fusion in complex manipulation tasks.
Robot Learning for Multi-Stage Tasks. Complex multi-stage tasks have always been a focal point of research in robot manipulation. Previous works hierarchically decompose the multi-stage tasks, where upper-level models predict the next sub-goal, and lower-level networks predict specific actions based on the sub-goal. Following this paradigm, many hierarchical reinforcement learn- ing [7, 40, 41, 42, 43, 44] and imitation learning [5, 6, 45, 46, 47) methods have demonstrated their effectiveness across various tasks. [8, 9, 10] use LLMs or VLMs to decompose long- nong-horizon tasks. Recently, [48] introduces Mhain-of-Though into robot control. While these works primarily focus on uni-sensory scenarios, we aim to investigate the impact of stage changes on multi-sensory fusion.
Dynamic Multi-Modal Fusion. Dynamic multi-modal fusion essentially adapts the fusion process based on the input data by assigning different weights to the uni-modal features. A simple and ef- fective way to assign weights is to train a gate network that scores based on the quality of uni-modal features [49, 50, 51, 52]. Another approach to assessing the quality of modalities and dynamically fusing them is uncertainty- -based fusion [53, 54, 55]. With the introduction of the transformer [17], self-attention provides a natural mechanism to connect multi-modal signals [56]. Hence, attention- based fusion methods rapidly become a focal point of research [11, 57, 58, 59]. However, these methods primarily perform dynamic fusion based solely on the inputs themselves, while we design dynamic fusion from the perspective of task stages in robotic scenarios.
3 Method
In this section, we discuss the challenges in multi-sensory imitation learning from the perspective of task stages and propose the dynamic multi-sensory fusion framework MS-Bot, based on coarse- to-fine task stage understanding, to overcome these challenges. Specifically, the fine-grained state refers to a concept that describes the current situation of the environment, while the coarse-grand -gaained stage, divided by sub-goals of the task, is collection composed of many states, follows:
an .1).
(2),
aatt)) ):
X1it)
.0t (0g+1)
(tz))
State
State
State &
State 4++1
State l2
Stage
Stage
where E is an episode, X, is the multi-sensory observation at timestep t and a is the action.
3.1 Challenges in Multi-Sensory Imitation learning
Imitation learning is data-efficient robot learning algorithm where robot learns task by mim- icking the behavior demonstrated by an expert. However, directly applying it to complex tasks with multi-sensory data could encounter issues, such as low confidence fluctuations and changing modality importance shown in Fig. 1. We attribute the causes of these issues to two key challenges:
Non-Markovity. Training data for imitation learning is typically derived from human demonstra- tions, usually collected through teleoperation devices. However, when humans manipulate robots, their actions are not solely based on data observed by robot sensors, but are also influenced by mem- ory. This becomes particularly significant in more complex multi-stage tasks, as memory can pro- vide cues about the task stages. Identifying the current stage solely based current multi-sensory observations can be much more challenging. This challenge is not brought by the introduction of
3

---

Multi-Sensory
Action History
Observations
(Memory)
Feature Extractor
LSOM
Gale [Sefimaa))
Recadare
MILP
Frature
Taken
Stade Toleetzzrr
Tokic
Dynamic Fusion
Singe Comprehension
Stags Nomr
Vater
Stage Gaided Dynamic Fision
Crean Vitaadios
MLP
Learaable
Bg (Prediction)
lale
Figure 2:
The pipeline of our method MS-Bot. consists of four parts: feature extractor, state tokenizer, stage comprehension module, and dynamic fusion module.
multiple sensors. Some works have already relaxed the Markov assumption and utilized action his- tory as an additional input or leveraged sequence models to enhance performance [60, 61]. We identify the action history as crucial cue for understanding the current stage and the fine- fine-grained state within the stage, and use it predict stages rather than directly predicting the next action.
Modality Temporality. In a complex manipulation task, the importance of various uni-modal fea- tures could change over stages. At timesteps from different stages, a particular modality may con- tribute significantly to the prediction, or serve as a supplementary role to the primary modality, or provide little useful information. Moreover, different states within a stage, such as the begin- ning and end, may also exhibit minor changes in modality importance. We distinguish them as coarse-grained and fine-grained importance change. However, previous works typically treated each modality equally and did not consider the issue of changing modality importance in method design. This could lead to sub-optimal outcomes as uni-modal features with low quality could disrupt the overall prediction. On the contrary, we utilize task stage information to guide multi-sensory fusion, dynamically assigning weights of modalities based on the fine-grained state in the current stage.
3.2 Stage-Guided Dynamic Multi-Sensory Fusion for Robot (MS-Bot)
In this section, we introduce how our method MS-Bot addresses the above challenges. Considering Non-Markovity, we use action history as input to assist the model in assessing the current task state. For Modality Temporality, we establish a coarse-to-fine fine stage comprehension, decomposing the task state into coarse- coarse-grained stages and fine-graie mne-grained states within a stage. Dynamic fusion guided by the fine-grained state within the current stage employed to handle the changing modality importance.
To introduce the understanding of stages into the process of imitation learning, we incorporate task stages divided by sub-goals into the training samples. Given sample (X,, az) at timestep / in trajectory where X, {X] xx? X. } is the observation of M modalities and ae is the corre- sponding action, a stage label 8, E {1,2, 2.., S) is added, where is the numher of stages the task divided by different sub-goals. The sample at timestep then represented as (X,, ag st). Based on the sample with stage division, we then introduce the four main components of our method MS-Bot:
This component consists of several uni-modal encoders. Each encoder takes brief history of observations XT E R Hm W., XCm of the modality m as input, where T is the timestep number of the brief history and Hm: Wm, Cm indicates the input shape of modality m. These observations are then encoded into feature tokens f, E M xTxxx where d denotes dimension.
Feature Extractor.
State Tokenizer. This component aims to encode the observations and action history (a1, ..,.... a1-1) into token that represent the current state. Action history is similar to human memory and can help to indicate the current state within the whole task. We input the action history

---

as a one-hot sequence into an LSTM [62], then concatenate the output with the feature tokens and encode them into a state token = ztaat through a Multi-layer Perceptron (MIP).
Stage Comprehension Module. Merely using the state token alone insufficient for achieving a comprehensive understanding of task stages. This module aims to perform coarse-to-grained stage understanding by injecting stage information into the state token. For task with S stages, S learnable stage tokens [stagei]. .... [stages] to represent each stage. After warmup training, each stage token is initialized with the mean of all state tokens on samples within the stage. Next, we use a gate network (MIP) to predict the current stage, i.e., coarse-graine boarse-grained stage comprehension. The S-dimensional output scores gr of the gate are poltmax-normalinn
and multiplied by each of the stage at timestep via: stage tokens, followed by summing up the results to obtain the stage token 24
gr -ge m. gtt) softmax 2oftmaz(MLPPC (M
stage 2t
>>
Σ (ot [stage,]).
5 j=1
We use softmax score instead of one-hot encoding because the transition of stages is a continuous and gradual process. Finally, we compute the weighted sum of the state token zt zętate and the current stage token 2, stage using weight / to obtain the stage-injected state token 2;:
zi B state + (1 3) zt
stage
(3)
Different from the old state token zttate the new state token z represents the fine-grained state within stage. In this case, 2t stage is regarded an anchor stage, while 2,tate can indicate the shift inside the stage, thereby achieving coarse-to-fine stage comprehension.
During the training process, we utilize stage labels to supervise the stage scores output by the gate net. Specifically, we penalize scores that do not correspond to the current stage. Additionally, for samples within range near the stage boundaries, we do not penalize its score on the nearest stage, achieving soft constraint effect. The loss for the gate net fgate the i-th sample is follows:
ggate,
E(w gf), j E 21,,,...,S) f=1
(4)
{ O, (s+= or (3k, Ik - S 9k),
= 1, otherwise,
where l: indicates nearby sample in the same trajectory, 8; and 8y represent stage labels and y is hyper-parameter used to determine the range near the stage boundaries. This soft constraint allows for smoother variations in the predicted stage scores.
Dynamic Fusion Module. Due to variations in modality importance across different stages and the minor importance changes within a stage, we aim for the model dynamically select the modalities of interest based on the fine-grained state within the current stage. We use the state token with stage information ot query, and the feature tokens f, as key and value for cross-attention [17]. This mechanism dynamically allocates attention scores to feature tokens based on the state token 2f, thereby achieving dynamic multi-sensory fusion over time. It also contributes to more stable multi-sensory fusion, as the anchor sstag changes minimally within a stage. The features from all modalities are integrated into a fusion token 2.. based on the current stage's 's requirements. Finally, the fusion token 2 ue is fed into an MLP to predict the next action at-
In order to prevent the model from simply memorizing the actions corresponding to attention score patterns, we also introduce random attention blur mechanism. For each input, we replace the atten- tion scores on feature tokens with the same average value 1 with probability p. We encourage the model to simultaneously learn both stage information and feature information in the fusion token.
The loss during training consists of two components: the classification loss for action prediction Lote and the penalty on the scores of the gate network Lyate. The total training loss L is as follows:
L= Lets >Legtte
(5)
where X is hyper-parameter control the intensity of the score penalty.
5

---

7
Cytinder with 91/120g beads iniiilll
Taraet Cylinder
with key kayway
Peg Insertion with Keyway
Pouring
Figure 3: An illustration of the task setup for peg insertion with keyway and pouring task.
Experiments
In this section, we explore the answers the following questions through experiments: (Q1) Com- pared previous methods, how much performance improvement does stage-guided dynamic multi- sensory fusion provide? (Q2) Why is stage-based dynamic fusion superior to fusion based solely on the input itself? How does stage comprehension help? (Q3) Can stage-guided dynamic fusion maintain its advantage over static fusion in out-of-distribution settings?
4.1 Task Setup
Pouring, For the pouring task, the robot needs to pour tiny steel beads of specific quality from the cylinder in the hand into another cylinder. This task consists of four stages [44]: 1) Aligning, where the robot needs to move a graduated cylinder containing beads and align it with the target cylinder, 2) Start Pouring, where the robot rotates the end effector pour the beads an appropriate speed, 3) Holding Still, where the robot maintains its posture to continue pouring out beads steadily, and 4) End Pouring, where the robot rotates the end effector at the appropriate time to stop the flow of beads. The initial mass of the beads is 90g/ 120g, while the target mass for pouring out is 40g/60g, both indicated by prompts. We use audio, vision (RGB) and touch modalities in this task.
Peg Insertion with Keyway. This task is an upgraded version of peg insertion, where the robot needs to align peg with key to the keyway the base by rotating and then insert the peg fully. The alignment primarily relies on tactile feedback, as the camera cannot observe inside the hole. This task consists of three stages: 1) First Insertion, where the robot aligns the peg with the hole and inserts it until the key collides with the base, 2) Rotating, where the robot aligns the key with the keyway on the base by rotating the peg based on tactile feedback, and 3) Second Insertion, where the robot further inserts the peg to the bottom. We use RGB, depth and touch modalities in this task.
Fig. 3 briefly shows the two tasks. See Appendix A for more detailed task setup for both tasks.
4.2 Physical Setup and Experimental Details
Robot Setup and Data Collection. We use 6-DoF UFACTORY xArm 6 robotic arm equipped with a Robotiq 2F-140 gripper in all experiments. For both tasks, we generate Cartesian space displacement commands at a policy frequency of 5 Hz. We use an Intel RealSense D435i camera to record visual data (RGB and depth) with resolution of 640 x 480 at a frequency of 30Hz. We use a desktop microphone to record audio at a sampling rate 4kHz. Tactile data are recorded by GelSight [1] mini sensor with resolution of 400 300 and frequency of 15Hz.
We compare our method with three baselines in both tasks: 1) the concat model which directly concatenates all the uni-modal features [12, 13, 14, 15], 2) Du et al. [63] that uses LSTM to fuse the uni-l features and the additional proprioceptive information, and 3) MULSA [11] which fuses the uni-modal features self-attention. We also compare our method with two variants in each task: MS-Bot (w/o A/D) and MS-Bot (w/o T/R) where one modality in the task removed (audio and touch for pouring while depth and RGB for peg insertion).
Baselines.

---

Pouring- Pouring-Initial Mass (g)
Pouring-Target Mass (g)
Insertion Success Rate
Method
90
120
Concat
4.80 = 1.14
8.72 + 2.39
8.40 +: 2.21
6.54 1 2.14
5/10
et al. [63]
4.32 1.22
7.79 2.11
8.54 2.04
6.26 2.01
5/10
MULSA [11]
3.05 = 1.01
6,42 = 1,98
7.12 + 1.66
4.19 = 1.24
6/10
MS-Bot (w/o A/D)
10.55 + 2.25
15.76 + 3.18
15.64 4 3.25
13.02 + 3.13
5/10
MS-Bot (w/o T/R)
8.32 = 1.74
12.99
+ 3,04
13.02 + 3.13
9.77 + 1.65
5/10
MS-Bot
1.60 = 1.10
5.58
+ 1.79
6.48 + 1.55
1.80 + 0.95
8/10
Table 1: Comparison of performance on pouring (mean = standard deviation) and peg insertion with keyway (Q1). A: Audio, D: Depth, T: Touch, R: RGB.
Artmniin
Pouring
Soore
Aadle x
Valuo
Teack
Time
Stape Score
Stage 1: Aligsing
Stags 2: Start Pauring
Stage 3: Holding Soll
Stage 4: Ead Pauring
Timm
Figure 4: Visualization of the aggregated attention scores for each modality and stage scores in the pouring task (Q2). At each timestep, we average the attention scores on all feature tokens of each modality separately. The stage score is the output of the gate network after softmax normalization.
4.3 How much improvement does stage-guided dynamic multi-sensory fusion provide?
To evaluate how much improvement MS-Bot brings compared to previous methods, we conduct comparative experiments on the two tasks. For each task (considering changing initial weight and target weight as two tasks), we collect 30 human demonstrations and use 20 of them as the training set, leaving the remaining 10 as the validation set. For both tasks, we conduct 10 real-worke tests.
Tab. shows the performance of baselines and our method on two tasks. Our MS-Bot outperforms all baselines in both the pouring task and peg insertion task, demonstrating the superiority of stage- guided dynamic multi-sensory fusion. Despite using proprioceptive information, the improvement of Du et al. [63] over the concat model is minimal, indicating that the inadequate understanding of the coarse-grained and fine-grained modality importance change is the critical barrier. Moreover, these two baselines exhibit delayed responses to stage changes, causing excessive pouring in the pouring task and misalignment in the insertion task. MULSA employs self-attention for dynamic fusion, but its lack of thorough stage understanding prevents from fully leveraging the advantages of dynamic fusion. Our MS-Bot achieves the best performance by better allocating modality weights during fusion based on the understanding of the fine-grained state within the current stage (Q1).
The notable performance drop when one modality is removed indicates that each modality con- tributes significantly. In the pouring task, removing audio and tactile modalities degrades the model's ability assess the quality of poured beads. The impact is more significant when the audio modality is removed, corresponding to the notable decrease in confidence observed after masking the audio features in Stage 3 as shown in Fig. 1. In the peg insertion task, removing either the RGB or depth modality will degrade the alignment performance. is worth noting that even without the assistance of the depth or RGB modality, our MS-Bot achieves an equal success rate compared to concat model and Du et al. [63], demonstrating the superiority of stage comprehension and dynamic fusion (Q1).
4.4 Why is stage-based dynamic fusion superior fusion based solely on the input itself?
To better understand the working principles of MS-Bot and the reasons for performance improve- ment, we visualize the attention scores of each modality and the scores of the stage tokens. As shown
7

---

Pouring Color
Insertion
Method
Overall
Method
Color
Mess
MS-Bot
3,88 + 2.55
Concat
8.75 2.02
2/10
3/10
Attention Blur
4.01 +: 2.54
Du al. [63]
8.04 1.85
3/10
3/10
Stage Comprehend
4.62 + 2.34
MULSA [11]
5.56 = 1.13
4/10
5/10
State Tokenizer
5.19 + 2.21
MS-Bot
2.41
1.47
6/10
6/10
Table 2:
Impact of each component of our Table 3: Comparison of models in scenes with vi-
method in pouring task (Q2). *-' indicates further removing the module from the model in the pre- of the cylinder the base changed and "Mess" vious line. Overall error on all settings is shown. indicates there are some clutter near the base.
sual distractors (Q3). "Color" indicates the color
in Fig. 4, MS-Bot accurately predicts the rapid stage changes. Due the model's coarse-to to-fiee task stage comprehension, the aggregated attention scores of the three modalities remain relatively stable, exhibiting clear inter-stage changes and minor intra-stage adjustment (Q2). Vision initially domi- nates during the first stage. Upon entering the second stage, the model begins to use the sound of beads to find the appropriate angle and distinguishes this stage from the last stage through touch. During the holding still stage, the model primarily relies on audio and tactile deformation to assess the mass of beads. In the final stage, the model discerns the completion of the pouring based on tac- tile deformation and the rise of tactile attention scores is observed. The changes in attention scores for the peg insertion task (see Fig. 12b) also follow similar pattern, demonstrating the validity and explainability of the proposed stage-guided dynamic multi-sensory fusion (Q2).
We also conduct ablation experiments on the pouring task to examine the contributions of each module, as shown in Tab. 2. We observe significant contributions to performance from both the state tokenizer and the stage comprehension module (Q2). These two modules provide critical task state understanding and coarse-to-fing tine stage comprehension. In addition, removing random attention blur results in minor performance decline. The complete ablation results located in Tab. 8.
4.5
Can stage-guided dynamic fusion maintain its advantage in out-of-distribation settings?
To verify the generalization of our method to distractors, we conduct experiments with visual dis- tractors in both tasks. In the pouring task, we changed the cylinder's color from white to red. For the peg insertion task, we altered the base color from black to green ("Color"), and placed clutter around the base ("Mess"). As shown Tab. 3, our method exhibits minimal impact across var- ious scenarios with distractors and consistently maintains performance superiority, demonstrating the generalizability of stage comprehension (Q1, Q3). Two baseline methods, the concat model and Du al. [63], suffer from performance degradation when the visual modality is disturbed. Due to their lack of ability to understand the stages and dynamically adjust modality weights, even when information from visual modality is not needed in particular stage, the model is still affected by distractors. This leads to unsatisfying performance. Our method dynamically allocates modality weights based on the understanding of the current stage, thereby reducing the impact of visual dis- tractors on the fused features (Q2, Q3). Consequently, it outperforms the two baseline methods using static fusion and the MULSA that solely relies on the current observation for dynamic fusion.
Conclusion and Limitations
We present MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage comprehension, which dynamically focuses on relevant modalities based on the state within the current stage. Evaluated on the challenging pouring and peg insertion with keyway tasks, our method outperforms previous static and dynamic fusion methods. We find that stage comprehension can be generalized to scenes with various distractors, reducing the impact of interference from one modality on multi-sensory fusion. We hope our work can inspire future work on multi-sensory robots.
Limitations: Our stage division involves human factors. An improvement could be leveraging LLMs for automatic stage labeling. Additionally, proposing more challenging multi-sensory manip- ulation tasks would be interesting for future works. More limitations discussed in Appendix L.

---

Acknowledgments
The project was supported by National Natural Science Foundation of China (NO.62106272).
References
[1] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017.
[2] Schmidt, Scaife, M. Harville, S. Liman, and A. Ahmed. Intel realsense tracking camera 1265 and intel@ realsense depth camera d435-tracking and depth. Real Sense, 2019.
[3] Tesla. Tesla bot. Website, 2024. attps://wwen tesla. com/AI.
[4] Figure. Figure 01 openai: Speech-to-speech reasoning and end-to-end neural network. Web- site, 2024. https: /ww::://wwwww... www.ffiguue... ai/.
[5] Luo, Xu, X. Geng, G. Feng, Fang, L. Tan, S. Schaal,, and S. Levine. Multi-stage cable routing through hierarchical imitation learning. IEEE Transactions on Robotics, 2024.
[6] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Newral Information Processing Systems, 32, 2019.
[7] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcemmnt learning.
Advances in neural information processing systems, 31, 2018.
[8] Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language. arXiv preprint arXiv:2403.01823, 2024.
[9] W. Huang, F. Xia, T. Xiao, Chan, Liang, Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chehotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Conference
Robot Learning, pages 1769-1782. PMLR, 2023.
[10] Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, Gopalakrishnan, C. Chan, G. Dulac- Robovqa: Multimodal long-horizon reasoning for
Arnold, N. J. Joshi, Florence, robotics. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
[11] H. Li, Y. Zhang, J. Zhu, Wang, M. A. Lee, H. Xu, E. Adelson, Fei-Fei, R. Gao, and J. Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. Conference on Robot Learning, pages 1368-1378. PMLR, 2023.
[12] Y. Han, K. Yu, R. Batra, N. Boyd, C. Mehta, Zhao, Y. She, S. Hutchinson, and Y. Zhao. Learning generalizable vision-tactile robotic grasping strategy for deformable objects via trans- former. arXiv preprint arXiv:21 2021.
[13] I. Guzey, B. Evans, S. Chintala, and L. Pinto. Dexterity from touch: Self-supervieed pre- training of tactile representations with robotic play. In Conference on Robot Learning, pages 3142-3166. PMLR, 2023.
[14] H. Liang, C. Zhou, S. Li, X. Ma, N. Hendrich, T. Gerkmann, F. Sun, M. Stoffel, and J. Zhang. Robust robotic pouring using audition and haptics. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10880-10887. IEEE, 2020.
[15] L. Fu, H. Huang, L. Berscheid, H. Li, K. Goldberg, and S. Chitta. Safe self-supervised learning in real of visuo-tactile feedback policies for industrial insertion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10380-10386. IEEE, 2023.
[16] C. Sferrazza, Y. Seo, H. Liu, Y. Lee, and P. Abbeel. The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning. 2023.

---

[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[18] H.-S. Fang. C. Wang, M. Gou, and C. Lu. Graspnet-lbillion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 1 11444-11433 11453, 2020.
[19] S. Yu, D.-H. Zhai, Y. Xia, H. Wu, and J. Liao. Se-resunet: A novel robotic grasp detection method. IEEE Robotics and Automation Letters, 7(2):5238-5255, 2022.
[20] Y. Song, J. Wen, D. Liu, and C. Yu. Deep robotic grasping prediction with hierarchical rgb-d fusion. International Journal of Control, Automation and Systems, 20(1):243-254, 2022.
[21] R. Qin, H. Ma, B. Gao, and D. Huang. Rgb-d grasp detection via depth guided learning with cross-modal attention. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 8003-8009. IEEE, 2023.
[22] X. Pang, W. Xia, Z. Wang, B. Zhao, D. Hu, D. Wang, and X. Li. Depth helps: Improving pre- trained rgb-based policy with depth information injection. arXiv preprint 7xX1:22408.01107, 2024.
[23] L. Wellhausen, R. Ranftl, and M. Hutter. Safe robot navigation via multi-modal anomaly detection. IEEE Robotics and Automation Letters, 5(2):1326-1333, 2020.
[24] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In 2023 IEEE International Conference on Robolics and Automation (ICRA), pages 10608- 10615. IEEE, 2023.
[25] C. Xu, J. Chen, M. Yao, Zhou, L. Zhang, and Y. Liu. 6dof pose estimation of transparent object from single rgb-d image. Sensors, 20723):6790, 2020.
[26] M. Tian, L. Pan, M. H. Ang, and G. H. Lee. Robust 6d object pose estimation by learning rgb-d features. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6218-6224. IEEE, 2020.
[27] Y. He, H. Huang, H. Fan, Q Chen, and J. Sun. Ffb6d: A full flow bidirectional fusion network for 6d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3003-3013, 2021.
[28] S. Cui, Wang, Wei, J. Hu, and S. Wang. Self-attention based visual-tactile fusion learning for predicting grasp outcomes. IEEE Robotics and Automation Letters, 5(4):5827-588 2020.
[29] S. Cui, R. Wang, J. Wei, F. Li, and S. Wang. Grasp state assessment of deformable objects using visual-tactile fusion perception. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 538-544 IEEE, 2020.
[30] E. Smith, R. Calandra, A. Romero, G. Gkioxari, D. Meger, J. Malik, and M. Drozdzal. 3d shape reconstruction from vision and touch. Advances in Neural Information Processing Systems, 33: 1429,, 2020.
[31] Y. Wang, W. Huang, B. Fang, F. Sun, and C. Li. Elastic tactile simulation towards tactile-visua perception. In Proceedings of the 29th ACM International Conference Multimedia, pages 2690-2699 2021.
[32] M. A. Lee, Y. Zhu, K. Srinivasan, Shah, Savarese, L. Fei-Fei, A. Garg, and Bohg- Making sense vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference Robotics and Automation (ICRA), pages 8943-8950. IEEE, 2019.
10

---

[33] J. Wilson and M. C. Lin. Avot: Audio-visual object tracking of multiple objects for robotics. In 2020 JEEE International Conference on Robotics and Automation (ICRA), pages 10045- 10051. IEEE, 2020.
[34] X. Qian, Z. Wang, J. Wang, G. Guan, and H. Li. Audio-visual cross-attention network for robotic speaker tracking. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, 31:550-562, 2022.
[35] C. Gan, Y. Zhang, J. Wu, B. Gong, and J. B. Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 9701-9707. IEEE, 2020.
[36] C. Chen, U. Jain, C. Schissler, S. V. A. Gari, Z. Al-Halah, V. K. Ithapu, P. Robinson, and K. Grauman. Soundspaces: Audio-visual navigation 3d environments. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part VI 16, pages 17-36. Springer, 2020.
[37] A. Younes, D. Honerkamp, Welschehold, and A. Valada. Catch me you hear me: Audio- visual navigation in complex unmapped environments with moving sounds. IEEE Robotics and Automation Letters, 8(2):928-99) 905,9355 2023.
[38] R. Gao, Y.-Y. Chang, S. Mall, L. Fei-Fei, and J. Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In Conference on Robot Learning, pages 466-476. PMLR, 2022.
[39] R. Gao, Z. Si, Y.-Y. Chang, S. Clarke, J. Bohg, L.. Fei-Fei, W. Yuan, and Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10598-10608, 2022.
[40] X. Yang, Z. Ji, J. Wu, Y.KK. Lai, C. Wei, G. Liu, and R. Setchi. Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems, 33(9):4727-474) 2021.
[41] B. Beyret, A. Shafti, and A. A. Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 JEEE/RS International Conference on intelligent robots and systems (IROS), pages
IEEE, 2019.
[42] M. M. Botvinick. Hierarchical reinforcement learning and decision making. Current opinion in neurobiology, 22(6):956 (0t0)999666666-000 2012.
[43] Z. Yang, K. Merrick, L. Jin, and H. A. Abbass. Hierarchical deep reinforcement learning for continuous action control. IEEE transactions on neural nefworks and learning systems, 29 (11)55174-518,, 2018.
[44] D. Zhang. Q. Li, Y. Zheng, L. Wei, D. Zhang, and Z. Zhang. Explainable hierarchical imitation learning for robotic drink pouring. IEEE Transactions on Automation Science and Engineer- ing, (94)33871-3887 2021.
[45] F. Xie, A. Chowdhury, M. De Paolis Kaluza, L. Zhao, L. Wong, and R. Yu. Deep imitation learning for bimanual robotic manipulation. Advances in neural information processing sys- tems, 33:2327-2337, 2020.
[46] B. Li. J. Li. T. Lu, Y. Cai, and S. Wang. Hierarchical learning from demonstrations for long- horizon tasks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1545-4551 IEEE, 2021.
[47] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin. Hierarchical few-shot imi- tation with skill transition models. In International Conference Learning Representattons, 2021.

---

[48] Z. Jia, F. Liu, V. Thumuluri, L. Chen, Z. Huang, and H. Su. Chain-of- of-uhh predictive control, arXiv preprint arXiv:2304.00776, 2023.
[49] J. Arevalo, T. Solorio, M. Montes-y Gómez, and F. A. González. Gated multimodal units for information fusion. arXiv preprint arXiv: 2017.
[50] J. Kim, J. Koh, Y. Kim, J. Choi, Y. Hwang, and J. W. Choi. Robust deep multi-modal learning based on gated information fusion network. In Asian Conference on Computer Vision, pages 90-106. Springer, 2018.
[51] Wang. Zhang, and C. Zong. Learning multimodal word representation via dynamic fusion methods. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[52] Arevalo, T. Solorio, M. Montes-y Gomez, and A. González. Gated multimodal networks. Neural Computing and Applications, 32:10209-10228, 2020.
[53] Z. Han, Yang. J. Huang, C. Zhang, and J. Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 20707-20717, 2022.
[54] Q. Zhang, H. Wu, C. Zhang, Q. Hu, H. Fu, J. T. Zhou, and X. Peng. Provable dynamic fusion for low-quality multimodal data. In International conference machine learning, pages 41753-41769, PMLR, 2023.
[55] M. K. Tellamekala, S. Amiriparian, B. W. Schuller, E. André, T. Giesbrecht, and M. Valstar. Cold fusion: Calibrated and ordinal latent distribution fusion for uncertainty-aware multimodal emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[56] A. Nagrani, S. Yang. A. Arnab, A. Jansen, C. Schmid, and C. Sun. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34:1 1200-1 2021.
[57] Li, C. Zou, Y. Li. X. Zhao, and Y. Gao. Attention-based multi-modal fusion network for semantic scene completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11402-11 1922-11499 11409, 2020.
[58] V. Chudasama, P. Kar, A. Gudmalwar, N. Shah, P. Wasnik, and N. Onoe. M2fnet: Multi-modal fusion network for emotion recognition in conversation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4652-4661, 2022.
[59] G. Li, Y. Wei, Y. Tian, C. Xu, J.-R. Wen, and D. Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19118, 2022.
[60] P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. Conference on Robot Learning, pages 175-187. PMLR, 2023.
[61] X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y. Jing, W. Zhang, H. Liu, et al. Vision- languaee foundation models as effective robot imitators. In The Twelfth International Conference Learning Representations, 2023.
[62] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 988) 1780, 1997.
[63] M. Du. O. Y. Lee, Nair and C. Finn. Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning. arXiv preprint 2rXiv:2205.1480 2022.
12

---

[64] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[65] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 412.6980, 2014.
[66] W. Xia, D. Wang, X. Pang. Wang, B. Zhao, D. Hu, and X. Li. Kinematic-aware prompt- ing for generalizable articulated object manipulation with 1lms. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 2073-2080. IEEE, 2024.
[67] Anthropic. Claude 3.5 sonnet. Website, 2024. https://www: http://wwww.ttt.o.ccccc cww/co////ce/ee nnww///w// claude-3- Sande-3-5-sonnes sonnet.
[68] C. Chi, S. Feng, Y. Du. Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[69] Y. Wei, R. Feng, Z. Wang, and D. Hu. Enhancing multimodal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27338-27347, 2024.
13

---

Appendix
6cm, as A Details of the Task Setup
In Sec. 4.1 of the main paper, we briefly introduced the basic information of the pouring task and the peg insertion with keyway task, including the task objectives and stage divisions. In this section, we provide a more detailed introduction to the setup of these two tasks. We control the robot arm through a keyboard to complete the tasks and collect human demonstrations.
A.1 Pouring
Cylinder with 20/ brads inttially
Target Cyllnder
Shift training demonstrattios
Shift real-world testing
Figure 5: Illustration of the pouring task. We randomly shift the fixed target cylinder sideways by 0 ~ 3cm (indicated by the blue arrow) in training demonstrations, and shift by 0 ~ 6cm (indicated by the orange arrow) during testing.
Setup Details. For the pouring task, the robot needs to pour tiny steel beads of specific quality from the cylinder the hand into another cylinder. In the demonstrations, we randomly shift the fixed target cylinder sideways by 3em, while during testing, this range expands small beads with diameter of 1mm
illustrated in Fig. 5. Following the previous work [11], to simulate liquids. The initial mass of the beads is 90g/ while the target mass for pouring out is 40g/60g, both indicated by prompts. Since the camera cannot capture the interior of the target cylinder, other modalities are needed to assess whether the poured-out quantity meets the target. Hence, we use vision (RGB), audio and touch modalities in this task.
Robot Action Space. In this task, the robot can in a 2-dimensional action space along the axis and o, where represents the horizontal movement and represents the rotation of the gripper. The action step size is At 0.5mm and Aø 0.12 There are total of 5 possible actions (#Ar, =AØ and 0), corresponding to the two directions of the two dimensions of (a, Φ) and holding still.
Stage Division. This task consists of four stages [44]: 1) Aligning, where the robot needs to move graduated cylinder containing beads and align it with the target cylinder. 2) Start Pouring, where the robot rotates the end effector to pour out the beads at an appropriate speed, 3) Holding Still, where the robot maintains its posture continue pouring out beads steadily, and 4) End Pouring, where the robot rotates the end effector at the appropriate time to stop the flow of beads. In trajectories, we consider the timesteps of the first downward rotation of the gripper (-AØ), the first holding still after rotation, and the first upward rotation of the gripper (+A0) as stage transition points.
A.2 Peg Insertion with Keyway.
Setup Details. This task is an upgraded version of peg insertion, where the robot needs to first align peg with key the keyway the base by rotating, and then insert the peg fully. The alignment between the key and the keyway in this task primarily relies tactile feedback, as the
14

---

2
the tranning dessunct stiine
lattal postioa robec
Ieliti protiia robuut daringg torting
Figure 6: Illustration of the peg insertion with keyway task. We randomly shift the fixed base along a 12cm-long parallel line on the desktop (the yellow arrow), and randomly initialize the po- sition of the robot arm inside 40cm x 20cm rectangular area (the green rectangle) in training demonstrations. During testing, we fix the position of the base (the red points) and the robot arm (the green points) to several pre-defined points.
camera cannot observe inside the hole. Hence, we use RGB, depth and touch modalities in this task. Considering generalization, randomly fix the base any position along a 12cm-long parallel line the desktop in demonstrations, illustrated in Fig. 6. The robot arm holding the peg can also be initialized inside 40cm 20em rectangular area around the base. During testing, to ensure fairness, the positions of the base and the robot arm are several pre-defined points.
Robot Action Space. In this task, the robot arm can move on the three axes x, y, z of Cartesian coordinate, where æ, represents the horizontal movement and represents the vertical movement. The gripper can also rotate along axis e to align the key with the keyway. Since the vertical move- ment of the robot arm (=) and the rotation of the gripper (+AØ) both unidirectional, there are total of 7 possible actions AtAz +Ay, Az + and 0).
Stage Division. This task consists of three stages: 1) First Insertion, where the robot aligns the peg with the hole and inserts it until the key collides with the base, 2) Rotating, where the robot aligns the key with the keyway on the base by rotating the peg based on tactile feedback, and 3) Second Insertion, where the robot further inserts the peg the bottom. Similar to the pouring task, consider the timesteps of the first gripper rotation (+1)) and the first downward movement (-Az) after rotation stage transition points.
A.3 Generalization Experiments with Distractors
To verify the generalization of our framework distractors in the environment, we conduct experi- ments with visual distractors on both tasks. For the pouring task, change the color of the cylinder from white to red. As for the peg insertion task, respectively change the color of the base from black to green, and scatter some clutter around the base. These task settings are illustrated in Fig. 7. The distractors only exist during testing. Besides introducing distractors, the settings for these tasks remain consistent with the main experiments.
B Implementation Details
Following [11], we resize visual and tactile frames to 140 x 105 and randomly crop them to 128 96 during training. We also use color jitter for image augmentation. For audio modality, we resample the wave signal at 16kHZ and generate 64 x 50 mel-spectrogram through short-time Fourier trans- form, with 400 FFT windows, hop length of 160, and mel bin of 64. We employ ResNet-18 [64] network as the uni-modal encoder. Each encoder takes brief history of observations spanning
15

---

(a) Color (Pouring)
(b) Color (Insertion)
(c) Mess (Insertion)
Figure 7: Illustration of tasks with distractors. For the pouring task, we change the color of the cylinder from white to red (denoted as "Color (Pouring)"). For the peg insertion with keyway task, respectively change the color of the base from black to green (denoted "Color (Insertion)"), and seatter some clutter around the base (denoted as "Mess (Insertion)").
= 6 timesteps. We train all the models using Adam [65] with a learning rate of 10-4 for 75 epochs. We perform linear learning rate decay every two epochs, with a decay factor 0.9.
For action history, hot encoding. Action sequences shorter than 200 padded with zeros. For both tasks, consider the 15 timesteps near the stage transition point as the soft constraint range (? 15). We set À = 5.0 for both tasks to control the intensity of the penalty. For the learnable stage token [stage;], we initialize with the mean of state tokens on samples within the i-th stage after warmup training for epoch. We use 3 0.5 for the calculation of the stage-injected state token at. Moreover, to prevent gradients from becoming too large, we also truncated the gradients of the stage score penalty, restricting them solely influencing the gate network.
a buffer of length 200 store actions. Each action encoded using one-
C Random Attention Blur
Stage-Guided Artentittonionn
Key
(10 x (olens)
Query token)
Afteaction
(M x)
Random Attention Blur (probability
Asaraped Atteacion
(f x.
Figure 8: Illustration of random attention blur in the stage-guided dynamic fusion module. We randomly replace the attention scores on all feature tokens with the same average value MxT with a probability p. which a hyper-parametes
In order to prevent the model from simply memorizing the actions corresponding to attention score patterns, we introduce random attention blur mechanism to the stage-guie sagge-guieed dynamic fusion mod- ule. For each input, we replace the attention scores on all feature tokens with the same average value MxT with a probability p. where M the number of modalities and 7 is the timestep number of the brief observation history, as illustrated Fig. We set p = 0.25 in both tasks.
We introduce this mechanism due to the potential overfitting issue in the model, where the policy head (MLP in the dynamic fusion module) learns the correspondence between the distribution of
16

---

attention scores and actions. For instance, when using the trained MS-Bot model (without random attention blur) to complete pouring tasks, manually increasing the attention scores on tactile feature tokens invariably leads the model to predict the next action as upward rotation (+AØ), regardless of the current stage of the task. This phenomenon suggests that the stage comprehension module in the model partially assumes the role of action prediction rather than focusing solely on stage understanding. Therefore, randomly blurring attention scores compel the policy head to focus on the information from the feature tokens and better decouple the stage comprehension module from the dynamic fusion module.
D LLM-Based Automatic Stage Labeling
Our strategy for stage labeling is to segment the action sequence of the trajectory based on pre-set rules (such as considering the timestep of the first gripper rotation as a stage transition point). This does indeed require some time to pre-define the stages and design specific rules for division. How- ever, extracting stage information without human annotation is also feasible. One possible approach is to use LLMs, which have demonstrated strong comprehension ability in robotic manipulation [66], for automatic stage annotation of action sequences. We use Claude 3.5 Sonnet [67] as our LLM, and construct prompts to describe the task setup and include an example action sequence to better help the LLM understand the process of this task. We then ask the LLM to provide the specific stage division, as shown in Fig. 9.
We want the rohotic arm complete a insertion with keyway task. In this tusk, the robotic arm needs to align peg with the keyway on hase by rotation, then fully insert peg. At each moment. the robotic arm chooses action to execute from seven options: move left, move right, move forward, move backward, mave down, rotate the peg, and stay, Here is example of the complete action sequence: [left, left, left, left, left, backward, backeward, backward, backward, backward, buckward, left, left, left, left, down, down, down, down, down, down, down, down, left, down, down, down, down, down, down, rotate, rotate, rotate, rotate, down, down, down, down, stay, stay] How many main stages are there this task? Please divide the action sequence into rough stages based the sub-goals stage sepurate stages.
User
Pleuse do not fine adjustments within a main
To unalyze this action sequence and divide into stages, let's look at the major movements and
goals. ('ll group similar consecutive actions together and identify the stages based the apparent sub-goals goals of the task. Here's the breakdown:
LLM
Horizontal Alignment (1 ef movementy: The sequence starts with long series "left"
movements (95 times).
2. Depth Adjustment (Rackward movement): Following the left movements, there's significant series "backward" movements (43 times).
3 Fine Horizontal Adjustment: A small series of "left" movements (9 times) follows. This is likely a fine adjustment, we'll consider part the previous stage rather separate one. Vertical Approach (Down movement): There's long series of ddwnn" movements (159 times), bringing the peg closer to the insertion point.
Rolation Alignment: significant number of "rotate" actions (146 times) occur, likely to aliggn the peg with the keywny.
6. Final Inscation: The sequence ends
scries of "down" movements (64 times), followed by several "stay" commands (13 times).
Based on this analysis, we can identify 4 main stages in this task:
Linitial, Positioning: Horizontal alignment and depth adjustment (includos the major movement series and the small fine adjustment)
Vertical Approach: Bringing the peg down towards the insertion point
Rotation Alignment: Aligning the peg with the keyway
4. Final Inseation: Insorting the peg and maintaining position
stages correspond
sub-goals of peg insertion with keyway task:
positioning the arm above the insertion point, approaching the insertion point, aligning the pog with the keyway. and finally inserting the peg.
Figure 9: An example of automatic stage division by LLM. We provide the task setup, robot action space and an example action sequence to the LLM to obtain the main stage division. For ease of presentation, the action sequence in the figure has been abbreviated.
17

---

After obtaining the stage divisions provided by the LLM, we then construct another prompt and ask the LLM to label the action sequence according to the stage divisions, as shown in Fig. 10. This strategy allows for obtaining stage labels for all samples without requiring human annotation. We find that the stage divisions provided by the LLM are usually more detailed. For example, divides the peg insertion with keyway task into four stages (splitting our First Insertion stage into two stages: Initial Positioning and Vertical Approach), which is also reasonable. Since model does not explicitly require that the modality importance must be different between stages, more detailed stage division is acceptable some extent.
Through this method, we perform automatic stage annotation on our imitation learning dataset. To evaluate the accuracy of the LLM's annotations, we merge the first two stages divided by the LLM into single First Insertion stage, and the human-annotated stage labels ground truth. The evaluation results show that the accuracy of the stage labels annotated by the LLM is 95.68% (10,096 / 10,552, about 15 errors per trajectory).
For tasks like peg insertion with keyway, where many actions are continuous, we can merge consec- utive actions to obtain a compressed action sequence like rurtiionmumm..aattiiin Fig. 11. This approach helps reduce annotation errors caused by LLM counting mistakes and increases the accuracy 99.65% (10,521 10,552, about 3 errors per trajectory). These results demonstrate that automatic stage annotation using LLMs without pre-defined stages is feasible. The experiment is only relatively simple attempt. We believe that with more detailed prompt engineer- ing and the use of more powerful LLMs, the accuracy can be further improved.
as shown
E Comparison of Attention Scores between MULSA and MS-Bot
In Sec. 4.4 of the main paper, we illustrate the aggregated attention scores for each modality and the stage scores of MS-Bot the pouring task, as shown in Fig. 12a. We also record the changes in attention scores and stage scores in the peg insertion with keyway task, shown in Fig. 12b. The results in the figure demonstrate that model accurately predicts the rapid changes in stages for both tasks. As result, the attention scores across modalities guided by stage comprehension also relatively stable, exhibiting clear inter-stage changes and minor intra-stage adjustment.
As comparison, we also visualize the attention scores of another baseline MULSA [11] with self- attention fusion, shown in Fig. 13. The range of the attention score axis the figure consistent with Fig. 12 (0.0 1.0). It is evident that the attention scores of the modalities in the MULSA model close and exhibit small variation along the time axis, lacking clear stage characteristics. This indicates that the MUI.SA model fails to fully leverage the advantages of dynamic fusion compared to the concat model.
Combination with Diffusion Policy
Fusion Module
Policy Network
Success Rate
Concat
MLP
7/15
Concat
Diffusion Policy [68]
11/15
Stage-guin guiged Dynamic Fusion stage-guided Dynamic Fusion
MLP
13/15
Diffusion Policy [68]
13/15
Table Comparison of performance with Concat and Diffusion Policy basclines on the peg insertion with keyway task.
To verify the applicability of our proposed stage-guided dynamic multi-sensory fusion to different policy networks, we replace the policy network with the Diffusion Policy [68] and conduct experi- ments the peg insertion with keyway task. Specifically, we replace the MLP policy network with an UNet-based Diffusion Policy in the Concat baseline and our MS-Bot model, and evaluate the performance of stage-guided dynamic fusion models using different policy networks.
18

---

We want the robotic arm complete a peg insertion with keyway task. task, the robotic needs to align peg with the keyway the buse rolation, and then fully insert the peg. At each moment, the robotic arm chooses one action to esccute from seven options: move left, move right, move forward, move
User
move
peg. stay.
This task contains main stages:
Initial Positioning: Horizontal alignment and depth adjustment (includes the first two major series the small fine adjustment)
Vertical Approach: Bringing the down towards the insertion point
Rotation Aligmment: Aligning peg the keyway
Final Insertion: Inserting the peg and maintaining position
Please label the stuge number to which each uction the action sequence belongs order, and
list like: [l, 1,1,2,2,3,3,4
Here is the action sequence with a length of 528: [lett. left, left, left. left, left. backward, backward, backward, backward, buckward, backward, down, down, down, down, down, left, down, down, down, down, down, down. rotatio rotate, rotate, down, down, down, down, stay, stay]
down, down, down,
label the stapes for each action in sequence, I'll through the actions assign stages based the description provided. Here's the breakdown:
LLM
Initislioe empply bold stage
stage labels
Initial Positionigg (Stage 1)
97 left movements
stage labels extend() 97)
43 backward movements
stape labol. extenlt] 43)
more left movements
stage labels.extend([]
Approach (Stage
el 158 down mevements (including the one mised with lef)
stape labeb. (bbebceseed((]]
Romation Alignment (Stage
146 rotate actions
stage Jabels. [3]
Final Inseriion (Sage
down movements
stage labels (44 * 63)
stay
stage labels. exteeee 12)
Full list of singg labels
print(stage labels)
with corresponding stage. The resulting list has 528 elements, matching the longth of your original action sequence, Here's summary the stage labeling:Stago (Initial Positioning):
I've created Python script that labels each action in the
first
backward, 9 lett)Stugee 2 (Vertical Approach): The next 158 actions (all down movements SStuee (Retation Aligament): The ncxt actions (all rotate movements |Staee (Final Insertion): The Final actions (63 down movements. 12 actions)You
the Hall list
labels the artifact above. The first few elaments are [1. I. I.I. I, I. 1], and the last fow elements [4,
4j. should accurately represent the stages of the insertion with keyway task based the action sequence you provided. Let me know you need any clarification have any questions about the stage labeling!
aligns
stage breakdown. This labeling
Figure 10: An example of automatic stage labeling by LLM. We provide the task setup, robot action space and an target action sequence to the LLM to obtain the stage labels. For ease of presentation, the action sequence in the figure has been abbreviated.
The results Tab. 4 show that replacing the policy network in the Concat baseline with the Diffusion Policy leads to a significant performance improvement, but there still gap compared to MS-Bot. This is because the Diffusion Policy still lacks an explicit feature fusion module (apart from simple concatenation). The fusion of multi-sensory features this baseline still occurs within the policy network, similar to the concat baseline, albeit with more capable policy network. After further replacing its fusion module with proposed stage stiged dynamic fusion, it achieves performance
19

---

We want the robotic arm complete a peg insertion with keyway task. task, the robotic needs to align peg with the keyway the buse rotation, and then fully insert the peg. At each moment, the robotic amm chooses one action to esccute from seven options: move left, move right, move forward, move backward, move This task contains main stages:
User
peg. stay.
Initial Positioning: Horizontal alignment and depth adjustment (includes the first two major series the small fine adjustment)
Vertical Approach: Bringing the down towards the insertion point
Rotation Aligmment: Aligning peg the keyway
Final Insertion: Inserring the peg and maintaining position
Please label the stuge number to which each uction the action sequence belongs order, and
list like: [l, 3,,,34
Here is the action sequence with a length of 528: [let*97, buckward 1eft*9, down"44, lef
rotate" down*63. stay"12]
To label the stages for each action in the given sequenoe, let's break down based on the description of the four main stages:
LLM
Positioning: This includes
alignment depch adjustment. given
sequence, this corresponds the initial left and backward movements.
Vertical Appronch: This stage brings the peg down towards insertion point. It corresponds to the first sct of dowaward movements.
Rotation Alignment: This stage aligns the with the keyway through rotation.
Final Insertion: This stage the pog and maintains the position. corresponds final dowzward movements and the stay action.
Now, let's label each action in the sequence:
917 backward"43, left*9): Stage (Initial Positioning)
[down"44, left*1, down" 113j: Stage (Vertical Approach) [rotate" 146): Stage 3 (Rotation Aligament)
[down"63, 12]]: Stage (Final Insertion)
To the list of stage labels, Here's the resulting list:
expand based its repetition count.
[1] 128 146 [4]*75
gives with 149 I'ss (for stage 1), followedd by 158 2's (for stage 2), then 146 3's (for stage 3), and finally 75 4's (Jor stage 4).
elements, matching longth
sequence,
Figure 11: An example of automatic stage labeling by LLM. We provide the task setup, robot action space and an compressed target action sequence to the LLM to obtain the stage labels.
comparable to MS-Bot. This demonstrates that our proposed explicit stage understanding and stage- guided dynamic fusion are still effective for models with more powerful policy networks. Our proposed fusion module can help the policy network focus more on generating actions using the already fused multi-sensor features. However, under the same conditions of using stage-guided dynamic fusion, the model with diffusion policy the policy network does outperform the model with MLP. We think this may be due the relatively small scale of data in this task, which does not allow the Diffusion Policy to fully leverage its advantages.
G Comparison with Stage-Aware and Context-Aware Baselines
Due to not fully considering the importance of explicit stage understanding in guiding multi- fusion, the original Concat and MULSA baselines does not utilize contextual inputs and additional stage information. make fairer comparison with our MS-Bot, we respectively construct Concat and MULSA baselines that incorporate stage prediction auxiliary task and multi-sensory observa- tion context. Specifically, add extra head (an MLP identical to the gate network in MS-Bol to both the Concat and MULSA models to predict the stage, trained with the same form of loss
20

---

Seat D.
Seu Pul
Saacriica
(a) Pouring
(h) Peg insertion with keyway
Figure 12: Visualization of the aggregated attention scores for each modality and stage scores of MS-Bot in both tasks. At each timestep, we average the attention scores on all feature tokens of each modality separately. The stage score is the softmax normalized output of the gate network.
Pegteeriae
(a) Pouring
(b) Peg insertion with keyway
Figure 13: Visualization of the aggregated attention scores of MULSA for each modality in both tasks. At each timestep, we average the attention scores on all feature tokens of each modality separately. The range of the attention score axis in the figure is consistent with Fig. 12.
used to penalize the scores of the gate network in MS-Bot. We also attempt to modify the structure of the MUL-S model to accept observation context as input. We input the observations from 5s, 7.5s, 10s, 12.5s, 15s, 17.5s, and 20s before the current timestep into the model, and encode these observation contexts by each modality's 's encoder into 7 tokens (per modality). These tokens are then concatenated with the original 6 tokens of the current timestep observation of each modality, result- ing in total of 13 tokens for each modality. We evaluate these baselines on the peg insertion with keyway task. The results in Tab. shows that introducing an explicit stage prediction task could improve the performance of the Concat baseline, but it provides minimal benefit for the MULSA method. These baselines still have a performance gap compared to our MS-Bot. This indicates that explicit stage understanding is meaningful, but introducing context to aid stage understanding and specially designed stage-guided dynamic fusion modules are also necessary. results show that inputting observation contexts into MULSA results in virtually no improvement in the model's performance. This indicates that mixing the contexts and current observations in self-attention calculations is highly inefficient for helping the model understand stages and allocate modality weights, as it reduces focus the more important current observations and significantly increases GPU memory usage. In contrast, our MS-Bot leverages a long action history context to enhance its understanding of task stages. This approach simplifies processing for the model and offers finer granularity, effectively guiding dynamic multi-sensor fusion.
addition, the
H Comparison under the Multi-Camera Setting
In robotic manipulation, a common setup involves using multiple cameras to capture visual infor- mation from different views. By treating visual images from different views as distinct modalities, our MS-Bot is also able to handle multi-camera observations. To verify this capability, we conduct a comparison with baseline methods on the peg insert with keyway task under the setting with multiple cameras. Specifically, we place another camera the side of the base, so that the lines connect- ing these two cameras to the base roughly form right angle. We replace the depth observation from the first camera with the RGB observation from the second camera, as the RGB observations
21

---

Method
Success Rate
Concat
7/15
MULSA
11/15
Concat +
Stage Auxiliary Task
9/15
MULSA + Stage Auxiliary Task
11/15
MULSA Observation Context 11/15
MS-Bot
13/15
Table 5: Comparison of performance with stage- aagee and context-awand baselines on the peg inser- tion with keyway task. We introduced stage prediction auxiliary tasks and qulti-sensory observation context into the Concat and MULSA baselines for more comprehensive comparison.
Method
Success Rate
Concat 9/15
MULSA
12/15
MS-Bot
14/15
Table Comparison under the setting of multiple cameras on the peg insertion with keyway task.
from these two cameras already provide sufficient spatial information for this task. The results in Tab. show that in the multi-camera setup, our MS-Bot method still outperforms the baselines. This demonstrates that our proposed stage-guided dynamic fusion can also be applied fuse features from multiple cameras.
I Combination with Audio Modality
Method
Success Rate
Concat
8/15
MULSA
11/15
MS-Bot
13/15
Table Comparison on the peg insertion with keyway task using RGB, depth, tactile and audio.
For tasks involving using lock bick-picking tools to unlock in real life, audio can indeed serve as the basis for completing the task, as these locks typically have specific mechanical structures that produce distinct sounds. Similarly, audio can indicate whether the key and keyway are aligned, thereby assisting with the peg insertion with keyway task. To verify the effectiveness of our model after further incorporating audio, we conduct experiments on the peg insertion with keyway task using RGB, depth, tactile and audio modality. The results in Tab. 7 show that the addition of the audio modality only brings improvement to the weakest Concat baseline. For MULSA and our MS-Bot method, the benefits are minimal. This is because the audio in this task is short and minimal due to the 3D-printed materials and is vulnerable to noise disturbance.
Detailed Experimental Results of Pouring
In Sec. 4.4 and 4.5 of the main paper, we show the overall error on all settings the pouring task.
In this section, we comprehensively present the detailed result of the component ablation and the distractor experiment on all the settings of the pouring task in Tab. and 9, As shown in Tab. 8, the contributions from both the state tokenizer and the stage comprehension module on all settings demonstrate the importance of the coarse-to-fine finie stage comprehension. The consistent lead of MS-
22

---

Pouring Initial (g)
Pouring Target (g)
Method
90
120
40
60
MS-Bot
1.60 + 1.10
5.58 + 1.79
6,48 + 1.55
1.80 + 0.95
Attention Blur
1.72 1.09
5.70 + 1.74
6.55 + 1.38
1.95
1.32
Stage Comprehension
2.52 + 1.12
6.15 - 1.64
6.80
+ 1.59
2.92 + 1.40
State Tokenizer
3.05 + 1.01
6.42 + 1.98
7.12 + 1.66
4.19 1.24
Table 8: Impact of each component of framework in the pouring task (mean + standard devia-
tion). *_' indicates further removing the module from the model the previous line.
Pouring Initial (g)
Pouring Target (g)
Method
120
Concat
8.75 + 2.02
10.69 2.45
10.72 2.35
8.46 2.03
Du et al. [63]
8.51 + 1.79
9.54 2.10
9.98 + 2.20
8.04 = 1.85
MULSA [11]
4.72 + 1.30
7.83 + 1.71
8.45 + 1.50
5.56 1.13
MS-Bot
2.04 + 1.40
6.10 + 1.49
7.62 + 1.94
2.41 = 1.47
Table 9: Comparison of performance in scenes with visual distractors in the pouring task(mean standard deviation). We change the color of the cylinder from white to red during testing.
BOT across all settings of the pouring task in Tab. 3 also demonstrates the generalizability of stage comprehension.
K Evaluation of Hyper-parameter Settings
(a) Stage Score Penalty Intensity À
(b) Soft Constraint Range
(c) Attention Blur Probability
Figure 14: Performance of MS-Bot on one setting the pouring task when changing the hyper-parameters X, y and p. We fix the others when changing one hyper-parameter We report the mean error of the concat model, MULSA and AS-Bot. The error bars represent the standard deviation of errors for MS-Bot.
In this section, we test the sensitivity of MS-Bot to different hyper-parameter settings. Specifically, we test the performance of MS-Bot under different penalty intensity X, soft constraint range y and probability of random attention blur p in one setting of the pouring task. We train all the models to pour out 40g of small beads with different initial masses (90g/120g). We set = 5.0, 7 15 and p 0.25 by default, and keep the others fixed when we modify one of the hyper-parameters.
We present the evaluation results in Fig. 14. Our MS-Bot consistently outperforms both the concat model and MULSA across various hyper-paramette settings, indicating that the performance of our MS-Bot is relatively stable to pyper-parament
variations. However, we also observe that when setting À and 7 to small values (A 1.0 in Fig. 14a and y = in Fig. 14b). the performance of MS- Bot drops and becomes closer to MULSA with simple self-attention fusion. This is because when small, the prediction of the current stage becomes inaccurate due the insufficient training. When y is too small, the model is forced to make drastic changes in the stage score prediction within very few timesteps, leading to unstable stage predictions. Both of these factors weaken the efficacy
23

---

of stage comprehension within MS-Bot. We also find that setting too large (p > 0.3 in Fig. 14c) can bring negative impacts as the attention blur truncates the gradients backpropagated to the stage comprehension module and the state tokenizer. Excessive attention blur can impair the training of these two modules.
L Limitations and Future Improvements
In this section, we discuss the potential limitations of our work and the corresponding solutions:
In many robotic tasks this kind of dynamic weighting is not necessary. Indeed, there many tasks that can be completed with just the visual modality, where readings from other sensors may not provide useful information. We believe that one solution is to let the model first select the necessary subset of sensors based on the task (possibly using an LLM),, and then perform dynamic multi- sensor fusion based on stage understanding. For tasks with only one stage, this can be considered as special case where |S| 1.
The stage labeling may be costly. Currently, strategy for stage labeling is to segment the action sequence of the trajectory based on pre-set rules. This requires time to set up detailed rules and becomes more challenging when there are more stages and more complex stage transitions. A possible solution is the LLM-based automatic labeling method mentioned earlier, although a small amount human effort is still necessary, such as constructing prompts to accurately describe the task setup and providing example for in-context learning.
The lack of comparison to larger models. Currently, there is no well-trained open-source multi- sensory large foundation model for robot manipulation (including vision, hearing, touch, etc.). Therefore, we are temporarily unable to verify the role of explicit stage understanding in large models, nor can we directly compare our model with multi- large models. However, we be- lieve that even for larger models, explicit stage understanding remains meaningful when introducing new modalities and training is required using small amount of paired multi-sensory data, or when fine-tuning in new scenarios with limited data scale.
Robustness issue in the case of missing modalities. Since in both of our tasks there are stages that depend on a single modality, masking uni-modal features during testing may directly lead to failure. This is a problem common to all multi-sensory models, and we did not make improvements specifically for this issue. cases where multiple sensors complement each other, the absence of uni-modal features may have compounding effect stage understanding and dynamic fusion. We believe that possible solution is randomly dropout uni-modal features during the training process [69]. Some other methods for robust multi-modal learning could also be referenced.
Transferability of the current dynamic multi-sensory fusion architecture. Since the multi- sensory fusion architecture we use in the dynamic fusion module is based on cross-attention rather than self-attention, there are difficulties in directly transferring the original form of this fusion method to transformers with multiple self-attention blocks. One potential strategy is use the stage token as special token to replace the class token. It could be used to predict the stage under the supervision of stage labels and thereby achieving the effect of stage-guided multi-modal fusion.
Low integration level of current multi-sensory robot system. Our current multi-sensory robot system has not well integrated the various sensors: the camera positioned at a third sard-perron per- spective, the microphone is placed the desktop, and the tactile sensor is the end of the robotic arm. They located in different positions. This low level of integration means that the system can only complete relatively simple tasks, while more complex multi-sensory tasks that require mobility cannot be accomplished. This multi-sensory system still needs further refinement and upgrading.
24


Reviews:
[{"review_id": "qCLSDoThEu", "reviewer": "Reviewer_rczR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an imitation learning method that learns to fuse features from different modalities using additional information about the stage within the task. In particular, action history and current features are used to extract current state and stage, and this is information is used to weight the multimodal features, and weighted features are used for predicting actions. Experiments show that the proposed method outperforms baselines that don't explicitly consider such stage information.", "summary_of_recommendation": "I enjoyed reading the paper but the proposed idea is a bit limited in that it heavily depends on expensive stage annotation, and the experiments are missing some baselines and analysis that could further strengthen the claims made in the paper.  I'm currently in a borderline but slightly leaning towards weak reject. But I'm willing to change my score after reading other reviewers' opinions and the rebuttal response.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- See Review", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHaG5PK66M", "reviewer": "Reviewer_ConB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper explores the how to adaptively weigh individual modalities (\"modality importance\") of a multi-sensory robot when handling mutli-stage tasks within the context of complex sub-goals that require different levels of focus on given modalities. The core intuition is that sensory fusion needs to change for the given sub-task to be the most effective. They build a multi-sensory robot and use imitation learning to train a MULSA model, they additionally add a method to identify task stage and use this classification to adjust modality weights. The proposed method (MS-Bot) is applied to a pouring and a peg insertion with keyway task.", "summary_of_recommendation": "Overall I think the paper is of high enough quality for CoRL. The authors present a novel task (Modality Temporality) and a strong performing solution to this task, experiments show both the reason this presented task are important and the strengths of their proposed solution. Given the intuition and results presented, this task may become an area of interesting research in the future. A great paper and a joy to read.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Section 1:\n- Line 23: need to cite or show that optimus and figure 1 are multi-sensory robots.\n- Line 43-46: Is the general sentiment of this sequence correct? Do the later layers in sensory fusion methods not weigh the given sensor features naturally? I would expect this to be the case within the context of an end-to-end system. I am open to discussion on this but my intuition was that end-to-end systems would learn to weigh individual modalities during the course of a multi-stage task, expecially if enough observational context is given to understand the task stage (it would have to understand the task stage to complete the multi-stage tasks), some logic behind your call for explicit stage understanding within the context of sensor modality weighting, in comparison to an end-2-end approach with enough context to know what stage it is in, would help motivate this research to the reader.\n\nSection 2:\n- Clear concise and well written, giving the reader enough context to understand the proposed solutions and research space. I have no comments that need addressing for this section. \n\nSection 3:\n3.1:\n- Line 106-107: What is the reasoning behind not using previous sensor readings in this method, is it to keep the model simple or is there a more complex reason behind this decision? A little sentence here would be useful for readers to understand the design decisions behind the model.\n- Line 112-121: Again this is very well written and makes perfect sense to me, I once again wonder if your intuition is correct. You state \"could\" so there is nothing wrong with this paragraph at all, however it \"could\" be improved if you had some backup evidence for your intuition, the results show that this method out performs MULSA which is end-to-end so you can use this to clarify to the reader that this intuition is currently correct, but given the lack of context in MULSA it is difficult to explore if this weighting would occure with multi-modal robot foundation models which take a context sequence of observations as input.\n\n3.2: \n- You could add z^{fus}_{t} to Figure 2 so that the figure better aligns with the Dynamic fusion description (Line: 171,172)\n\nSection 4:\n- Implementation of the methods details would be good to know, barebones essential parameters for the architecture and the training. Especially if there are querks to getting this system trained well.\n- Would be good to get more details on the 1st baseline ( the concat model which directly concatenates all the uni-modal features) - is it literally just this or are there other aspects to this?\n- I would like to see a comparison with a larger end-to-end solution that may make explicit tuning of modality weighting redundant, specific focus on the dynamic modality focus would be a great way to motivate the necessity to solve and explore the presented \"Modality Temporality\" challange and invite research in this area, helping produce a larger impact for this work. Although I understand this maybe a large undertaking and I do not think it is essential for this paper to be accepted. I think the best version of this paper would show that large models will still require this explicit sub-task modality weighting.\n- I would like to see a comparison of attention scores between the MULSA method and the method proposed in this work. That would provide insight into the level of impact the explict stage-based dynamic fusion is having.\n\nSection 5:\n- Limitations of the work are not discussed at a high enough standard, I think a more comprehensive analysis of the works limitations would really help readers understand where this work is currently failing and can be improved. Some limitations to address maybe that in many robotics tasks this kind of weighting is not necessary, the cost of labelling demonstrations with sub-task class, comparison to larger models, how this trained system may lack the robustness of an integrated system when dealing with uni-modality occlusions. I'm open to discussion on whether any of these are in fact limitations. Likewise I'm sure there are more to add.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x6wLPQs3Uq", "reviewer": "Reviewer_mfxN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose MS-Bot, a novel stage-guided multi-sensory fusion method with coarse-to-fine stage comprehension. Stage modules are designed to encode course-grained and fine-grained states. Authors demonstrate performance increase in pouring and insertion tasks.", "summary_of_recommendation": "The paper presents a novel method and complete analysis, and is clear to read. I recommend accept this paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Table I pouring results is a bit confusing, which of initial weight corresponds to which target weight?\n- For the key task, why do you decide not use audio in this setup? I can imagine that when opening a door lock, the sound actually help me get a sense if my key is correctly inserted. Also, people uses audio feedback when they use those lock picking tools to unlock doors.\n- Could you elaborate more on how you will handle long-horizon tasks with more stages?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Clear motivation and presentation;
- Innovative idea;
- Real-world results.

Weaknesses:
- The idea may not be timeless as larger models evolve;
- The stages need to be pre-defined;
- Some baseline comparisons may be included.

Post rebuttal: The authors did an excellent job during the rebuttal, after which all three reviewers recommended acceptance, including two strong accepts. The AC agrees with the recommendation. The authors are encouraged to revise the submission to clearly acknowledge the limitations, such as pre-defined stages, in the camera-ready.
Decision: Accept