Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation

Authors: ruoxuan  feng
di  hu
wenke  ma
xuelong  li
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-03 03:57:09

Abstract: Humans possess a remarkable talent for flexibly alternating to different senses when interacting with the environment. Picture a chef skillfully gauging the timing of ingredient additions and controlling the heat according to the colors, sounds, and aromas, seamlessly navigating through every stage of the complex cooking process. This ability is founded upon a thorough comprehension of task stages, as achieving the sub-goal within each stage can necessitate the utilization of different senses. In order to endow robots with similar ability, we incorporate the task stages divided by sub-goals into the imitation learning process to accordingly guide dynamic multi-sensory fusion. We propose MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage understanding, which dynamically adjusts the priority of modalities based on the fine-grained state within the predicted current stage. We train a robot system equipped with visual, auditory, and tactile sensors to accomplish challenging robotic manipulation tasks: pouring and peg insertion with keyway. Experimental results indicate that our approach enables more effective and explainable dynamic fusion, aligning more closely with the human fusion process than existing methods.

Main paper: 
 introduction: 
 Humans are blessed with the ability to flexibly use various sensors to perceive and interact with the world. Over the years, endowing robots with this potent capability has remained a dream for humanity. Fortunately, with the advancement of computing devices and sensors [1, 2] , building a multi-sensory robot system to assist humans has gradually become achievable. We have witnessed the emergence of some exciting multi-sensory robots, such as Optimus [3] and Figure 01 [4] .One challenge in tasks that robots assist humans with is the complex object manipulation task, which requires achieving a series of sub-goals. These sub-goals naturally divide a complex task into multiple stages. Many efforts attempt to make models comprehend these task stages, whether through hierarchical learning [5, 6, 7] or employing LLMs [8, 9, 10] . However, when incorporated into multi-sensory robots, this challenge becomes more profound. We need to not only understand the stages themselves but also rethink multi-sensory fusion from the fresh perspective of task stages.Completing sub-goals within a complex task may require different senses, with varying modality importance at each stage. Thus, stage transitions may entail changes in modality importance. To illustrate this, we build a multi-sensory robot system (Fig. 1 , left). We then use imitation learning to train a MULSA [11] model with self-attention fusion to complete pouring task and record the testing confidence (Fig. 1, right) . We observe correlations between the confidence and task stages after Figure 1 : An illustration for our multi-sensory robot system and a visualization of Modality Temporality in a multi-stage task: pouring. We show confidence in action prediction (maximum softmax score) when using the inputs of all modalities and selectively masking uni-modal features. Due to the changing importance of modalities, both evident inter-stage (coarse-grained) and minor intra-stage (fine-grained) changes in confidence are observed when masking uni-model features. The low confidence fluctuations near stage boundaries also reflect insufficient task stage understanding. masking one modality. For example, vision is most important in the aligning stage, and masking it causes a notable confidence drop. We also note minor modality confidence changes inside stages. These indicate that modality importance undergoes both inter-stage changes and intra-stage adjustments. We summarize this as a challenge in multi-sensory imitation learning: Modality Temporality.The key to addressing this challenge is understanding the current task stage and its concrete internal state, i.e., coarse-to-fine stage comprehension. Multi-sensory features should be dynamically fused to meet the needs of the coarse-grained stage and adjust to the fine-grained internal state. However, most of the existing works that utilize multiple sensors [12, 13, 14, 15] essentially use static fusion like concatenating. They lack the adaptability to dynamic changes of modality importance in complex manipulation tasks. Some recent works [16, 11] attempt to use attention for dynamic fusion, but they still fail to break out the paradigm of fusion based solely on current observations. They neglect the importance of stage understanding in multi-sensory fusion.In light of this key insight, we incorporate the stages divided by sub-goals into multi-sensory fusion. To perform coarse-to-fine stage understanding, we first encode current observations and historical actions into a state token and identify the current stage based on it. We then inject the stage information into the state token by weighting learnable stage tokens. For fusion, we propose a dynamic Multi-Sensory fusion method for roBot (MS-Bot) to allocate modality weights based on the state within current stage. Using cross-attention [17] with stage-injected state token as query, we dynamically select multi-sensory features of interest based on the fine-grained internal state. The dynamic allocation of attention score to feature tokens ensures effective fusion aligned with requirements.To evaluate our MS-Bot method in the real world, we build a robot system that can perceive the environment with audio, visual and tactile sensors. We test it on two challenging manipulation tasks: pouring, where the robot needs to pour tiny beads of specific quality, and peg insertion with keyway, where robot needs to rotate the peg with a key to align the keyway during the insertion. To summarize, our contributions are as follows:• We summarize Modality Temporality as a key challenge and identify the coarse-to-fine stage comprehension as a crucial point to handle the changing modality importance in complex tasks.• Based on the above insight, we propose a dynamic multi-sensory fusion method guided by the prediction of fine-grained state within the identified task stage.• We build a multi-sensory robot system and evaluate our method on two challenging robotic manipulation tasks: pouring and peg insertion with keyway.Experimental results show that MS-Bot better comprehends the fine-grained state in current task stage and dynamically adjusts modality attention as needed, outperforming prior methods in both tasks. We hope this work opens up a new perspective on multi-sensor fusion, inspiring future works. 
related work: 
 Multi-Sensory Robot Learning. Recent works have extensively integrated visual, auditory, and tactile sensors into robot systems. The two common visual modalities, RGB and depth, are widely combined and applied in grasping [18, 19, 20, 21, 22] , navigation [23, 24] , and pose estimation [25, 26, 27] tasks. In recent years, the development of tactile sensors has sparked interest in the study of tactile modality. Fine-grained tactile data can assist in tasks such as grasping [28, 29, 12, 13] , shape reconstruction [30, 31] and peg insertion [32, 15] . Audio modalities are typically applied in tasks such as pouring [11, 14] , object tracking [33, 34] , and navigation [35, 36, 37] . Some works even attempt to jointly use visual, audio, and tactile modalities to manipulate objects [38, 39, 11] . Based on these efforts, we further explore effective multi-sensory fusion in complex manipulation tasks.Robot Learning for Multi-Stage Tasks. Complex multi-stage tasks have always been a focal point of research in robot manipulation. Previous works hierarchically decompose the multi-stage tasks, where upper-level models predict the next sub-goal, and lower-level networks predict specific actions based on the sub-goal. Following this paradigm, many hierarchical reinforcement learning [7, 40, 41, 42, 43, 44] and imitation learning [5, 6, 45, 46, 47] methods have demonstrated their effectiveness across various tasks. [8, 9, 10] use LLMs or VLMs to decompose long-horizon tasks.Recently, [48] introduces Chain-of-Thought into robot control. While these works primarily focus on uni-sensory scenarios, we aim to investigate the impact of stage changes on multi-sensory fusion.Dynamic Multi-Modal Fusion. Dynamic multi-modal fusion essentially adapts the fusion process based on the input data by assigning different weights to the uni-modal features. A simple and effective way to assign weights is to train a gate network that scores based on the quality of uni-modal features [49, 50, 51, 52] . Another approach to assessing the quality of modalities and dynamically fusing them is uncertainty-based fusion [53, 54, 55] . With the introduction of the transformer [17] , self-attention provides a natural mechanism to connect multi-modal signals [56] . Hence, attentionbased fusion methods rapidly become a focal point of research [11, 57, 58, 59 ]. However, these methods primarily perform dynamic fusion based solely on the inputs themselves, while we design dynamic fusion from the perspective of task stages in robotic scenarios. 
method: 
 In this section, we discuss the challenges in multi-sensory imitation learning from the perspective of task stages and propose the dynamic multi-sensory fusion framework MS-Bot, based on coarseto-fine task stage understanding, to overcome these challenges. Specifically, the fine-grained state refers to a concept that describes the current situation of the environment, while the coarse-grained stage, divided by sub-goals of the task, is a collection composed of many states, as follows:EQUATIONwhere E is an episode, X t is the multi-sensory observation at timestep t and a t is the action.Imitation learning is a data-efficient robot learning algorithm where a robot learns a task by mimicking the behavior demonstrated by an expert. However, directly applying it to complex tasks with multi-sensory data could encounter issues, such as low confidence fluctuations and changing modality importance shown in Fig. 1 . We attribute the causes of these issues to two key challenges:Non-Markovity. Training data for imitation learning is typically derived from human demonstrations, usually collected through teleoperation devices. However, when humans manipulate robots, their actions are not solely based on data observed by robot sensors, but are also influenced by memory. This becomes particularly significant in more complex multi-stage tasks, as memory can provide cues about the task stages. Identifying the current stage solely based on current multi-sensory observations can be much more challenging. This challenge is not brought by the introduction of multiple sensors. Some works have already relaxed the Markov assumption and utilized action history as an additional input or leveraged sequence models to enhance performance [60, 61] . We identify the action history as a crucial cue for understanding the current stage and the fine-grained state within the stage, and use it to predict stages rather than directly predicting the next action. Modality Temporality. In a complex manipulation task, the importance of various uni-modal features could change over stages. At timesteps from different stages, a particular modality may contribute significantly to the prediction, or serve as a supplementary role to the primary modality, or provide little useful information. Moreover, different states within a stage, such as the beginning and end, may also exhibit minor changes in modality importance. We distinguish them as coarse-grained and fine-grained importance change. However, previous works typically treated each modality equally and did not consider the issue of changing modality importance in method design. This could lead to sub-optimal outcomes as uni-modal features with low quality could disrupt the overall prediction. On the contrary, we utilize task stage information to guide multi-sensory fusion, dynamically assigning weights of modalities based on the fine-grained state in the current stage.In this section, we introduce how our method MS-Bot addresses the above challenges. Considering Non-Markovity, we use action history as input to assist the model in assessing the current task state. For Modality Temporality, we establish a coarse-to-fine stage comprehension, decomposing the task state into coarse-grained stages and fine-grained states within a stage. Dynamic fusion guided by the fine-grained state within the current stage is employed to handle the changing modality importance.To introduce the understanding of stages into the process of imitation learning, we incorporate task stages divided by sub-goals into the training samples. Given a sample (X t , a t ) at timestep t in a trajectory where X t = {X 1 t , X 2 t , ..., X M t } is the observation of M modalities and a t is the corresponding action, a stage label s t ∈ {1, 2, ..., S} is added, where S is the number of stages in the task divided by different sub-goals. The sample at timestep t is then represented as (X t , a t , s t ). Based on the sample with stage division, we then introduce the four main components of our method MS-Bot:Feature Extractor. This component consists of several uni-modal encoders. Each encoder takes a brief history of observations X m t ∈ R T ×Hm×Wm×Cm of the modality m as input, where T is the timestep number of the brief history and H m , W m , C m indicates the input shape of modality m. These observations are then encoded into feature tokens f t ∈ R M ×T ×d where d denotes dimension.This component aims to encode the observations and action history (a 1 , a 2 , ..., a t-1 ) into a token that can represent the current state. Action history is similar to human memory and can help to indicate the current state within the whole task. We input the action history as a one-hot sequence into an LSTM [62] , then concatenate the output with the feature tokens and encode them into a state token z state t through a Multi-Layer Perceptron (MLP).Stage Comprehension Module. Merely using the state token alone is insufficient for achieving a comprehensive understanding of task stages. This module aims to perform coarse-to-grained stage understanding by injecting stage information into the state token. For a task with S stages, we use S learnable stage tokens [stage 1 ], ..., [stage S ] to represent each stage. After warmup training, each stage token is initialized with the mean of all state tokens on samples within the stage. Next, we use a gate network (MLP) to predict the current stage, i.e., coarse-grained stage comprehension. The S-dimensional output scores g t of the gate are softmax-normalized and multiplied by each of the S stage tokens, followed by summing up the results to obtain the stage token z stage t at timestep t via:g t = (g 1 t , ..., g S t ) = sof tmax(M LP (z state t)),EQUATIONWe use a softmax score instead of one-hot encoding because the transition of stages is a continuous and gradual process. Finally, we compute the weighted sum of the state token z state t and the current stage token z stage t using a weight β to obtain the stage-injected state token z * t : During the training process, we utilize stage labels to supervise the stage scores output by the gate net. Specifically, we penalize scores that do not correspond to the current stage. Additionally, for samples within a range near the stage boundaries, we do not penalize its score on the nearest stage, achieving a soft constraint effect. The loss for the gate net L gate on the i-th sample is as follows:EQUATIONEQUATIONwhere k indicates a nearby sample in the same trajectory, s i and s k represent stage labels and γ is a hyper-parameter used to determine the range near the stage boundaries. This soft constraint allows for smoother variations in the predicted stage scores.Dynamic Fusion Module. Due to variations in modality importance across different stages and the minor importance changes within a stage, we aim for the model to dynamically select the modalities of interest based on the fine-grained state within the current stage. We use the state token with stage information z * t as query, and the feature tokens f t as key and value for cross-attention [17] . This mechanism dynamically allocates attention scores to feature tokens based on the state token z * t , thereby achieving dynamic multi-sensory fusion over time. It also contributes to a more stable multi-sensory fusion, as the anchor z stage t changes minimally within a stage. The features from all modalities are integrated into a fusion token z f us t based on the current stage's requirements. Finally, the fusion token z f us t is fed into an MLP to predict the next action a t .In order to prevent the model from simply memorizing the actions corresponding to attention score patterns, we also introduce random attention blur mechanism. For each input, we replace the attention scores on feature tokens with the same average value 1 M ×T with a probability p. We encourage the model to simultaneously learn both stage information and feature information in the fusion token.The loss during training consists of two components: the classification loss for action prediction L cls and the penalty on the scores of the gate network L gate . The total training loss L is as follows:EQUATIONwhere λ is a hyper-parameter to control the intensity of the score penalty.  
experiments: 
 In this section, we explore the answers to the following questions through experiments: (Q1) Compared to previous methods, how much performance improvement does stage-guided dynamic multisensory fusion provide? (Q2) Why is stage-based dynamic fusion superior to fusion based solely on the input itself? How does stage comprehension help? (Q3) Can stage-guided dynamic fusion maintain its advantage over static fusion in out-of-distribution settings?Pouring. For the pouring task, the robot needs to pour tiny steel beads of specific quality from the cylinder in the hand into another cylinder. This task consists of four stages [44] : 1) Aligning, where the robot needs to move a graduated cylinder containing beads and align it with the target cylinder, 2) Start Pouring, where the robot rotates the end effector to pour out the beads at an appropriate speed, 3) Holding Still, where the robot maintains its posture to continue pouring out beads steadily, and 4) End Pouring, where the robot rotates the end effector at the appropriate time to stop the flow of beads. The initial mass of the beads is 90g/120g, while the target mass for pouring out is 40g/60g, both indicated by prompts. We use audio, vision (RGB) and touch modalities in this task.Peg Insertion with Keyway. This task is an upgraded version of peg insertion, where the robot needs to align a peg with a key to the keyway on the base by rotating and then insert the peg fully.The alignment primarily relies on tactile feedback, as the camera cannot observe inside the hole. This task consists of three stages: 1) First Insertion, where the robot aligns the peg with the hole and inserts it until the key collides with the base, 2) Rotating, where the robot aligns the key with the keyway on the base by rotating the peg based on tactile feedback, and 3) Second Insertion, where the robot further inserts the peg to the bottom. We use RGB, depth and touch modalities in this task.Fig. 3 briefly shows the two tasks. See Appendix A for a more detailed task setup for both tasks.Robot Setup and Data Collection. We use a 6-DoF UFACTORY xArm 6 robotic arm equipped with a Robotiq 2F-140 gripper in all experiments. For both tasks, we generate Cartesian space displacement commands at a policy frequency of 5 Hz. We use an Intel RealSense D435i camera to record visual data (RGB and depth) with a resolution of 640 × 480 at a frequency of 30Hz. We use a desktop microphone to record audio at a sampling rate of 44.1kHz. Tactile data are recorded by a GelSight [1] mini sensor with a resolution of 400 × 300 and a frequency of 15Hz.Baselines. We compare our method with three baselines in both tasks: 1) the concat model which directly concatenates all the uni-modal features [12, 13, 14, 15] , 2) Du et al. [63] that uses LSTM to fuse the uni-modal features and the additional proprioceptive information, and 3) MULSA [11] which fuses the uni-modal features via self-attention. We also compare our method with two variants in each task: MS-Bot (w/o A/D) and MS-Bot (w/o T/R) where one modality in the task is removed (audio and touch for pouring while depth and RGB for peg insertion). To evaluate how much improvement MS-Bot brings compared to previous methods, we conduct comparative experiments on the two tasks. For each task (considering changing initial weight and target weight as two tasks), we collect 30 human demonstrations and use 20 of them as the training set, leaving the remaining 10 as the validation set. For both tasks, we conduct 10 real-world tests.Tab. 1 shows the performance of baselines and our method on two tasks. Our MS-Bot outperforms all baselines in both the pouring task and peg insertion task, demonstrating the superiority of stageguided dynamic multi-sensory fusion. Despite using proprioceptive information, the improvement of Du et al. [63] over the concat model is minimal, indicating that the inadequate understanding of the coarse-grained and fine-grained modality importance change is the critical barrier. Moreover, these two baselines exhibit delayed responses to stage changes, causing excessive pouring in the pouring task and misalignment in the insertion task. MULSA employs self-attention for dynamic fusion, but its lack of a thorough stage understanding prevents it from fully leveraging the advantages of dynamic fusion. Our MS-Bot achieves the best performance by better allocating modality weights during fusion based on the understanding of the fine-grained state within the current stage (Q1).The notable performance drop when one modality is removed indicates that each modality contributes significantly. In the pouring task, removing audio and tactile modalities degrades the model's ability to assess the quality of poured beads. The impact is more significant when the audio modality is removed, corresponding to the notable decrease in confidence observed after masking the audio features in Stage 3 as shown in Fig. 1 . In the peg insertion task, removing either the RGB or depth modality will degrade the alignment performance. It is worth noting that even without the assistance of the depth or RGB modality, our MS-Bot achieves an equal success rate compared to concat model and Du et al. [63] , demonstrating the superiority of stage comprehension and dynamic fusion (Q1).To better understand the working principles of MS-Bot and the reasons for performance improvement, we visualize the attention scores of each modality and the scores of the stage tokens. As shown in Fig. 4 , MS-Bot accurately predicts the rapid stage changes. Due to the model's coarse-to-fine task stage comprehension, the aggregated attention scores of the three modalities remain relatively stable, exhibiting clear inter-stage changes and minor intra-stage adjustment (Q2). Vision initially dominates during the first stage. Upon entering the second stage, the model begins to use the sound of beads to find the appropriate angle and distinguishes this stage from the last stage through touch.During the holding still stage, the model primarily relies on audio and tactile deformation to assess the mass of beads. In the final stage, the model discerns the completion of the pouring based on tactile deformation and the rise of tactile attention scores is observed. The changes in attention scores for the peg insertion task (see Fig. 12b ) also follow a similar pattern, demonstrating the validity and explainability of the proposed stage-guided dynamic multi-sensory fusion (Q2).We also conduct ablation experiments on the pouring task to examine the contributions of each module, as shown in Tab. 2. We observe significant contributions to performance from both the state tokenizer and the stage comprehension module (Q2). These two modules provide critical task state understanding and coarse-to-fine stage comprehension. In addition, removing random attention blur results in a minor performance decline. The complete ablation results are located in Tab. 8.To verify the generalization of our method to distractors, we conduct experiments with visual distractors in both tasks. In the pouring task, we changed the cylinder's color from white to red. For the peg insertion task, we altered the base color from black to green ("Color"), and placed clutter around the base ("Mess"). As shown in Tab. 3, our method exhibits minimal impact across various scenarios with distractors and consistently maintains performance superiority, demonstrating the generalizability of stage comprehension (Q1, Q3). Two baseline methods, the concat model and Du et al. [63] , suffer from performance degradation when the visual modality is disturbed. Due to their lack of ability to understand the stages and dynamically adjust modality weights, even when information from visual modality is not needed in a particular stage, the model is still affected by distractors. This leads to unsatisfying performance. Our method dynamically allocates modality weights based on the understanding of the current stage, thereby reducing the impact of visual distractors on the fused features (Q2, Q3). Consequently, it outperforms the two baseline methods using static fusion and the MULSA that solely relies on the current observation for dynamic fusion. 
conclusion and limitations: 
 We present MS-Bot, a stage-guided dynamic multi-sensory fusion method with coarse-to-fine stage comprehension, which dynamically focuses on relevant modalities based on the state within the current stage. Evaluated on the challenging pouring and peg insertion with keyway tasks, our method outperforms previous static and dynamic fusion methods. We find that stage comprehension can be generalized to scenes with various distractors, reducing the impact of interference from one modality on multi-sensory fusion. We hope our work can inspire future work on multi-sensory robots.Limitations: Our stage division involves human factors. An improvement could be leveraging LLMs for automatic stage labeling. Additionally, proposing more challenging multi-sensory manipulation tasks would be interesting for future works. More limitations are discussed in Appendix L. camera cannot observe inside the hole. Hence, we use RGB, depth and touch modalities in this task.Considering generalization, we randomly fix the base at any position along a 12cm-long parallel line on the desktop in demonstrations, as illustrated in Fig. 6 . The robot arm holding the peg can also be initialized inside a 40cm × 20cm rectangular area around the base. During testing, to ensure fairness, the positions of the base and the robot arm are several pre-defined points.Robot Action Space. In this task, the robot arm can move on the three axes x, y, z of Cartesian coordinate, where x, y represents the horizontal movement and z represents the vertical movement.The gripper can also rotate along axis ϕ to align the key with the keyway. Since the vertical movement of the robot arm (-∆z) and the rotation of the gripper (+∆ϕ) are both unidirectional, there are a total of 7 possible actions (±∆x, ±∆y, -∆z, +∆ϕ and 0).Stage Division. This task consists of three stages: 1) First Insertion, where the robot aligns the peg with the hole and inserts it until the key collides with the base, 2) Rotating, where the robot aligns the key with the keyway on the base by rotating the peg based on tactile feedback, and 3) Second Insertion, where the robot further inserts the peg to the bottom. Similar to the pouring task, we consider the timesteps of the first gripper rotation (+∆ϕ) and the first downward movement (-∆z) after rotation as stage transition points. 
 
 Appendix: To verify the generalization of our framework to distractors in the environment, we conduct experiments with visual distractors on both tasks. For the pouring task, we change the color of the cylinder from white to red. As for the peg insertion task, we respectively change the color of the base from black to green, and scatter some clutter around the base. These task settings are illustrated in Fig. 7 .The distractors only exist during testing. Besides introducing distractors, the settings for these tasks remain consistent with the main experiments.Following [11] , we resize visual and tactile frames to 140×105 and randomly crop them to 128×96 during training. We also use color jitter for image augmentation. For audio modality, we resample the wave signal at 16kHZ and generate a 64 × 50 mel-spectrogram through short-time Fourier transform, with 400 FFT windows, hop length of 160, and mel bin of 64. We employ ResNet-18 [64] network as the uni-modal encoder. Each encoder takes a brief history of observations spanning Figure 7 : Illustration of tasks with distractors. For the pouring task, we change the color of the cylinder from white to red (denoted as "Color (Pouring)"). For the peg insertion with keyway task, we respectively change the color of the base from black to green (denoted as "Color (Insertion)"), and scatter some clutter around the base (denoted as "Mess (Insertion)").T = 6 timesteps. We train all the models using Adam [65] with a learning rate of 10 -4 for 75 epochs. We perform linear learning rate decay every two epochs, with a decay factor of 0.9.For action history, we use a buffer of length 200 to store actions. Each action is encoded using onehot encoding. Action sequences shorter than 200 are padded with zeros. For both tasks, we consider the 15 timesteps near the stage transition point as the soft constraint range (γ = 15). We set λ = 5.0 for both tasks to control the intensity of the penalty. For the learnable stage token [stage i ], we initialize it with the mean of all state tokens on samples within the i-th stage after warmup training for 1 epoch. We use β = 0.5 for the calculation of the stage-injected state token z * t . Moreover, to prevent gradients from becoming too large, we also truncated the gradients of the stage score penalty, restricting them to solely influencing the gate network. We randomly replace the attention scores on all feature tokens with the same average value 1 M ×T with a probability p, which is a hyper-parameter.In order to prevent the model from simply memorizing the actions corresponding to attention score patterns, we introduce a random attention blur mechanism to the stage-guided dynamic fusion module. For each input, we replace the attention scores on all feature tokens with the same average value 1 M ×T with a probability p, where M is the number of modalities and T is the timestep number of the brief observation history, as illustrated in Fig. 8 . We set p = 0.25 in both tasks.We introduce this mechanism due to the potential overfitting issue in the model, where the policy head (MLP in the dynamic fusion module) learns the correspondence between the distribution of attention scores and actions. For instance, when using the trained MS-Bot model (without random attention blur) to complete pouring tasks, manually increasing the attention scores on tactile feature tokens invariably leads the model to predict the next action as upward rotation (+∆ϕ), regardless of the current stage of the task. This phenomenon suggests that the stage comprehension module in the model partially assumes the role of action prediction rather than focusing solely on stage understanding. Therefore, randomly blurring attention scores can compel the policy head to focus on the information from the feature tokens and better decouple the stage comprehension module from the dynamic fusion module.Our strategy for stage labeling is to segment the action sequence of the trajectory based on pre-set rules (such as considering the timestep of the first gripper rotation as a stage transition point). This does indeed require some time to pre-define the stages and design specific rules for division. However, extracting stage information without human annotation is also feasible. One possible approach is to use LLMs, which have demonstrated strong comprehension ability in robotic manipulation [66] , for automatic stage annotation of action sequences. We use Claude 3.5 Sonnet [67] as our LLM, and construct prompts to describe the task setup and include an example action sequence to better help the LLM understand the process of this task. We then ask the LLM to provide the specific stage division, as shown in Fig. 9 .We want the robotic arm to complete a peg insertion with keyway task. In this task, the robotic arm needs to align the peg with the keyway on the base by rotation, and then fully insert the peg. At each moment, the robotic arm chooses one action to execute from seven options: move left, move right, move forward, move backward, move down, rotate the peg, and stay.Here is an example of the complete action sequence: [left, left, left, left, left, left, backward, backward, backward, backward, backward, backward, left, left, left, left, down, down, down, down, down, down, down, down, left, down, down, down, down, down, down, rotate, rotate, rotate, rotate, down, down, down, down, stay, stay] How many main stages are there in this task? Please divide the action sequence into rough stages based on the sub-goals of the task. Please do not list fine adjustments within a main stage as separate stages.To analyze this action sequence and divide it into stages, let's look at the major movements and goals. I'll group similar consecutive actions together and identify the main stages based on the apparent sub-goals of the task. Here's the breakdown: 1.Horizontal Alignment (Left movement): The sequence starts with a long series of "left" movements (95 times). 2.Depth Adjustment (Backward movement): Following the left movements, there's a significant series of "backward" movements (43 times). 3.Fine Horizontal Adjustment: A small series of "left" movements (9 times) follows. This is likely a fine adjustment, so we'll consider it part of the previous stage rather than a separate one. 4.Vertical Approach (Down movement): There's a long series of "down" movements (159 times), bringing the peg closer to the insertion point. 5.Rotation Alignment: A significant number of "rotate" actions (146 times) occur, likely to align the peg with the keyway. 6.Final Insertion: The sequence ends with a series of "down" movements (64 times), followed by several "stay" commands (13 times). Based on this analysis, we can identify 4 main stages in this task: 1.Initial Positioning: Horizontal alignment and depth adjustment (includes the first two major movement series and the small fine adjustment) 2.Vertical Approach: Bringing the peg down towards the insertion point 3.Rotation Alignment: Aligning the peg with the keyway 4.Final Insertion: Inserting the peg and maintaining position These stages correspond to the main sub-goals of the peg insertion with keyway task: positioning the arm above the insertion point, approaching the insertion point, aligning the peg with the keyway, and finally inserting the peg. User LLM Figure 9 : An example of automatic stage division by LLM. We provide the task setup, robot action space and an example action sequence to the LLM to obtain the main stage division. For ease of presentation, the action sequence in the figure has been abbreviated.After obtaining the stage divisions provided by the LLM, we then construct another prompt and ask the LLM to label the action sequence according to the stage divisions, as shown in Fig. 10 . This strategy allows for obtaining stage labels for all samples without requiring human annotation. We find that the stage divisions provided by the LLM are usually more detailed. For example, it divides the peg insertion with keyway task into four stages (splitting our First Insertion stage into two stages: Initial Positioning and Vertical Approach), which is also reasonable. Since our model does not explicitly require that the modality importance must be different between stages, a more detailed stage division is acceptable to some extent.Through this method, we perform automatic stage annotation on our imitation learning dataset. To evaluate the accuracy of the LLM's annotations, we merge the first two stages divided by the LLM into a single First Insertion stage, and use the human-annotated stage labels as ground truth. The evaluation results show that the accuracy of the stage labels annotated by the LLM is 95.68% (10,096 / 10,552, about 15 errors per trajectory).For tasks like peg insertion with keyway, where many actions are continuous, we can merge consecutive actions to obtain a compressed action sequence like "[action*num,...,action*num]", as shown in Fig. 11 . This approach helps reduce annotation errors caused by LLM counting mistakes and increases the accuracy to 99.65% (10,521 / 10,552, about 3 errors per trajectory). These results demonstrate that automatic stage annotation using LLMs without pre-defined stages is feasible. The experiment is only a relatively simple attempt. We believe that with more detailed prompt engineering and the use of more powerful LLMs, the accuracy can be further improved.In Sec. 4.4 of the main paper, we illustrate the aggregated attention scores for each modality and the stage scores of MS-Bot in the pouring task, as shown in Fig. 12a . We also record the changes in attention scores and stage scores in the peg insertion with keyway task, as shown in Fig. 12b . The results in the figure demonstrate that our model accurately predicts the rapid changes in stages for both tasks. As a result, the attention scores across modalities guided by stage comprehension are also relatively stable, exhibiting clear inter-stage changes and minor intra-stage adjustment.As a comparison, we also visualize the attention scores of another baseline MULSA [11] with selfattention fusion, as shown in Fig. 13 . The range of the attention score axis in the figure is consistent with Fig. 12 (0.0 ∼ 1.0). It is evident that the attention scores of the modalities in the MULSA model are close and exhibit a small variation along the time axis, lacking clear stage characteristics. This indicates that the MULSA model fails to fully leverage the advantages of dynamic fusion compared to the concat model. To verify the applicability of our proposed stage-guided dynamic multi-sensory fusion to different policy networks, we replace the policy network with the Diffusion Policy [68] and conduct experiments on the peg insertion with keyway task. Specifically, we replace the MLP policy network with an UNet-based Diffusion Policy in the Concat baseline and our MS-Bot model, and evaluate the performance of stage-guided dynamic fusion on models using different policy networks.We want the robotic arm to complete a peg insertion with keyway task. In this task, the robotic arm needs to align the peg with the keyway on the base by rotation, and then fully insert the peg. At each moment, the robotic arm chooses one action to execute from seven options: move left, move right, move forward, move backward, move down, rotate the peg, and stay. This task contains 4 main stages: Initial Positioning: Horizontal alignment and depth adjustment (includes the first two major movement series and the small fine adjustment) Vertical Approach: Bringing the peg down towards the insertion point Rotation Alignment: Aligning the peg with the keyway Final Insertion: Inserting the peg and maintaining position Please label the stage number to which each action in the action sequence belongs in order, and return a list like: , and the last few elements are [4, 4, # Full list of stage labels print(stage_labels) Figure 10 : An example of automatic stage labeling by LLM. We provide the task setup, robot action space and an target action sequence to the LLM to obtain the stage labels. For ease of presentation, the action sequence in the figure has been abbreviated.The results in Tab. 4 show that replacing the policy network in the Concat baseline with the Diffusion Policy leads to a significant performance improvement, but there is still a gap compared to MS-Bot. This is because the Diffusion Policy still lacks an explicit feature fusion module (apart from simple concatenation). The fusion of multi-sensory features in this baseline still occurs within the policy network, similar to the concat baseline, albeit with a more capable policy network. After further replacing its fusion module with our proposed stage-guided dynamic fusion, it achieves performance To label the stages for each action in the given sequence, let's break it down based on the description of the four main stages:1. Initial Positioning: This includes horizontal alignment and depth adjustment. In the given sequence, this corresponds to the initial left and backward movements. To create the list of stage labels, we need to expand each action based on its repetition count.Here's the resulting list: This gives us a list of 528 elements, matching the length of the action sequence. The list starts with 149 1's (for stage 1), followed by 158 2's (for stage 2), then 146 3's (for stage 3), and finally 75 4's (for stage 4).User LLM Figure 11 : An example of automatic stage labeling by LLM. We provide the task setup, robot action space and an compressed target action sequence to the LLM to obtain the stage labels.comparable to MS-Bot. This demonstrates that our proposed explicit stage understanding and stageguided dynamic fusion are still effective for models with more powerful policy networks. Our proposed fusion module can help the policy network focus more on generating actions using the already fused multi-sensor features. However, under the same conditions of using stage-guided dynamic fusion, the model with diffusion policy as the policy network does not outperform the model with MLP. We think this may be due to the relatively small scale of data in this task, which does not allow the Diffusion Policy to fully leverage its advantages.Due to not fully considering the importance of explicit stage understanding in guiding multi-sensory fusion, the original Concat and MULSA baselines does not utilize contextual inputs and additional stage information. To make a fairer comparison with our MS-Bot, we respectively construct Concat and MULSA baselines that incorporate a stage prediction auxiliary task and multi-sensory observation context. Specifically, we add an extra head (an MLP identical to the gate network in MS-Bot) to both the Concat and MULSA models to predict the stage, trained with the same form of loss used to penalize the scores of the gate network in MS-Bot. We also attempt to modify the structure of the MULSA model to accept observation context as input. We input the observations from 5s, 7.5s, 10s, 12.5s, 15s, 17.5s, and 20s before the current timestep into the model, and encode these observation contexts by each modality's encoder into 7 tokens (per modality). These tokens are then concatenated with the original 6 tokens of the current timestep observation of each modality, resulting in a total of 13 tokens for each modality. We evaluate these baselines on the peg insertion with keyway task. The results in Tab. 5 shows that introducing an explicit stage prediction task could improve the performance of the Concat baseline, but it provides minimal benefit for the MULSA method. These baselines still have a performance gap compared to our MS-Bot. This indicates that explicit stage understanding is meaningful, but introducing context to aid stage understanding and specially designed stage-guided dynamic fusion modules are also necessary. In addition, the results show that inputting observation contexts into MULSA results in virtually no improvement in the model's performance. This indicates that mixing the contexts and current observations in self-attention calculations is highly inefficient for helping the model understand stages and allocate modality weights, as it reduces focus on the more important current observations and significantly increases GPU memory usage. In contrast, our MS-Bot leverages a long action history as context to enhance its understanding of task stages. This approach simplifies processing for the model and offers finer granularity, effectively guiding dynamic multi-sensor fusion.In robotic manipulation, a common setup involves using multiple cameras to capture visual information from different views. By treating visual images from different views as distinct modalities, our MS-Bot is also able to handle multi-camera observations. To verify this capability, we conduct a comparison with baseline methods on the peg insert with keyway task under the setting with multiple cameras. Specifically, we place another camera on the side of the base, so that the lines connecting these two cameras to the base roughly form a right angle. We replace the depth observation from the first camera with the RGB observation from the second camera, as the RGB observations For tasks involving using lock-picking tools to unlock in real life, audio can indeed serve as the basis for completing the task, as these locks typically have specific mechanical structures that produce distinct sounds. Similarly, audio can indicate whether the key and keyway are aligned, thereby assisting with the peg insertion with keyway task. To verify the effectiveness of our model after further incorporating audio, we conduct experiments on the peg insertion with keyway task using RGB, depth, tactile and audio modality. The results in Tab. 7 show that the addition of the audio modality only brings improvement to the weakest Concat baseline. For MULSA and our MS-Bot method, the benefits are minimal. This is because the audio in this task is short and minimal due to the 3D-printed materials and is vulnerable to noise disturbance.In Sec. 4.4 and 4.5 of the main paper, we show the overall error on all settings of the pouring task.In this section, we comprehensively present the detailed result of the component ablation and the distractor experiment on all the settings of the pouring task in Tab. 8 and 9. As shown in Tab. 8, the contributions from both the state tokenizer and the stage comprehension module on all settings demonstrate the importance of the coarse-to-fine stage comprehension. The consistent lead of MS-of stage comprehension within MS-Bot. We also find that setting too large p (p > 0.3 in Fig. 14c ) can bring negative impacts as the attention blur truncates the gradients backpropagated to the stage comprehension module and the state tokenizer. Excessive attention blur can impair the training of these two modules.In this section, we discuss the potential limitations of our work and the corresponding solutions:In many robotic tasks this kind of dynamic weighting is not necessary. Indeed, there are many tasks that can be completed with just the visual modality, where readings from other sensors may not provide useful information. We believe that one solution is to let the model first select the necessary subset of sensors based on the task (possibly using an LLM), and then perform dynamic multisensor fusion based on stage understanding. For tasks with only one stage, this can be considered as a special case where |S| = 1.The stage labeling may be costly. Currently, our strategy for stage labeling is to segment the action sequence of the trajectory based on pre-set rules. This requires time to set up detailed rules and becomes more challenging when there are more stages and more complex stage transitions. A possible solution is the LLM-based automatic labeling method mentioned earlier, although a small amount of human effort is still necessary, such as constructing prompts to accurately describe the task setup and providing example for in-context learning.The lack of comparison to larger models. Currently, there is no well-trained open-source multisensory large foundation model for robot manipulation (including vision, hearing, touch, etc.). Therefore, we are temporarily unable to verify the role of explicit stage understanding in large models, nor can we directly compare our model with multi-sensory large models. However, we believe that even for larger models, explicit stage understanding remains meaningful when introducing new modalities and training is required using a small amount of paired multi-sensory data, or when fine-tuning in new scenarios with limited data scale.Robustness issue in the case of missing modalities. Since in both of our tasks there are stages that depend on a single modality, masking uni-modal features during testing may directly lead to failure. This is a problem common to all multi-sensory models, and we did not make improvements specifically for this issue. In cases where multiple sensors complement each other, the absence of uni-modal features may have a compounding effect on stage understanding and dynamic fusion. We believe that one possible solution is to randomly dropout uni-modal features during the training process [69] . Some other methods for robust multi-modal learning could also be referenced.Transferability of the current dynamic multi-sensory fusion architecture. Since the multisensory fusion architecture we use in the dynamic fusion module is based on cross-attention rather than self-attention, there are difficulties in directly transferring the original form of this fusion method to transformers with multiple self-attention blocks. One potential strategy is to use the stage token as a special token to replace the class token. It could be used to predict the stage under the supervision of stage labels and thereby achieving the effect of stage-guided multi-modal fusion.Low integration level of current multi-sensory robot system. Our current multi-sensory robot system has not well integrated the various sensors: the camera is positioned at a third-person perspective, the microphone is placed on the desktop, and the tactile sensor is at the end of the robotic arm. They are located in different positions. This low level of integration means that the system can only complete relatively simple tasks, while more complex multi-sensory tasks that require mobility cannot be accomplished. This multi-sensory system still needs further refinement and upgrading. 


Decision: Strengths:
- Clear motivation and presentation;
- Innovative idea;
- Real-world results.

Weaknesses:
- The idea may not be timeless as larger models evolve;
- The stages need to be pre-defined;
- Some baseline comparisons may be included.

Post rebuttal: The authors did an excellent job during the rebuttal, after which all three reviewers recommended acceptance, including two strong accepts. The AC agrees with the recommendation. The authors are encouraged to revise the submission to clearly acknowledge the limitations, such as pre-defined stages, in the camera-ready.

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper presents an imitation learning method that learns to fuse features from different modalities using additional information about the stage within the task. In particular, action history and current features are used to extract current state and stage, and this is information is used to weight the multimodal features, and weighted features are used for predicting actions. Experiments show that the proposed method outperforms baselines that don't explicitly consider such stage information.
main_review: main_review: Strengths
- Interesting idea to incorporate stage information for mixing the information from multi-modal sensors
- Ablation studies that show the importance of each component
- Writing is mostly clear and helps understanding the main idea
- Real-robot experiments

Weaknesses
- The proposed idea heavily depends on the additional information about the stage within the episode. This could be very difficult to obtain in practice. Would there be any alternative way of extracting such stage information from the data without human annotation? Any preliminary experiments can strengthen the claims made in the paper.
- Baselines don't have an access to additional stage information, which makes it difficult to think that the results are from fair comparison. For instance, very naive baseline that incorporates stage information by having an auxiliary objective that predicts the stage can be a useful baseline to have.
- One of missing baselines is that training DiffusionPolicy with multimodal features, as its transformer network can in principle learn to focus on useful features via self-attention. And investigating what happens when we use more expressive policy network with multi-modal inputs can be helpful for strengthening the claims made in the paper.
- One missing experimental setup that could be important is how it handles features from multiple cameras -- which is more of a common setup than using audio or tactile sensors.
- It could be nice to add more results that show how the proposed method is sensitive to the hyperparameters.
- More formal definition or decsription of states and stages could be provided in the initial part of Method section,  with clear description on how a stage label is added in line 131.
- What does 'stage after warmup' mean in line 147?
- Motivative experiments in Introduction needs to be more self-contained as there is no explanation about what confidence means. and how it is computed.
questions: questions: - See Review
limitations: limitations: 2
review_summary: review_summary: I enjoyed reading the paper but the proposed idea is a bit limited in that it heavily depends on expensive stage annotation, and the experiments are missing some baselines and analysis that could further strengthen the claims made in the paper.  I'm currently in a borderline but slightly leaning towards weak reject. But I'm willing to change my score after reading other reviewers' opinions and the rebuttal response.
title: title: MDMP: Multimodal Diffusion for Motion Prediction with uncertainty in Human-Robot Collaboration
paper_summary: paper_summary: The paper explores the how to adaptively weigh individual modalities ("modality importance") of a multi-sensory robot when handling mutli-stage tasks within the context of complex sub-goals that require different levels of focus on given modalities. The core intuition is that sensory fusion needs to change for the given sub-task to be the most effective. They build a multi-sensory robot and use imitation learning to train a MULSA model, they additionally add a method to identify task stage and use this classification to adjust modality weights. The proposed method (MS-Bot) is applied to a pouring and a peg insertion with keyway task.
main_review: main_review: The quality of this work is excellent, beyond the clarity of writing, the authors posit the interesting challange and an intuitive, well thought through solution. However I do have reservations about the performance benefit of the presented system in comparison to large multi-modal models, I would expect the blending, tuning and focus of individual modalities to be equally present in an end-2-end solution. Without this kind of comparison it is difficult to see if the method proposed beats state-of-the-art multi-modal models because it is actually essential in multi-modal robot systems to explicitly calculate stage state and weigh the multi-modal fusion based on this or if its because the current state-of-the-art methods (Du et al. and MULSA) are quite small models and require this explicit stage comprehension but larger models can handle it internally. Maybe there are reasons for the proposed approach from the perspective of demonstration dataset size conditions that can keep the impact of this work high. One method may be to compare this work against multi-modal versions of robot foundation models like Octo and OpenVLA, I understand this maybe a large effort/engineering challange to get those working, but it will validate the "stage comprehension" and "state tokenizer" aspects of this work which are the key novelties. The significance of the work is a little hard to picture because of the point raised above, however I think the research domain carved out in the introduction and related works is wide enough that this method and the challange presented in this paper of active modality weighting may become important in the future of robotics and I am excited to see where this research goes in the future, so for these reasons I think this work could have a strong impact. One barrier to high impact may be the requirements to label datasets with sub-task classes, to my knowledge the majority if not all of the common large robotics datasets do not contain this feature, and so larger implementations of this work may require significant efforts to enable large scale training.

The introduction is very well written, simple to understand and the authors present the ideas and intuition behind their work very well. The work is motiveated cleanly and the contributions are explained concretely. The related works is written equally well and covers enough ground to give the reader enough context to understand the work. This clarity continues throught the work and it was a joy to read (well done authors!)

List of Strengths:
- motivation for the work is clear and concise, with up to date referencing and context.
- They propose the interesting idea of "Modality Temporality", the challange of switching and focusing on different modalities for different robotics tasks.
- Contributions are clearly and concisely stated.
- The method is novel, intuitive and presents a nice pocket (active modality weighting) for research in the future.
- The method description is extremel well written, its clear, simple, the Figure 2 which describes the system coveres nearly all aspects of the method and pairs very well with the Section 3.2. 
- The choices behind the proposed systems design is backed up throughout the methodology section and is easily understandable to the reader.
- The experimental section clearly states the key questions they try to answer about their proposed system. The questions are to the point and focused on the core questions I thibk readers will have about the proposed system.
- Experiments are made in the real-world with real hardware and applied to classic robotics benchmarks (peg in hole and pouring) and the visual distraction tests are a nice way to show benefit of the proposed method in comparison to SOTA multi-modal appraoches.
- The video was clear and concise. Speach overlay was really appreciated and made the video much more understandable. It may be worth editing so that the live weight during the pouring task is shown in a more readable way.
 
Weaknesses:
- As stated above, the issues come down to comparison with larger models that may make this kind of explicit sub-task comprehension redundant.
- limitations of the work are lacking, I think a more comprihensive analysis of the work would really help readers understand where this work is currently failing and can be improved. Some limitations to address maybe that in many robotics tasks this kind of weighting is not necessary, the cost of labelling demonstrations with sub-task class, comparison to larger models, how this trained system may lack the robustness of an integrated system when dealing with uni-modality occlusions.

Overall I think the paper is of high quality and well suited to CoRL. The authors present a novel task (Modality Temporality) and a strong performing solution to this task, experiments show both the reason this presented task are important and the strengths of their proposed solution. Given the intuition and results presented, this task may become an area of interesting research in the future. The work lacks comparison to a model large enough to potentially perform sensor modality weighting in an end-2-end manner that may make the task and therefor solution redundant. However, that being said I think that comparison would likely need to be a seperate paper due to it's scale. A great paper and a joy to read.
questions: questions: Section 1:
- Line 23: need to cite or show that optimus and figure 1 are multi-sensory robots.
- Line 43-46: Is the general sentiment of this sequence correct? Do the later layers in sensory fusion methods not weigh the given sensor features naturally? I would expect this to be the case within the context of an end-to-end system. I am open to discussion on this but my intuition was that end-to-end systems would learn to weigh individual modalities during the course of a multi-stage task, expecially if enough observational context is given to understand the task stage (it would have to understand the task stage to complete the multi-stage tasks), some logic behind your call for explicit stage understanding within the context of sensor modality weighting, in comparison to an end-2-end approach with enough context to know what stage it is in, would help motivate this research to the reader.

Section 2:
- Clear concise and well written, giving the reader enough context to understand the proposed solutions and research space. I have no comments that need addressing for this section. 

Section 3:
3.1:
- Line 106-107: What is the reasoning behind not using previous sensor readings in this method, is it to keep the model simple or is there a more complex reason behind this decision? A little sentence here would be useful for readers to understand the design decisions behind the model.
- Line 112-121: Again this is very well written and makes perfect sense to me, I once again wonder if your intuition is correct. You state "could" so there is nothing wrong with this paragraph at all, however it "could" be improved if you had some backup evidence for your intuition, the results show that this method out performs MULSA which is end-to-end so you can use this to clarify to the reader that this intuition is currently correct, but given the lack of context in MULSA it is difficult to explore if this weighting would occure with multi-modal robot foundation models which take a context sequence of observations as input.

3.2: 
- You could add z^{fus}_{t} to Figure 2 so that the figure better aligns with the Dynamic fusion description (Line: 171,172)

Section 4:
- Implementation of the methods details would be good to know, barebones essential parameters for the architecture and the training. Especially if there are querks to getting this system trained well.
- Would be good to get more details on the 1st baseline ( the concat model which directly concatenates all the uni-modal features) - is it literally just this or are there other aspects to this?
- I would like to see a comparison with a larger end-to-end solution that may make explicit tuning of modality weighting redundant, specific focus on the dynamic modality focus would be a great way to motivate the necessity to solve and explore the presented "Modality Temporality" challange and invite research in this area, helping produce a larger impact for this work. Although I understand this maybe a large undertaking and I do not think it is essential for this paper to be accepted. I think the best version of this paper would show that large models will still require this explicit sub-task modality weighting.
- I would like to see a comparison of attention scores between the MULSA method and the method proposed in this work. That would provide insight into the level of impact the explict stage-based dynamic fusion is having.

Section 5:
- Limitations of the work are not discussed at a high enough standard, I think a more comprehensive analysis of the works limitations would really help readers understand where this work is currently failing and can be improved. Some limitations to address maybe that in many robotics tasks this kind of weighting is not necessary, the cost of labelling demonstrations with sub-task class, comparison to larger models, how this trained system may lack the robustness of an integrated system when dealing with uni-modality occlusions. I'm open to discussion on whether any of these are in fact limitations. Likewise I'm sure there are more to add.
limitations: limitations: 1
review_summary: review_summary: Overall I think the paper is of high enough quality for CoRL. The authors present a novel task (Modality Temporality) and a strong performing solution to this task, experiments show both the reason this presented task are important and the strengths of their proposed solution. Given the intuition and results presented, this task may become an area of interesting research in the future. A great paper and a joy to read.
title: title: stage-integrated multisensory model; challenging manipulation tasks; ablation studies by masking different modalities
paper_summary: paper_summary: The authors propose MS-Bot, a novel stage-guided multi-sensory fusion method with coarse-to-fine stage comprehension. Stage modules are designed to encode course-grained and fine-grained states. Authors demonstrate performance increase in pouring and insertion tasks.
main_review: main_review: The paper addresses a challenge in robotic manipulation with multisensory information. They introduce their ideas by presenting analysis of low confidence in action predictions during stage changing. Then, they propose a new method that take stage into consideration. The experiments are well-designed and analysis of the results are complete. The idea of their method is novel and original, and their paper is well-structured. The concept is quite inspiring and has is significant in multisensory robot learning.

Strength:
- An intuitive analysis on confidence in action prediction by masking out different modalities
- Propose a novel method for multisensory fusion with stage comprehension
- Challenging tasks design
- A comprehensive analysis of the experiment results to demonstrate performance improvement and significance of the stage module.

Weakness:  
- The stages are pre-defined, so as for the stage feature dimensions. It might be not very ideal in real-world scenarios, but I can imagine that LLMs can be used to help label the stages.
questions: questions: - Table I pouring results is a bit confusing, which of initial weight corresponds to which target weight?
- For the key task, why do you decide not use audio in this setup? I can imagine that when opening a door lock, the sound actually help me get a sense if my key is correctly inserted. Also, people uses audio feedback when they use those lock picking tools to unlock doors.
- Could you elaborate more on how you will handle long-horizon tasks with more stages?
limitations: limitations: 2
review_summary: review_summary: The paper presents a novel method and complete analysis, and is clear to read. I recommend accept this paper.