$\alpha$-MDF: An Attention-based Multimodal Differentiable Filter for Robot State Estimation
Authors: Xiao Liu
Yifan Zhou
Shuhei Ikemoto
Heni Ben Amor
Venue: CoRL 2023 Poster (2023)
Keywords: Differentiable Filters
Sensor Fusion
Multimodal Learning

Abstract: Differentiable Filters are recursive Bayesian estimators that derive the state transition and measurement models from data alone. Their data-driven nature eschews the need for explicit analytical models, while remaining algorithmic components of the filtering process intact. As a result, the gain mechanism -- a critical component of the filtering process -- remains non-differentiable and cannot be adjusted to the specific nature of the task or context. In this paper, we propose an attention-based Multimodal Differentiable Filter ($\alpha$-MDF) which utilizes modern attention mechanisms to learn multimodal latent representations. Unlike previous differentiable filter frameworks, $\alpha$-MDF substitutes the traditional gain, e.g., the Kalman gain, with a neural attention mechanism. The approach generates specialized, context-dependent gains that can effectively combine multiple input modalities and observed variables. We validate $\alpha$-MDF on a diverse set of robot state estimation tasks in real world and simulation. Our results show $\alpha$-MDF achieves significant reductions in state estimation errors, demonstrating nearly 4-fold improvements compared to state-of-the-art sensor fusion strategies for rigid body robots. Additionally, the $\alpha$-MDF consistently outperforms differentiable filter baselines by up to 45% in soft robotics tasks. The project  is available at alpha-mdf.github.io and the codebase is at github.com/ir-lab/alpha-MDF

a-MDF: An Attention-based Multimodal Differentiable Filter for Robot State Estimation
Xiao Liu', Yifan Zhou', Shuhei Ikemoto", and Heni Ben Amor
Interactive Robotics Lab, Arizona State University
Kyuuhu Institute of Technology
[x1iu33) ywwouegghhooooo h hbenghorraaaaaaaaaaaaa edu ".kemoto@brain wwwpschhiinkttcchhc.
Rigid Body Dynamics
Soft Robot Dynamics
"Pick"
"Push"
"Put down"
Depth
Joints
Force/ Torque
RGB
Depth
IMUs
Q-MDF
P d0b
C-MDF
Q State
999 Cx-MD
Cc-MF
QAP
Q-MDF
Q-MDF
State
680
DOD
State
6
State
d00
The attention-basee Multimodal Differentiable Filter (a-MDF) framework enables robot state esti- mation in multimodal scttings, applicable both rigid body robots and soft robots.
Figure 1:
Differentiable Filters are recursive Bayesian estimators that derive the state transition and measurement models from data alone. Their data-driven na- ture eschews the need for explicit analytical models, while remaining algorith- mic components the filtering process intact. As result, the gain mecha- nism critical component of the filtering process remains con-differentiable and cannot be adjusted the specific nature of the task context. In this pa- per, propose attention-baun based Multimodal Differentiable Filter (a-MDF) which utilizes modern attention mechanisms to learn multimodal latent repre- sentations. Unlike previous differentiable filter frameworks, a-MDF substitutes the traditional gain, e.g., the Kalman gain, with a neural attention mechanism. The approach generates specialized, context-dependent gains that can effectively validate a-MDF
Abstract:
combine multiple input modalities and observed variables. on diverse set robot state estimation tasks real world and simulation. Our results show a-MDF achieves significant reductions in state estimation rors, demonstrating nearly 4-fold improvements compared to state-of-the-art sen- sor fusion strategies for rigid body robots. Additionally, the a-MDF consis- tently outperforms differentiable filter basclines by up to 45% in soft robotics tasks. The project available at alpha-mdf. siiiaamdf.gggghbbbb....a io and the codebase at gi github. cow/ir-lmaa aar--abballhaaaae
Keywords: Differentiable Filters, Sensor Fusion, Multimodal Learning.
1 Introduction
Recursive Bayesian filters, in particular Kalman filters, are core component of many robotic and autonomous systems [1]. These filters offer probabilistic framework that enables effective state es- timation and allowing robots to perceive and respond to dynamic environmental conditions [2, 3, 4].
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Constructing the analytical models and characterizing their noise profiles can be an overwhelm- ing undertaking and requires supplementary measures, e.g., system identification [5]. In addition, scalability still poses a significant obstacle, particularly when dealing with non-linear and high- dimensional systems. Advanced techniques such as the ensemble Kalman filter [6] have been de- veloped to tackle this challenge. However, they may still require careful manual design of the data pipeline and filtering process, especially in the presence of multimodal data sources. A potential alternative methodology is to derive the underlying models for filtering from data alone. Recent ad- vancements in Deep state-space models (DSSMs) [7] provide effective solutions for understanding the state and measurement estimation from observed sequences as data-driven approaches [7, 8, 9]. Such approaches do not need to derive explicit system dynamics, which is essential and challenging in traditional filtering techniques. subclass of algorithms derived from DSSMs, called Differ- entiable Filters (DFs), focus on learning state transition and measurement models from data while retaining the fundamental principles of Bayesian recursive filtering. This combination of proper- ties renders DFs particularly well-suited for systems with complex dynamics and diverse sensor observations.
In this paper, we introduce a novel class of differentiable filters built upon neural attention mecha- nisms. The key innovation lies in the substitution of the traditional Kalman gain with an attention mechanism for filtering with multimodal observations. This approach allows for the learning of a highly specialized and task-specific gain mechanism. The utilization of multimodal observations, also known as multimodal learning [10], has shown substantial advantages in various robotics appli- cations [11, 12, 13]. By harnessing information from diverse modalities such as vision [14, 15, 16), language [17, 18, 19, 20], and tactile sensing [21, 22], robots can learn to better interpret their sur- roundings, produce more accurate estimates of their own internal state, and consequently improve the overall decision-making process. We propose the attention ters (a-MDF) framework, as shown in Fig. each module is learnable and operates in latent space. The primary contributions are: (1) Attention Gain: Our approach is an attention-based strategy that replaces the conventional Kalman gain in the measurement update step, depicted by the colored blocks in Fig. 2. The gain mechanism is learned to update the current state based on multimodal observations. (2) Latent Space Filtering: our proposed differentiable framework operates in a la- tent space, learning nigh-dimensional representations of system dynamics and capturing intricate nonlinear relationships. This approach proves particularly advantageous for highly nonlinear sys- tems. (3) Empirical evaluations: o-MDF achieves significant reductions in state estimation errors, demonstrating nearly 4-fold improvements compared to state-of-the-ar sensor fusion strategies in multimodal manipulation tasks. Furthermore, a-MDF accurately models the non-linear dynamics of soft robots, consistently surpassing differentiable filter baselines by up to 45%.
Multimodal Differentiable Fil-
2 Related Work
Differentiable Filters (DFs) have garnered attention as learnable non-linear state-space mod- els [23, 24, 25J. Previous works [26, 27] have integrated neural network components into robotic algorithms, such as BackpropKF, which combines backpropagation with neural networks to train Kalman Filters. Similarly, research in Differentiable Particle Filters (DPFs) [28, 29] has also lever- aged learnable modules to address the challenges of filtering and state tracking. Algorithmic priors have been utilized improve the learning efficiency of DPFs. and adversarial methods have been used for posterior estimation [30]. However, the gain mechanism in the traditional Kalman filter is not differentiable and has not been incorporated into the learning procedure of the DFs mentioned earlier. In recent study [9], the effectiveness of DFs in training and modeling uncertainty with noise profiles has been demonstrated. Typically, multi-layer perceptrons integrated with an RNN layer are employed in the implementation of DFs. Performance on real-world tasks has shown considerable improvement in state tracking accuracy [9, 22, 24, 31, 32], and results indicate that the adoption of end-to-end learning is crucial for accurately learning noise models. However, as noted [8], the use of RNN has been shown to be "a limiting factor for learning accurate models" and may "lead to mon-Markovian state-space". Furthermore, the traditional Kalman gain for DFs in [9, 22, 24] remains non-learnable, despite the numerous advancements made thus far in differentiable filters

---

RGB
Depth
Joints
roude
+
V X
Transformer Process Model fo
Sensor Encoders
Attention Gain
K
state X
K
acant deservation
lotent obsenattin Y
stent obsenation Y
Attention map
Decodar
state attention
observation attention
Státe
Figure 2:
The a-MDF framework consists of transformer process model, encoders, and attention gain update model. transformer process model predicts the following latent state, while the sensor encoders learn latent representations from observations. The attention gain model then corrects the predicted latent state using the learned representations.
and particle filters. While DFs have demonstrated considerable promise as differentiable sequen- tial models, their application in multimodal settings has thus far received limited attention. One notable exception is the recent work in [22], which proposes sensor fusion strategies associated with DFs. Employing multimodal environments often necessitates the adoption of representation learning methodologies [33], which require learning latent representation for capturing intricate static and dynamic features [8, 12, 34). Studies such as [11] have highlighted the effectiveness of blending networks in learning shared representation based Conditional Neural Processes [35]. Other studies, including [21, 12], have adopted self-supervised approach involving variational autoencoders (VAE) [36] to discover latent representations for stable manipulation policies. The majority of multimodal latent representations are obtained through policy learning. However, in certain situations, gating technique [37] that focuses on specific modalities can employed to en- hance policy robustness. light of this, we propose an alternative approach which utilizes modern attention mechanisms learn multimodal latent representations both the level of modalities and observations. The novel approach fuses multiple types of modalities and variables in dynamic, context-dependent fashion, thereby enabling synergies between their respective qualities, structures, relevance, and degrees of redundancy.
Multimodal Differentiable Filters
We introduce novel approach called attention-based Multimodal Differentiable Filters (a-MDF), which combines differentiable filters with insights from transformer models. This approach per- forms state estimation and fusion of multiple sensor modalities in unified and differentiable man- ner. We first discuss Recursive Bayesian filtering as the general technique used to estimate the state X, of a discrete-time dynamical system. Thereafter, we provide the details of our specific algorithm. Given a sequence actions a and noisy observations yıa, the posterior distribution the state can be represented by the following equation:
p(xe|a1:t> 1:t, x1:+-1) ply: |ar, X+) p(x: (1xa|anz-11 11a-1, X1:t-1)-
We can denote the belief of the state bel(x;) p(x//ara: VI... x1=--)). Assuming the Markov property, where the next state dependent only on the current state, we get the following expression:
state transition model
bel(x.) p(y: Xx)
Il pxxx xt bel(x:-1),
observation model
where y is normalization factor, p(y: xx) is the observation model and p(x:la,, x;-1) the tran- sition model. The transition model describes the laws that govern the evolution of the system state, while the observation model identifies the relationship between the hidden, internal state of the sys- tem and observed, noisy measurements.

---

3.1 Q-MDF
We utilize an ensemble method for Bayesian filtering wherein each ensemble member represents a compact robot state. Figure 2 shows the procedural steps of how this compact representation, known as the latent state, obtained and get updated. The filtering process includes two essential steps, namely prediction and update, both of which are also implemented through neural networks. Most importantly, we replace the Kalman gain step with attention mechanism, which is trained to weigh observations against predictions based on the current context. Additionally, we demonstrate that attention can be used to balance and weigh different modalities, e.g., video, depth, inertial measurements, against each other. We will see that both steps can be naturally integrated into single attention gain (AG) module.
Let Xo.N denote the latent states with dimension d2 of N steps in with number of E ensemble members, Xo.N -_. N.. where F. E Z+.
....
Prediction step: In this step, the state transition model takes the previous states with the current ac- tion, and predicts the next state. To this end, we leverage the capabilities transformer-style neural networks [38]. In addition, we generate probability distribution over the posterior by implement- ing the state transition model as a stochastic neural network. Therefore, we can use the following prediction step update each ensemble member, given sequence of latent states N
x x ~ [ x/, N(- Vi E.
(3)
Where fo is a transformer-style neural network with multiple attention layers. In our framework, the latent state and the action at l are processed by positional and type embedding layers [38] prior to being fed into fo. Matrix - E x holds the updated ensemble members which are propagated one step forward in latent space. For simplicity, we represent xjtt-Ntt11 as X, denote the predicted state. Further elaboration positional embeddings, type embeddings, and filter initialization be found in Appendix A. for more comprehensive details.
Update step: A crucial step of the filter- ing process is the update step, which in- volves calculating the gain value. Tradi- tional KF uses the Kalman gain to correct the state by comparing the uncertainty covariance obtained from state space and observation space, it requires an explicit function to map the state to the measure- ment. As a result, some sensor measure- ments like images or deep-learned features are unable be used in the formulation di- rectly. The proposed attention gain (AG) module, the other hand, eliminates the need for explicit observation model and can directly utilize high-dimensional features. By lever- aging this approach, our framework enables more flexible and efficient integration of measure- ments without the explicit requirement of a mapping function from the state to the measurement do- main. Instead of using one sensor encoder, we use multiple sensor encoders [s s-), s²), ..)...... s (.)l to learn latent observations from each modality:
state
observation
observation
K Keys
Ensemble
Auxilarr
quary
Causality Attention Map
Figure Attention gain (AG) module uses Iearned causality-enforeed attention map to replace Kalman gain.
y (i.m) st (gl (k.m) (y"",, ViE E, m M.
M is the number of modalities in the system, M E Z The encoders generate series of latent observations, Y, [Y!.... Y: E Md where Y.m [y(1.m] ..... y (E..)) E R
The latent observations are then concatenated with predicted state X, as input to the AG model:
X, softmax (Q(X) + "I)TT
M )M
(X. (X. 4 Y.).
(5)
VE

---

where "Ø" denotes the concatenation and "o" is the Hadamard product, and X, is the final output. In general, an attention module typically receives three sequences of tokens: queries Q, keys and values V. In our case, we define (X; + Y!) as the K tokens, where X and Y are obtained by zero-centering, and the actual values of (X, + Y.) are regarded as the V tokens. As illustrated in Fig. the length of the K tokens is denoted as dR (M + 1)dg. where each token has dimension of E, representing the distribution along this particular token index.
In traditional attention mechanism, the proximity of Q and K is measured, and V that is associated with K is utilized to generate outputs. However, we posit that within each latent vector, every index probabilistically independent, and index latent observation. To accomplish this, we utilize matrix M to retain only the diagonal elements of each (d x dz)) attention map, which enforces causality and allows the attention weights to be determined according the corresponding indices. As depicted in Fig. the red line represents the mapping for single latent state token index. Auxiliary query tokens Q Idd X are introduced as trainable parameters in the neural network to facilitate learning. It is important to note that both the Q and K tokens undergo positional embedding before being fed into the AG module.
latent state should only consider index of each
Within the framework of Kalman filters, the update step plays crucial role in aligning the predicted observation with the observations obtained from sensors. Within the framework of a-MDF, we ensure consistency in the latent space by introduc- ing decoder model D. This decoder model, implemented using multilayer perceptrons, projects the latent space onto the actual state space. By doing so, we resolve the alignment challenges in multimodal learning [39], and gain meaningful comparisons when conducting sensor fusion and measurement update. Let xe be the ground truth state at t, the loss functions are defined as:
Placing Conditions on the Latent Space:
Sio- [D((e(X))) - 2el12, Lete ID(&)) - 1: C. = (D(s""(y"")
Iz 12.
(6)
The final loss function is L L + Laze + L... where Leze is the end-to-end loss. Li is used to supervise the state transition model. The latent observation conditioning is provided with L, during the training process, note that the conditioning operation is applied when the modalities collectively provide information about the complete state. The modular architecture of a-MDF provides a key advantage in facilitating training and testing with masked modalities. The attention matrix M can be disabled (set the attention values to zero) based on different input sensor modalities, thus improving the model's resilience missing modalities.
4 Experiments
We conduct series of experiments to evaluate the efficacy of the a-MDF framework. Specifically, aim to answer the following questions: (a) Can the o-MDF framework generalize across various tasks? (b) To what extent does the new filtering mechanism improve state tracking performance when compared to the current state-of-the-art? (c) How does the use of multiple modalities compare to a subset of modalities for state estimation with differentiable filters? Therefore, we evaluate the effectiveness of a-MDF across multiple roboties tasks, each with distinct setups: (1) Visual odometry for autonomous driving, (2) Robot manipulation employing multi-modalities in both real-world and simulation, and (3) Soft robot modeling task. Our study examines two categories of baselines: (a) DF baselines such those proposed in [9, 28, 26], including dEKF [9]. DPF [28], and dPF-M-Irn [9]; and (b) sensor fusion baselines proposed in [22]. Additional details on the baselines can be found in Appendix A.3.
4.1 Visual Odometry Task
In this experiment, we evaluate the performance of o-MDF on the popular KITTI Visual Odometry dataset [40]. Since the visual odometry task uses a single modality. we only consider RGB images as the input modality in order to make a fair comparison with the baselines [9, 28, 26). The actual state is defined as 5-dimensional vector æ [x, y, 0, 0j"", including the position and orientation of the vehicle, and the linear and angular velocity. We use the latent state x E 256 for a-MDF. In comparison to JEKF, DPF, and dPF-M-Im, observe reduction in the translational error of
5

---

Adtion: the robot pida
Actint: robet down
State
RGB
Depth
Joint
State
RGB
Joint
X, ottentiin
map
Figure 4: Learned attention gain. Left: manipulation in simulated environment with modalities [y y² ",, and right: real robot manipulation with modalities ,y',"",. The attention maps indicate the attention weights assigned to each modality during model inference. In the visualization, regions red correspond to low attention values, while those in blue indicate high attention values.
approximately 88%, 83%, and 79% for Test 100/200/400/800. The results also reflect considerable reduction in rotational eтror of approximately 64%, 54%, and 46% as compared to each of the baselines. We report a detailed experimental setup and thorough results in Appendix B.
4.2 Multimodal Manipulation Task
This experiment aims to evaluate the effectiveness of the o-MDF framework in robot manipulation scenario. Specifically. we use a-MDF for monitoring the state of URS robot during tabletop ar- rangement tasks. Similar to behavioral cloning from observation tasks [41]. actions are not available as inputs for this study. Instead, we train a-MDF to learn how to propagate the state of the robot over time. The evaluation involves three manipulation tasks, namely: (1) estimating the state of the robot in a simulated environment, (2) estimating the state of the real-world robot, and (3) estimating the joint state of the robot and the object being manipulated.
Task Setup and Data: For a-MDF, we define the latent state x E 256 for all the sub-tasks. The actual state of the UR5 robot is defined by æn. which consists of the joint angles (J1-J7) and the Cartesian coordinates (z, y, 2) of the robot's end-effector (EE). zo denotes the state of the object, which only includes the location (z. y, 2) of the object. The complete set of modalities comprises [y", y y yJ,, where y' 224x222x represents RGB images, y² 224xx22 represents depth maps, y E R represents proprioceptive inputs (joint angles), and y R represents Force/torque (F/T) sensor readings. However, the input modalities for each of the three tasks may differ; for task (1), it involves ,y,,2,y") for task (2), it comprises ly', y while task (3) has [y ,y² y y'I. A more detailed description of the task setup and data collection is supplied in Appendix B.2.
Results: Using the same comparison protocol in [9, 28, 26), Table com- pares the proposed framework's perfor- mance with other DF baselines. Note that all baselines perform tracking in actual space, therefore, we use a pre- trained sensor encoder to process RGB modality for all DFs and supplied the la- tent embedding a-MDF, as DF base- lines only take modality. c-MDF outperforms dEKF and DPF, reducing errors by 33% and 25% in real-world and 45% and 30% in simulation, with an average MAE of 3.81cm and de- viation of 1.06cm from ground truth for end-effector positions. Additionally, a-MDF exhibits 42% and 26% improvement in estimating joint angles compared to dPF-M-Im. In the case of fil- tering with multiple modalities, results presented in Table 2 show clear improvements achieved by o-MDF in comparison to other sensor fusion techniques. The baselines are reproduced following the procedure of [22] by providing the same pretrained sensor encoder each modality. a-MDF outperforms all other methods across all three manipulation tasks. In particular, it cuts the posi- tional error of the end-effector (EE) in half when compared to the crossmodal fusion strategy [22]
Table 1: Result evaluations on URS manipulation task
Real-world (MAH)
Simulation (MAF)
Method
Joint (deg)
EE (em)
Joint (deg)
EE (cm)
dEKF [9]
16.08=0.1
5.67.0.1
4.931:.22
1.91=0.1
DPF [28]
15.93=0.11
5.08=2033
4.46:10.2
1.51002
dPF M-Im [9]
12.83-0011
3.95+0.4
3.82+0022
1.26-0.1
o-MDF
7.49
3811:0.2
2.84:0.1
1.06:0.1
standas errors.

---

(6) manipulation task realoworld
Men
Prppleplies
(b) URS manipulation task simulation
Jeat
Joas-
123
No Proprioccption
RGB
Depth
16
200
30
Figure 5: Predicted joint angle trajectories and the corresponding accumulated attention values for each modal-
ity. (a) represents the results attained from the actual robot, whereas (b) illustrates attention values for all modalitics both with and without masking certain modalities.
Table 2: Result evaluations URS manipulation task with multimodal sensor fusion baselines.
Simulation (MAE)
Real-world (MAE)
Simulation with I/T (MAE)
Method
Joint (deg)
EE (cm)
Joint (deg)
EE (cm)
Joint (deg)
BB (cm)
Obj (cm)
Feature Fusion [22]
7.58=0..1
3.15
11.25:1.17
5.65::0.01
3.62:00.09
2.72:10.22
8.36:10.06
Unimodal [22]
7.46:10.32
3.18110.3
11.0210.08
9.52.10.07
3,9710008
3.63:10.05
10.23:10.10
Crossmodal [22]
3.64:10.3
1.9110.04
5.9820.08
7.35:10.55
3.1240.02
3.25:10.02
5.5410.02
o-MDF 2.19:10.09
0.75:10.1
5.24
3.04:0.01
1.110004
0.90:10.01
1.65:0.01
Means: standard emrrr.
on the real-robot (7.35em 3.04cm). In simulation tasks, it achieves
better reduction in tracking error (3.25cm - 0.90cm in simulation with F/T sensing). We present visualization of the leamed attention gain in the filtering process (Fig. 4) and state tracking results with and without cer- tain modalities (Fig. 5). Despite the attention values changing when certain modalities are missing, a-MDF still achieves stable results. Further results and explanations can be found in AppendixB.2
4.3 Soft robot Modeling
Tensegrity structures [42] have become popular recent years since they bridge the gap between an inherently flexible system and the ability to use rigid components [43, 44, 45). However, modeling such complex system continues to pose considerable challenges due the high nonlinearity. This experiment involves implementing the a-MDF to model the dynamics of a soft robot system, especially Tensegrity robot [45].
MoCap
Layer
Layer 4
Layer
The robot structure is shown Fig. 6 with 5 layers of tensegrity. The actual state of soft robot at time represented by 7 7-dimensional vector tion and orientation of the robot's hand tip. The quaternion vector q represents the posture of the robot w.r.t the base (layer I's bottom). In this task, we de- fine x E 256 as the latent state. The complete set of modalities comprises [y' y,, y2,, where y E 222xx244x3 represents RGB images, y² E 2224x224 is depth maps, and y 3R30 is proprioceptive inputs (IMUs). The action a, of the system is the pressure vector of the 40 pneumatic cylinder actuators, where a, E 10 In this experiment, synthetic depth maps generated offline using the DPT model [46]. Figure. 7 shows the recordings of RGB and the depth modalities, further details regarding the task setup and data collection is in Appendix B.3.
Task Setup and Data:
Layer
IMU-
Layer
[xr y, 2, qr. 9y 9= 9w. which denotes the posi-
Figure 6: The tenseg- rity robot structure.

---

2000
1=0.00
t=9.99s t.3.325
750
100
1750
Tramiry
Figure 7: Estimated end-effector (EE) positions for tensegrity robot. Left: the RGB and depth modalities
ly ly',"" ") and right: state estimation results with ensemble distribution.
Results: The soft robot modeling task is evaluated using 10-fold cross-validation and the mean absolute enror (MAE) metric, and the results are presented in Table 3. Our results demon- strate that a-MDF outperforms the state-of-the-art methods in terms of DFs, achieving MAE of 8. 8.99cm. Specifically, our approach yields an MAE on the end-effector (EE) position estimation that is 45%, 34%, and 29% lower than that obtained by dEKF, DPF, and dPF-M--m,, respectively. Of the sensor fusion baselines, cross- modal fusion [22] exhibits marginally better outcomes than others, although it do not show any advantages over a-MDF in predicting EE positions (2.14cm++1.67cm) Notably, a-MDF surpasses the feature fusion strategy by a significant margin of 4-fold. Addition- ally, appendix B.3 delves into an explo- ration of the potential benefits of modal- ity selection for state estimation, where optimal combinations can be selected to achieve even higher accuracy. The results presented in Fig. demonstrate the efficacy of c-MDF in accurately estimating the state of soft robots in a multimodal setting, the ensemble distribution is indicated by gray shade representing the model uncertainty. With stable performance achieved over an extended duration inference, a-MDF has shown the potential in modeling dynamics for various complex non-linear systems.
Table 3: Result evaluation on soft robot modeling task.
RGB
Depth
IMUs
EE (cm)
9 (10")
dEKF [9]
16.38++.110
DPF [28]
13.68-40.02
0.96++0.03
dPF-M-Im [9]
12.66++0.09
1.1040.03
d-MDF
8.99:1002
0.79:10.33
Feature Fusion [22]
8.35=20.22
0.60:0.03
Unimedal
2.7810.05
0.25:10.2
Crossoodal [22]
2.14:10.05
0.15
c-MD
Mstaddaa errors.
5 Conclusion
This paper illustrates how utilizing attention as gain mechanism in differentiable Bayesian filtering and multimodal learning can significantly enhance the accuracy of robot state estimation in numer- ous tasks. Proposed a-MDF is a unique differentiable filter that conducts filtering on a compressed multimodal latent representation, while preserving the integrity of the Kalman filter algorithm com- ponent. Our experiments demonstrate that a-MDF is appropriate for learning both rigid body and soft robot dynamics, exceeding baseline performance by up to 4-fold. Moving forward, plan to investigate the value of incorporating additional modalities, such as sound, temperature, and prox- imity sensing, into a-MDE
An obvious difference of o-MDF when compared to traditional filters is the required learning process this typically takes multiple hours of training on current machines. In similar vain, an inherent assumption that the training and test distributions do not differ substantially, ie., the problem of concept drift. To date, we have successfully tested the algorithm with latent states consisting of several hundred variables. We use 256 dimensions in the experiments for con- sistency. However, more research is required to understand a-MDF's performance when filtering over thousands of variables. As with any deep learning approaches, hyper-parameter tuning may be required to produce perforrenn models. Another practical observation is that utilizing more modalities does not always translate to improved performance, which is consistent with findings in [47]. Including redundant modalities can impose longer training times and pose greater difficulty for the model in extracting valuable information from the input modalities. A pre- pre-processing step for feature selection may be advisable.
Limitation:

---

Acknowledgments
This work has received partial support from the National Science Foundation under grants CNS- 1932068, IIS- IIS-1749783 Additionally, partial support has been provided by JSPS KAKENHI Grant Numbers 22H03671 and 22K19815. We would like to sincerely acknowledge the valuable com- ments and feedback provided by the reviewers. Our gratitude also goes to Yuhei Yoshimitsu for assisting in the data collection with the tensegrity robot. Furthermore, we would like to express our appreciation for the insightful discussions and constructive feedback received from Fabian Weigend and Shubham Sonawani during the review process.
References
[1] S. Thrun, W. Burgard, and D. Fox. Probabilistic robotics. MIT Press, Cambridge, Mass., 2005. ISBN 0262201625 9780262201629.
[2] L. Wang, G. Wang, S. Jia, A. Turner, and S. Ratchev. Imitation learning for coordinated human-robo collaboration based on hidden state-space models. Robotics and Computer- Integrated Manufacturing, 76: 16:102310, 2022. ISSN 0736-5845. dbithtts::/ddii.or//1..00 j.rcim2021.1023100
[3] S. Chen. Kalman filter for robot vision: survey, IEEE Transactions on industrial electronics, 59(11):4409- 4420, 2011.
[4] Reher, W.-L. Ma, and A. D. Ames. Dynamic walking with compliance on cassie bipedal robot. In 2019 18th European Control Conference (ECC), pages 2589-2595. IEFE, 2019.
[5] L. Ljung. System identification. Springer, 1998.
[6] G. Evensen. The ensemble kalman filter: Theoretical formulation and practical implementa- tion. Ocean dynamics, 53(4):343-367, 2003.
[7] S. S. Rangapuram, M. Seeger, Gasthaus, L. Stella, Wang, and Januschowski. Deep state space models for time series forecasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc., 2018.
[8] A. Klushyn, R. Kurle, M. Soelch, B. Cseke, and van der Smagt. Latent matters: Learning deep state-space models. Advances in Neural Information Processing Systems, 34, 2021.
[9] A. Kloss, G. Martius, and J. Bobg. How to train your differentiable filter. Autonomous Robots, pages 1-18, 2021.
[10] D. Ramachandram and G. W. Taylor. Deep multimodal learning: A survey recent advances and trends. IEEE signal processing magazine, 34(6):96-100 108, 2017.
[11] M. Y. Seker, A. Ahmetoglu, Y. Nagai, M. Asada, E. Oztop, and E. Ugur. Imitation and mirror systems in robots through deep modality blending networks. Neural Networks, 146:22-35, 2022.
[12] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and Bohg. Making sense of vision and touch: Learning multimodal representations for contact- rich tasks. IEEE Transactions on Robotics, 36(3):582-596, 2020.
[13] T. Xue, W. Wang, J. Ma, W. Liu, Z. Pan, and M. Han. Progress and prospects of multimodal fusion methods in physical human-robot interaction: A review, IEEE Sensors Jourmal, 20 20(18): 10355-10370, 2020.
[14] S. Sonawani, Y. Zhou, and H. B. Amor. Projecting robot intentions through visual cues: Static dynamic signaling. arXiv preprint arXiv:2308.09987 2023.

---

[15] Z. Yu, M. Chen, Z. Zhang, S. You, and F. Ren. Transupr: A cansformer-bssed uncertain point refiner for lidar point cloud semantic segmentation. arXiv preprint arXiv:2302.088544 2023.
[16] J. Huang, A. Mishra, B. C. Kwon, and C. Bryan. Conceptexplainer: Interactive explanation for deep neural networks from a concept perspective. IEEE Transactions on Visualization and Computer Graphics, 29(1):831-84) 2022.
[17] Y. Zhou, S. Sonawani, M. Phiclipp, H. Ben Amor, and S. Stepputtis. Learning modular language-condttioned robot policies through attention. Autonomous Robots, pages 1-21, 2023.
[18] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[19] Y. Zhou, Sonawani, M. Phielipp. Stepputtis, and H. B. Amor. Modularity through atten- tion: Efficient training and transfer of language-conditioned policies for robot manipulation. arXiv preprint arXiv:2212.04553, 2022.
[20] S. Stepputtis, J. Camphell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language- conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 23:13139-131555 2020.
[21] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8950. IEEE, 2019.
[22] M. A. Lee, B. Yi, R. Martin-Mattnn S. Savarese, and Bohg. Multimodal sensor fusion with differentiable filters. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10444-10451. IEEE, 2020.
[23] A. Corenflos, J. Thortoo,, Deligiannidis, and A. Doucet. Differentiable particle filtering via eegropy-regularizze optimal transport. In International Conference Machine Learning, pages 2100-2111. PMLR, 2021.
[24] N. A. Piga, U. Pattacini, and L. Natale. A differentiable extended kalman filter for object tracking under sliding regime. Frontiers in Robotics and AI, 8:686447, 2021.
[25] W. Li, X. Chen, W. Wang, V. Elvira, and Y. Li. Differentiable bootstrap particle filters for regime-switching models. arXiv preprint arXiv:2302.10319 2023.
[26] T. Haarnoja, A. Ajay, S. Levine, and P. Abbeel. Backprop kf: Learning discriminative deter- ministic state estimators. In Advances in neural information processing systems, pages 4376- 4384, 2016.
[27] Karkus, X. Ma, D. Hsu, L. Kaelbling, W. S. Lee, and Lozano-Pérez. Differentiable algorithm networks for composable robot learning. arXiv preprint arXiv: 2019.
[28] R. Jonschkowski, D. Rastogi, and O. Brock. Differentiable particle filters: End-to-end learning with algorithmic priors. arXiv preprint a80i /1/2, 2018.
[29] X. Chen, H. Wen, and Li. Differentiable particle filters through conditional normalizing flow. In 2021 IEEE 24th International Conference on Information Fusion (FUSION), pages 1 1-6. IEEE, 2021.
[30] Y. Wang, B. Liu, J. Wu, Y. Zhu, S. S. Du, L. Fei-Fei, and Tenenbaum. Dualsme: Tunneling differentiable filtering and planning under continuous pomdps. arXiv preprint arXiv: 13003, 2019.
10

---

[3] X. Liu, G. Clark, J. Campbell, Y. Zhou, and H. B. Amor. Enhancing state estimation in robots: A data-driven approach with differentiable ensemble kalman filters. arXiv preprint arXiv:2308.0990 09870, 2023.
[32] X. Liu, S. Ikemoto, Y. Yoshimitsu, and Amor. Learning soft robot dynamies using dif- ferentiable kalman filters and spatio-temporal embeddings. arXiv preprint arXiv:2308.0998 2023.
[33] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new per- spectives. IEEE transactions on pattern analysis and machine intelligence, 3788)11798-1888, 2013.
[34] G.-H. Liu, A. Siravuru, Prabhakar, M. Veloso, and G. Kantor. Learning end-to-end mul- timodal sensor policies for autonomous navigation. In Conference on Robot Learning, pages 249-20 261. PMLR, 2017.
[35] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In International conference on machine learning, pages 1704-1713. PMLR, 2018.
[36] H. Van Hoof, N. Chen, M. Karl, van der Smagt, and Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 3928-3934. IEEE, 2016.
[37] J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-rt: Learning multimodal manipulation policies with deep reinforcement learning. In 2022 International Conference on Robotics and Automation (ICRA), pages $298-8304 IEEE, 2022.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and Polo- sukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[39] P.P. Liang, A. Zadeh, and L.-P. Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint 2:Xiv:2009.03330 2022.
[40] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-338 3361. IEEE, 2012.
[41] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint 1rXiv:1805.01954, 2018.
[42] R. E. Skelton and M. C. Oliveira. Tensegrity Systems. Springer Nature, 2009. ISBN 978-1- 4419-4491-7.
[43] E. Jung, V. Ly, N. Cessna, M. Ngo, D. Castro, V. SunSpiral, and M. Teodorescu. Bio- inspired tensegrity flexural joints. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5561-5566. IEEE, 2018.
[44] K. Kim, A. K. Agogino, and A. M. Agogino. Rolling locomotion of cable-driven soft spherical tensegrity robots. Soft robotics, 7(3):346 361, 2020.
[45] S. Ikemoto, K. Tsukamoto, and Yoshimitsu. Development of a modular tensegrity robot arm capable continuous bending. Frontiers in Robotics and AI, 8, 2021.
[46] R. Ranftl, A. Bochkovskiy, and Koltun. Vision transformers for dense prediction. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pages 22179-12188 2021.
11

---

[47] W. Wang, D. Tran, and M. Feiszli. What makes training multi- modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pages 16695-12705 2020.
[48] L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun. Hands-on bayesian neural networks a tutorial for deep learning users. JEEE Computational Intelligence Magazine, 17 (2):29-48,
10010.00000M22222.....1
[49] Y. Gal and Z. Ghahramani. Dropout bayesian approximation: Representing model uncer- tainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[50] D. Kingma, Salimans, and M. Welling. Variational dropout and the local reparameteriza- tion trick. Advances neural information processing systems, 28, 2015.
[51] K. Lenac, J. Česié, I. Marković, and Petrović. Exactly sparse delayed state filter on lie groups for long-term pose graph slam. The International Journal of Robolics Research, 37(6): 585-610, 2018.
[52] J. Zhang and S. Singh. Visual-lidar odometry and mapping: Low-drift, robust, and fast. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2174-2181. IEEE, 2015.
[53] C.-C. Chou and C.-F. Chou. Efficient and accurate tightly-coupied visual-lidar slam. IEEE Transactions on Intelligent Transportation Systems, 2021.
[54] I. Cvisié, J. Česié, I. Marković, and I. Petrović. Soft-slam: Computationally efficient stereo vi- sual simultancous localization and mapping for autonomous unmanned aerial vehicles. Journal of field robotics, 35(4):578-95) 097788595,11 2018.
[55] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[56] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[57] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, (1(2))2336-236,, 2018.
[58) K. He, X. Zhang, S. Ren, and Sun. Identity mappings in deep residual networks. In Computer Fision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11- 14, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016.
[59] Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05101, 2017.
12

---

A Details in a-MDF
This section provides a detailed overview of the previously mentioned a-MDF modules, and de- scribes differentiable Ensemble Kalman filters as the underlying DF' framework for a-MDE.
A.1 Model Initialization and Embedding Functions
An auxiliary model A is supplied in the filtering process to support training by starting the filter via projecting the actual state 2t-Na-1 from low-dimensional space to latent space. The model is implemented using stochastic neural networks (SNNs) [48].
Xi-Na-1~ A -
E,
where x is one latent state, the latent state ensemble is obtained by sampling A for E times. During inference, we employ the trained sensor encoders' output, which the latent representation of RGB, depth, or proprioception, the initial state initiate the filtering process.
Regarding the prediction step of a-MDF, we apply positional embedding layers (sinusoidal func- tions) [38] in the transformer process model (Eq. 3) to generate e, N as the embedding for time-series data, e = fx E (dNx((N1)) The positional embedding layer is uti- lized to label the state by index it with time t. When activating the action a, in the process model, we also utilize type embedding layer that indexes N and a, with and and then fed sinusoidal functions. Subsequently, the element-wise summation of outputs obtained from the aforementioned procedures serve input to the transformer process model for further processing.
A.2 Differentiable Ensemble Kalman Filter
Unlike prior proposals for differentiable filters, such as dEKF [9] and DPF [28], Differentiable En- semble Kalman Filter [6] leverages recent advancements in stochastic neural networks (SNNs) [48]. Specifically, we draw inspiration from the work in [49]. which established theoretical connection between the Dropout training algorithm and Bayesian inference in deep Gaussian processes. As result, we can use stochastic forward passes produce empirical samples from the predictive poste- rior of a neural network trained with Dropout. Hence, for the purposes of filtering, we can implicitly model the process noise by sampling state from neural network trained on the transition dynamics, i..., (x) In contrast to previous approaches [28, 9], the transition network fe(-) models the system dynamics, as well as the inherent noise model in a consistent fashion without imposing diagonality.
initial ensemble of E members to represent the initial state distribution Xo [x]....,*f xf). E E z We leverage the stochastic forward passes from trained state transition model to update each ensemble member:
Prediction Step: Similar to c-MDF,
x f ~ fo fo(x///-//// fo(xx|--lxx_-
~
E.
x - holds the updated ensemble members which are propagated one step forward through the state space. Note that sampling from the transition model fo (using the SNN methodology described above) implicitly introduces a process noise.
Matrix X -
Update step: Given the updated ensemble members X nonlinear observation model h-) is applied to transform the ensemble members from the state space to observation space. Following our main rationale, the observation model is realized via a neural network with weights sb. Accordingly, the update equations become:
E hap (x))
E hap
1
H,A, = H,X,
(xi)
(9)
ÿ s(vil)))) V. (10)
F
E
1=1
H2X, is the predicted observation, and H, is the sample mean of the predicted observation at t. Traditional Ensemble Kalman Filter treats observations random variables. Hence, the ensemble
13

---

can incorporate a measurement perturbed by small stochastic noise to reflect the error covariance of the best state estimate [6]. In differentiable Ensemble Kalman Filter, we incorporate a Bayesian sensor encoder s(.) Sensor encoder serves to learn projections from observation space to latent space as in Eq. 10, where y, represents the noisy sensor observation. Sampling from sensor encoder yields latent observations Y, = [ ... 1. The KF update step can then be continued by using the learned observation and predicted observation:
K E E A.
E - (HA) + +R)
(11)
The measurement noise model R is implemented using multilayer perceptron (MLP), similar to the implementation in [9]. The MLP takes a learned observation Y, at time t and produces a noise covariance matrix. The final estimate of the ensemble &, is obtained by performing the measurement update step, given by:
X. X, + K, H,X,).
(12)
In inference, the ensemble mean X1I: + E E x is used as the updated state.
A.3 Baselines
In our study, we examine two categories of baselines: (a) DFs baselines, which consist of existing methods such as those proposed in [9, 28, 26]. and (b) sensor fusion strategies, as proposed in [22].
Table Dimensions pertinent to each of the robot state estimation tasks.
Visual Odometry
URS
Manipulation
Soft Robot Observation
Method
State
Observation
State
Observation
State
Action
dEKF [9]
2
10
10
40
DPF [28]
2
10
10
40
dPF-M-Imn [9]
5
2
10
10
7
40
Feature Fusion [22]
10/13
10/13
40
Unimodal [22]
10/13
10/13
40
Crossmodal [22]
10/13
10/13
7
40
o-MDF
256
256
256
256
256
256
40
Dimensionality: Table 4 presents the dimensions for the state, observations, and actions utilized for each of the tasks. To ensure consistency, we opt for a dimension of 256 for a-MDF in all tasks, thus, enabling filtering over high-dii sigh-dimmnsional spaces. Unlike the baseline methods, which use low- dimensional state definitions, we filter over higher dimension spaces with o-MDF
Differentiable Filters: To maintain consistency in the comparison of results against the DFs base- lines, we train c-MDF with single modality. The baselines in this category include the differen- tiable Extended Kalman filter (dEKF) [9], differentiable particle filter (DPF) [28], and the modified differentiable particle filter (dPF-M-Irn) [9], which uses learned process and process noise models. For dεKF, the Jacobian matrix in the prediction step can either be learned end-to-end or supplied if the motion model known. DPF employs 100 particles for both training and testing and also in- corporates an observation likelihood estimation model 1. This module takes an image embedding and produces likelihood that updates each particle's weight. Unlike DPF, dPF-M-IM implements learnable process noise model. It also adopts a Gaussian Mixture Model for calculating the like- lihood for all particles. It is worth noting that all the bascline methods perform Kalman filtering on low-dimensional actual state space, whereas a-MDF executes the filtering process in the latent space.
Sensor Fusion: Regarding sensor fusion baselines, we use three strategies discussed in [22], namely, Feature Fusion, Unimodal Fusion, and Crossmodal Fusion. The Feature Fusion strategy aims to process each modality individually and subsequently merge the modalities to generate multimodal feature set using neural networks, which is then used for state estimation. The Unimodal Fusion
14

---

treats each modality N ~ (Mt M: 1) and N ~ (n, modal distribution as one normally distributed multimodal distribution, 'N' (1y Ey):
M= "u, as distributions and fuse two uni-
# Mi
+ - Mz (E
= = - + (ー)
(13)
where the associative property can be used for fusing more than two modalities. For Crossmodal Fusion, information from one modality can be used determine the uncertainty of the other ones, two coefficients are proposed as B and B, where each coefficient has the same dimension of the state, the fused distribution is:
M + B Ma
Σ. = B.M EM + B, E,M
#1
B + B
B + B,
(14)
where B. (B As mentioned in [22], each sensor encoder independently trained and subsequently used for end-to-end training with DFs. We adopt a similar approach, but with differentiable Ensemble Kalman Filter backbone in place instead. The resampling procedure from the fused distribution in this scenario is achieved by using the reparematerization trick [50].
B Additional Experiments
This section presents supplementary experimental results for each task. For (1) Visual Odometry Tasks, we offer full detailed experiments: however, for (2) Multimodal Manipulation Tasks and (3) Soft Robot Modeling Tasks, concentrate mainly on ablation studies.
B.1 Visual Odometry Tasks
In this experiment, we investigate the performance of a-MDF on the popular KITTI Visual Odom- etry dataset [40]. We only consider RBG images as the input modality in order to make a fair com- parison with the baselines [9. 28, 26). Following the same evaluation procedure as our baselines, we define the actual state of the moving vehicle 5-dimensional vector {ar, y, 0, v. 6)"", including the position and orientation of the vehicle, and the linear and angular velocity w.r.t. the current head- ing direction 0. The raw observation y corresponds to the RGB camera image of the current frame and difference image between the current frame and the previous frame, where E 150x500x6 as shown in Fig. 8. The learned observation is defined as ý = [u. 0) since only the relative changes of position and orientation can be captured between two frames. We use the latent state x E RR256 for a-MDE
Data: The KITTI Visual Odometry dataset in- cludes 11 trajectories capturing the ground truth pose (translation and rotation matrices) of ve- hicle navigating urban areas at a data collec- tion rate of approximately 10Hz. To facilitate the learning process, we standardize the data by normalizing each dimension have a mean of 0 and a standard deviation of during training. To process the provided pose data, convert them to quaternions to capture the minimal changes between consecutive quaternion pairs. Subse- quently, the results are converted back to radians to represent the angular velocity 6. This conversion ensures that the angular velocity remains minimal and falls within the range of [-,,
RGB images
Difference map
Figure 8: KITTI visual inputs.
B.1.1 Results
The performance of state estimation evaluated using an 11-fold cross-validation, whereby tra- jectory withheld at each time. The standard KITTI benchmark metries, namely the translational error (m/m) and rotational error (deg/m), are reported in Table 5. The error metrics computed from the test trajectory over all subsequences of 100 timesteps, well all subsequences of 100,
15

---

Table Result evaluations on KITTI Visual Odometry task measured in m/m and deg/m denote the translational eπror and the rotational eπror.
Test 100
Test 100/200/400/000 m/m
Method
m/m
deg/m
deg/m
dEKF [9]
0.2646=00004
0.1386=:0.002
0.3159::0.002
0.0923==0.005
DPF [28]
0.1344=0.002
0.1203=+0.07
0.2255=0.001
0.0716=:00004
dPF-M-Irn [9]
0.1720=0.010
0.0974:10.009
0.1848110.004
0.0611100003
a-MDF
0.0718=0.001
0.095410.001
0.0379-0.002
0.0328-1.001
Means.Lstandard errors.
200, 400, and 800 timesteps. Figure 9 presents the performance of a-MDF and other differentiable filtering techniques. It important to note that incorporating domain- and data-specifie information, such using stereo images [51]. integrating LiDAR [52, 53], applying SLAM and loop-closure related assumptions [51, 54], can yield lower error metrics. However, to ensure fair and compara- ble evaluations, we utilize the most commonly used setup when comparing filtering techniques in a task-agnostic fashion (as performed [9, 28, 26)).
Table 5 presents the outcomes of pro- posed method in comparison with the ex- isting state-of-the-ar DFs, namely dEKF, DPF, and dPF-M-Imn. In order to pro- vide a fair comparison, we do not include unstructured LSTM models as baselines since prior works [26, 9] have shown that they do not achieve comparable results. The pre-trained sensor encoder with the same visual inputs is used and integrated into all the DF frameworks evaluated. In this experiment, the motion model the vehicle is known, and the only unknown part of the state is the velocities. In light of the above, we adopt learnable process model to update state variables alongside an established motion model to update the (x, y. 0) variables. While the computed Jacobian matrix is supplied in training and testing for dEKF, our a-MDF demonstrates significant improvements compared to dEKF, DPF, and dPF-M-Im. Specifically, we observed duction in the translational етrоr of approximately 88%, 83%, and 79% for Test 100/200/400/800. The results also reflect a considerable reduction in rotational error of approximately 64%, 54%, and 46% as compared to each the baselines. Our analysis a-MDF reveals that conducting filtering on high-dimensional observations the latent space yields better results than conducting filtering on the actual state space.
0.4
m/in
Test deg/m
Test 100/200/400/00 mim
0.3
Ervor zate
Test 100/200-400800 deg/m
0.
0.1
0.0
2.5TM
BK3
JEKF
DPF
4PF-M
MDD
Figure 9: Visual Odometry results with different differen- tiable filters: error rate LSTM and BKF are reported from [26]. dEKF, DPF, and dPF-M are reproduced.
B.1.2 Compare to EKF
In this section, present comparison of the results obtained from non-learning Extended Kalman Filter (EKF) and a-MDF on the KITTI Visual Odometry task. As previously mentioned, the actual state of the moving vehicle is represented by 5-dimensional vector æ [x. y, 0 v, 0)T while the observation ý is defined as ý = [v.0 0J"" The EKF can be formulated using the provided analytical model.
f(x, 1) Az; q qr (O Q:), ÿ h(z,) +r, H +
(15)
- N(O0 R,).
where H is identity matrix H = 110101 The EKF prediction step is: (000011 100011
f Ar,1 4. E, =
QQ.
(16)
16

---

where the Jacobian of the process model can be supplied via Taylor expansion,
0
sin 0 A
0
0
cos 0
sin O
0
cos 0 A
0
0
0
t sin O&L
COs A
A
0
0
1
0
A
F
0
0
1
0
A
0x1-1
0
0
0 0
0
1
0
0
0
0
1
0
0
0
0
1
(17)
The update step for EKF
S. H H R., K, E,H'
(18)
2; -; Rr (ý H E, = -I- K,H
To ensure fair and unbiased comparison, both the Extended Kalman Filter (EKF) and a-MDF models are provided with the same low-dimensional observation y + e, where is a noise sample obtained from a Gaussian distribution N ~ (0, 1.5 D.
We report a comparison via translational and rotational errors in Table 6. For the EKF model, the noise covariance matrices Q; and R3 are manually fine-tuned. Additionally, we initialize the fil- ter with Eg I. For a-MDF, keep the same framework when filtering over latent state with 256 dimensions. However, substitute the sensor encoder from s to s (refer to Table 12). This modification allows for projecting the low-dimensional ob- servation into the latent space. Our observations indicate that a-MDF, when utilizing attention gain, reduces the error over the EKE. a-MDF has the additional benefit of automatically learning noise profiles during the training process, thereby climinating the manual tuning step required by the EKF.
Table 6: Comparison between EKF and a-MDF.
Test 100
Test 100200/400/800
Method
m/m
deg/m
m/m
deg/m
EKF
0.2757-0.03 0.154810.02 0.239140.02
0.0623:10.01 0.1509=0.01 0.0327+0.01
o-MDF 0.1642++0.02
0.0593=0.01
Mcans=standard errors.
B.1.3 6D Motion State
To conduct a more comprehensive investigation into the visual odometry task, we extended our analysis by employing larger state space. In this section, consider the 6D motion of the vehicle where 3 different heading directions are defined namely yaw 0, pitch 1/ and roll ø. The actual state is defined as [z. V, Φ, s), 0, U1, 12, 03, Φ, và, 6)__. Similar to the previous setup, a-MDF takes the image pair at - and I as input, with the observation ý is defined as ý [ui, U2, Ca à, vi, 0J
The comparison results for translational and rota- tional errors, with an in- creased state space, are presented in Table 7. No- tably, the performance of a-MDF remains stable along the yaw axis, as ob- served in comparison the results reported in Table 5. Additionally, observe smaller translational and rotational rors on the pitch and roll axes. This observation can be attributed to the relatively minor deviations on the 2 axis, even during inclined maneuvers such as ascending or descending a hill. In conclusion, when the search space expands to include a larger state space, a-MDF demonstrates comparable results, indicating its ability to handle increased complexity and maintain performance.
Table Using 6D motion state for a-MDF
Test 100
Test 100/200/400/800
Method
Axis
m/m
deg/m
m/m deg/m
a-MDF Yaw 0
0.072.10.001
0.097110.004
0.04110.003
0.033.10.001
a-MDF
Pitch d
0.032-10.002
0.013-10.001
0.02810.003
0.01910.001
a-MDF
Roll
0.033:10.004
0.032=00010
0.049=0.001
0.02910.002
Mtansastandard emrrr..
B.2 Multimodal Manipulation Tasks
Task Setup: For a-MDF, we define the latent state E R for all the manipulation tasks. The actual state of the UR5 robot is described by zR. which consists of the seven joint angles (J1-J7)
17

---

Table 8: Ablation study UR5 manipulation task with different combination of the modalities.
RGB
Depth
Joint
F/T
Joint (deg)
RF (cm)
Obj (cm)
2.78+0.09
106=0001
3.65+0.10
1.38+0.05
9.53+0.20
3.22=0.14
Task (1)
2.39:#0.11
1001=0.02
2.69-+00001
1.09=0.03
1.91
0.64
2.1910009
0.75.10.11
7.49+0.06
3.81+0.17
Task (2)
5.47+0.08
3.32=0.04
5.24#0.04
3.04=0001
2.9310.01
2.26210.02
3.2611001
3.16+0.20
2.34=0.04
3.66#0.30
Task (3)
1.2++0.08
0.93+0001
1.47+0002
1.37++0.2
0.94+0.011
1.78+0.06
14110004
0.90=0.01
1.65+0.01
Mcanstsstadard errors.
and the Cartesian coordinates (I, y. of the robot's end-effector. This Cartesian coordinate sys- tem is centered at the manipulation platform's 's origin point (0, 0, 0). On the other hand, the state of the object being manipulated is represented by zo, which only includes the Cartesian coordinates (1r, y, z) of the object. The input modalities for each of the three tasks differ. task (I), input given through three modalities: y², y² and y The first modality y 12222x224xx is camera im- age captured from frontal angle. The second modality y 2R22x224x1 depicts depth maps from the same camera view. Lastly, y is proprioceptive input source with dimensions R2, representing the joint angles' values. In this task, the proprioceptive input specifically refers to the joint angles as the source. In task (2), input is given by only two modalities: y and y but from a real-world perspective. In task (3), input is received from four modalities: y²,y? y and y y contains the Force/torque (F/T) sensor readings from the robot gripper, where y E TR.. while the first two modalities are identical task (1).
Pick up the red
Put down the Pepsi
Push the red can to the left
Task
Task
Figure 10: The multimodal manipulation experiment involves the following subtasks: (a) Task I utilizing RGB, depth, and joint modalities, (b) Task 2 utilizing only RGB and joint modality. and (c) Task 3 utilizing RGB, depth. joint, and Force/torque (F/T) sensor modalities. The F/T sensor is mounted the grabber, depicted by the orange box.
Data: Data collection conducted for both simulation with MuJoCo [55] and real-world scenarios. We record the URS robot operating on random object by performing one of "pick", "push", and "put down" actions. We collect 2,000 demonstrations in simulation for task (1), and 100 on the real robot for task (2), with changing the location of each object for each demonstration. For task (3), we collect 2,000 demonstrations in simulation with adding the tactile sensors. We use ABR control and robosuite [56] in addition to MuJoCo to ensure rigorous dynamics in the simulator. Each demonstration sequence has a length of approximately 350 steps with timestep of 0.08 seconds. An 80/20 data split is utilized for training and testing each task. It should be noted that in all tasks, we normalize the joint modality y and apply Gaussian noise to each joint angle, drawn from the distribution N (0, o²I) where o² 0.1. We collect the F/T sensor readings directly from MuloCo's 's native touch sensor. Moreover, the depth maps obtained from MuJoCo are with no noise therefore can be regarded high-fidelity data.
18

---

B.2.1 Ablation Study
In addition to the findings presented Section 4.2, we perform a comprehensive ablation analysis for each manipulation task to address the question, "How does the use of multiple modalities com- pare to subset of modalities for state estimation with differentiable filters?". Table 8 displays the outcome for each task with various number of modalities using MAE metric. The highest margin of eπrоr is indicated by the red shading, while the complete modality is labeled by green shading for each task. Interestingly, even though using all modalities can generate comparable results, in certain tasks, utilizing all modalities does not necessarily guarantee superior performance compared to utilizing subset of modalities. Through our experiments Task (1), becomes apparent that the optimal performance is achieved by utilizing the subset of modalities [y', y"J. which yields an improvement of joint angles (2.19º + 1.91"). In Task (3), observe that diverse subsets of modalities lead to superior state estimation results for joint angles, EE, and the object locations respectively. Analysis of Table 8 indicates an important role played by the depth map ya when considering all observations. This suggests that y² is treated as high-fidelity data during training, thereby contributing the most towards the final results.
Henceforth, we conduct an additional ablation analysis to ascertain whether or not the use of a combination of high-fidelity and low-fidelity sensor inputs offers a potential benefit. As noted during data collection, the pro- prioceptive input yd comprising joint angles is obtained via adding Gaus- sian noise and is therefore considered low-fidelity input. Figure 11 illus- trates the scenario of using y and not using yd while applying distinct lev- els of Gaussian blur in the image and depth space. Notably, without em- ploying y the state estimation per- formance deteriorates the level of blur increases. On the other hand, y despite being classified low-fidelity modality contributes to the final state estimation. In particular, the highest level of blur, incorporating y yields a 29% improvement in joint angle estimation and 17% improve- ment for end-effector locations.
Propriocetion
150
No Propriuception BE (sm)
All Modaliy Joint (deg)
Eroor cate
125
Modalay BE (sm)
5.0
0.0
(5c3)
(tet)
(136113
(15615)
Conasian Bluringg
Figure 11: State estimation results are shown after introducing di- verse levels of noise to ly", y²]. The red group depicts results using ly y²1 modality, while the blue group represents results using (y*) y. y modality.
B.2.2 Sensitivity Analysis
In this study, we analyze the effects of three key factors on the performance of a-MDF. These fac- tors latent dimensions, the length of previous states, the number of latent ensemble members, whether using Transformers or Multilayer Perceptrons (MLPs) process model, and with with- out the matrix M. Our investigation focuses on understanding how these factors impact the overall performance of the a-MDF framework. In this experiment, we use robot manipulation task (1) as an example.
The findings from the sensitivity analysis are summarized in Table Regarding latent dimensions, observe that larger latent dimension does not consistently yield better error metrics. The op- timal latent dimension may vary for different tasks. Regarding the length of the previous state, we find that using X,-10:111 leads to more accurate results compared to using 3 This sug- gests that a longer history of states may not significantly contribute to estimating the current state. Therefore, we recommend using a smaller or medium window size for state transition models. As for the number of ensemble members, using a larger value for E. does improve accuracy. How- ever, is worth noting that increasing the number of ensemble members can result in larger state space, which may introduce inefficiency. In terms of utilizing Transformer-stee neural networks for process models, the results from Table 9 indicate an advantage for this approach, as indicated
19

---

Table 9: Sensitivity analysis within the a-MDF framework, focusing three factors: latent dimen- sions, length of previous states, and the number of ensemble members.
RGB
Depth
Joint
FIT
Joint (deg)
EE (cm)
G-MDF with 64 latents
2.54+0.06
0.87=0.04
a-MDF with 256 latents
4
2.1910.09
0.75100.01
o-MDF with 512 latents
4
4
2.38+00011
0.82+0.01
a-MDF with X 5:t-1
2.1910.09
0.7510.01
a-MDF with X 10:t-1
2.16110.66
0.83+0004
o-MDF -
V
2.72+00.33
0.79+0.007
o-MDF 25
2
v
2.67+0.12
1.10=0.02
o-MDF
2.19+00.99
0.75+0.01
o-MDF E 27
V
1.77++0.05
0.67+0.01
o-MDF with MLPs
2.4510.05
0.8310002
o-MDF with no
4
7.25+
2.23
M emror..
by the green-shaded row. However, it is important to acknowledge that for certain non-complex tasks, employing lightweight MLPs as process models can also be a suitable option. is crucial to consider the specific task requirements and complexity when deciding between Transformer-style neural networks and MLPs as process models.
B.2.3 Mask in Attention Gain
Within the attention gain module, we incorporate matrix M that selectively preserves diagonal elements of the attention map. This approach is based the assumption that within each latent vector, each index possesses probabilistic independence. To empirically verify this assumption, we conducted an additional experiment where we trained an alternative a-MDF framework. In this framework, we deliberately excluded the matrix M in the attention gain module for the specific purpose of evaluating the effect on robot manipulation task (1). The results our experiment are reported in Table as indicated by the red-shaded row. It shows significant increase in the Mean Absolute Error (MAE) for joint angle estimation when the causality- map M excluded from the attention gain module. Specifically, the MAE increased from 2.19 7.25%. Moreover, the MAE for tracking the end effector also deteriorated from 0.75cm to 2.23cm. Based these results, it is strongly recommended to utilize the gqusality-enforeee map M within the attention gain module for improved performance in both joint angle estimation and end effector tracking.
B.3 Soft Robot Modeling Tasks
This section presents comprehensive analysis of the tensegrity robot structure, the bending motion mechanism, and per- tinent sensory information, followed by a description of additional experimental out- comes related to this task.
MoCap
Intmslayer Actustor
Layer
Inter-byer Actustor
Layer
Cable
Layer
Layer
Preliminaries: Our research utilizes a
IMU:
tensegrity robot arm (developed in [45]) that follows strict tensegrity structure featuring struts, cables (including spring- loaded and actuated cables), and five lay- ers of arm-like tensegrity structures. which produce continuous bending postures when exposed to external forces. The longitudinal length is maintained by stiff cables, while the bending direction is solely determined by external forces. We determine the robot's 's kinematics through data from Inertial Measurement Units (IMUs), optical mo- tion capture (MoCap), and proportional pressure control valves, with each of the five struts in each
Layer
The tensegrity robot features 5 flexible layers, each a tensegrity module with struts, cables, and actuators.
Figure 12:
20

---

13.25
36.46
y(m)
1.0
0.5
0.3
0.0
-0.5
8
Pred
250
500
750
1000
1250
1500
1750
2000
Timestop
Figure 13: Predicted end-effector (EE) positions and quaternion vectors q in the soft robot modeling task. The top row displays the actual robot posture corresponding time, with the orange circle indicating the EE positions, which are not included the RGB modality input.
layer featuring an IMU. We also record the video by placing a camera in front of the robot while collecting all sensory data.
soft robot's state is a 7-dimensional vector x; [x, y,, qus by qus qwo indicating its po- sition and orientation relative to the base frame (layer I's bottom). q represents the robot's posture. The system's action is the pressure vector of its 40 pneumatic cylinder actuators (a, I2+4). Its raw observation is comprised of 5 IMU readings (y² E Re30), with each IMU measuring 6-dimensional vector of accelerations and angular velocities relative to its location. Fig. 12 illustrates the locations of the IMUs on the struts (blue cubes) in each layer.
The complete set of modalities comprises 1y² yy,,y1 where y 222x22243 represents RGB images, 2 E 2224x224 is synthetic depth maps which we generate from DPT repo [46] utilizing "Intel/dpt-largee. and 3 E R230 is proprioceptive inputs (IMUs). The dataset is generated by performing optical motion capture on the real tensegrity robot hand tip while randomly supplying desired pressure vectors to the pneumatic cylinder actuators. The action a, E R:40 IMU readings y? E 30 and 7-dimensional state X; are recorded, with 40-dimensional pressure vectors being used as control signal. total of 12,000 trials of robot motion are collected, with each trial involving moving the robot from its current equilibrium posture the next equilibrium posture by applying the desired pressure. All data collected via ROS2 network with sampling frequeney of 30Hz and are synchronized using the "message filters" package.
Data:
21

---

B.3.1 Ablation Study
In addition the results presented in Section 4.3, evaluate various com- binations of modalities to determine whether an optimal subset of modali- ties can be identified to attain compara- ble outcomes without using all modal- ities during the filtering operation. As demonstrated in Table utilizing only one modality fails to achieve compara- ble results, with the highest accuracy (2.07cm) exclusively from employing y' (RGB). The lowest error in pos- ture estimation for the robot is obtained by leveraging [y', y2], showing slight improvement (0.10~+0.13 over leveraging the full modalities [y', y yJ However, the lowest MAE error for the employed. Interestingly, using solely yd results in
Table 10: Ablation study on Tensegrity robot.
RGB
Depth
IMUs
EE (cm)
q(10')
2.07=00.33
0.31+:0.08
2.77=0001
0.19:00.55
8.99=0.02
0.79:1003
2.08+0003
0.14+0.02
1.73==0.05
0.12+0.02
1.74=10006
0,10
1.67=0009
0.12+0.01
Means--standard errors.
EE position persists even when all modalities the highest state estimation error, which aligns with the lowest attention value visualized in Fig 14. As depicted in Fig. 14, it is evident that o-MDF prioritizes y over other modalities. Interestingly, the attention values change upon turning off certain modalities while the system remains stable and functional.
Only IMUss
No RGB
Depth
Figure 14: The corresponding accumulated attention values for each modality during testing. The gray areas
show certain modalities selected not selected.
B.3.2 Concept Drifts
To investigate the effects concept drift and contextual changes [57] on the o-MDF framework, we incorporated background change at inference time. In particular, image blending is used to overlay different RGB picture into the background. The objective behavior when changes the environment occur. We evaluated the tracking performance at various blending levels, illustrated in Fig. 15. The results provide understanding of how effectively the a-MDF framework handles concept drift at different levels of intensity. It is noteworthy that despite substantially affecting the visual representation of the scene the achieved results (6.54cm) are comparable to utilizing only IMUs (8, (8.). This experiment provides an early insight into the utility of multimodality for mitigating the adverse effects due to contextual changes and concept drift.
experiment
inference
Accuncc
Eilicmi
0.1
0.7
Drift Intensity
Intensity
Intensity
Intensity
Figure 15: Concept drifts analysis by adding background change with scale in RGB space.
22

---

C Complexity and Training Details
In this section, we present an analysis of the computational complexity associated with each task by measuring the wall-clock time. Additionally, we provide comprehensive information regarding the model hyper-parameters and training curriculum employed for the experiments. These details offer insights into the computational requirements and settings utilized for training the models in our study.
C.1 Complexity
To assess the computational complexity of the proposed a-MDF framework alongside the baseline differentiable filters (DFs), we measured the wall-clock time during inference. The results, provided in Table 11, demonstrate the computational time for each approach. In the comparison with DFs baselines, only considered single modality. It is worth noting that in the multimodality setting, we observed only marginal increase in the elapsed time (0.03 sec) when handling multiple types of observations. This indicates that the proposed framework, a-MDF, is efficient and capable of effec- tively handling various modalities without significantly compromising computational performance.
Table 11: Wall-clock time (sec) for each task.
Visual Odometry
Robot Manipulation task(2)
Soft Robot task
Modality
task
(ask(1)
task(3)
dEKF [9]
0.0463:10.004
0.0469110.003
0.0472-00002
0.0474100.00
DPF [28]
0.0486+00005
0.0515++0.002
0.0509+0.002
0.0600+0.004
dPF-M-Im [9]
0.0693=0.011
0.085420.001
0.0844=0.002
0.0590=0.000
a-MDF
0.054710.002 0.0836+00002
0.055410.011
0.052410.003
0.0633
a-MDF
0.0873+0.004
0.0890+00000
0.0910++.000
Means-standard errors.
C.2 Training Details
Table 12 provides exhaustive enumeration of all learnable modules utilized in a-MDF, which includes three primary components: the state transition model fe. the sensor encoders [s ()), sM))), and the attention gain (AG) module. We adopt self-attention layers with dimension 256 and 8 heads, denoted as "Self Attn", in the state transition model. The cross-attention layers, denoted as "Cross Attn", with dimension 32 and heads in the AG module. The sensor encoders utilized in our approach and all baseline models are identical, with s acting on image- like modalities, utilizing ResNet18 [58] for learning high-dimensional observation representations, while s² pertains low-dimensional modalities such joint angles. The auxiliary model A and the decoder D shares similar structure to s², but with different number of neurons. Note that is the dimension of the actual state.
Table 12: a-MDF's learnable sub-modules. fo: 3x SNN(256. ReLu), Positional Embedding. 3x Self Attm(25688,, 2x SNN(256, ReLu), 1X SNN(d,.
s²: Ix Res.et18(h.w.ch), 2x fo(2048, ReLu), SNN(512, ReLu), SNN(d,.-)
s²: Ix SNN(128, ReLu), SNN(256, ReLu), SNN(512, ReLu), SNN(d,,-))
AG: Positional Embedding. 1x Cross Attm(32, mask)
A: 1x SNN(128, ReLu), 1x SNN(256, ReLu). 1x SNN(512, ReLu), 1x SNN(1024, ReLu), 1x SNN(d,.))
D: 1x fo(256, Rel 1x S88(128, Rel.u), 1x SNN(32, Relu), 1x SNN(r,
Ic: Fully Connected, SNN: Stochastic Neural network.
During a-MDF training, we employ the curriculum outlined in Algorithm 1. Note that some tasks may require pre-training the sensor encoders before performing end-to-end training the entire frame- work. For each task, we train a-MDF model with utilizing batch size of 64 on single NVIDIA A100 GPU for roughly 48 hours. For all the tasks, we use the Adamw [59] optimizer with learning rate of le-4.
23

---

Algorithm Condition in Latent Space: training algorithm return the weights w
Input:
a-MDF, dataloader (( (yr" weights while not converged do
Im=1, (y W1411m== (acli-1 [+1)
Output:
Call dataloader with a random timestep t.
timestep f- & è + do
61 - -m=1 [D(s th (yr")) xellē according to Eq. 6
t L (X,) + L2 (X,) according to Eq. 6
Et f E1 6
end for
+ Train (a-MDF, et + ex41)
end while return w
24


Reviews:
[{"review_id": "vfe80wzFj3O", "reviewer": "Reviewer_X85h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces an attention-based multimodal differentiable filter (α-MDF), which combines differentiable filters with attention mechanisms from transformer models for improved robot state estimation. The α-MDF framework extends the capabilities of traditional DFs by replacing conventional Kalman gain with a neural attention mechanism, effectively generating specialized, context-dependent gains that can efficiently merge multiple input modalities and observed variables. It operates in the latent space, learning high-dimensional representations of system dynamics and capturing complex nonlinear relationships. The paper empirically validates the α-MDF framework on various robot state estimation tasks. The results demonstrate reductions in state estimation errors, with nearly 4-fold improvements compared to other sensor fusion strategies for rigid body robots. Additionally, α-MDF outperforms differentiable filter baselines by up to 45% in soft robot modeling tasks.", "summary_of_recommendation": "The paper presents an attention-based multimodal differentiable filter (α-MDF) for robot state estimation. The proposal to use attention mechanisms to replace the traditional Kalman gain is an innovative approach, and the empirical results show promising improvements over existing state-of-the-art methods on various tasks. The writing is clear, and the structure is coherent, making the content easy to follow. However, the paper can be further improved in some areas. It has a few typographical errors and thus requires another pass of proofreading. The authors should provide a comprehensive comparison in terms of computational complexity with other models. It would be beneficial for the authors to provide details on the computational resources used for the experimental setup, detailed sensitivity analysis, ablation studies, details on hyper-parameter tuning, more insights into how they intend to address the limitations, and the scope of future research directions for their work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "It would be great if the authors could address the areas of improvement mentioned in the above section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents the α-MDF framework, an attention-based multimodal differentiable filter for improved robot state estimation. The proposed approach leverages attention mechanisms, commonly used in transformer-style neural networks, to learn multimodal latent representations. It substitutes the traditional Kalman gain with a neural attention mechanism for filtering with multimodal observations. This allows the model to learn a highly specialized and task-specific gain mechanism.\n\nThe α-MDF framework consists of a transformer process model, sensor encoders, and an attention gain update model. Each module is learnable and operates in the latent space, learning high-dimensional representations of system dynamics and capturing intricate nonlinear relationships. The modular architecture of α-MDF enables it to handle missing sensor modalities, thus improving the model's resilience and robustness. The paper is well-written and well-structured, providing a clear and detailed explanation of the proposed α-MDF framework.\n\nThe authors have empirically validated the α-MDF framework on various robot state estimation tasks in real-world settings and simulations. The results demonstrate that α-MDF significantly reduces state estimation errors with nearly a 4-fold improvement compared to current state-of-the-art sensor fusion strategies for rigid body robots. In addition, it outperforms differentiable filter baselines by up to 45% in soft robot modeling tasks.\n\nHowever, the paper can be further improved in the following areas:\n1. The paper has a few typographical errors and requires another pass of proofreading. Some examples are as follows:\na. On page 6, in the last paragraph, the l in \"dPF-M-Lrn\" is in uppercase. However, in all other instances in the paper, l is written in lowercase.\nb. The description of Figure 1 says \"Filters\" instead of \"Filter\".\n\n2. The paper lacks a comprehensive comparison in terms of computational complexity with current state-of-the-art models.\n\n3. The paper lacks details on the computational resources used for the experimental setup.\n\n4. It would be beneficial for the paper if the authors could provide detailed sensitivity analysis and ablation studies to demonstrate the effectiveness of the different components of the proposed framework.\n\n5. The authors should provide more details and guidance on hyper-parameter tuning and its impact on the performance of the model.\n\n6. The authors have mentioned limitations at the end of the paper. It would be great if they could provide more insights into how they intend to address these limitations and the scope of future research directions for their work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "pY9YBDcPMJ", "reviewer": "Reviewer_UVgt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a data-driven filtering process which eliminates the need for an explicit analytical model of the system of interest. The attention-based Multimodal Differentiable Filter ($\\alpha$-MDF) replaces the traditional gain matrix in the filtering process with an attention unit and a learnable query, which propagates through both the latent state and observations to output an attention gain used to update the estimate of the (latent) state. The flexibility of the attention mechanism also allows efficient fusion of multimodal observations which the authors show empirically, leads to superior state estimation performances compared to recent state-of-the-art.", "summary_of_recommendation": "While there are some issues that needs to be cleared up, the general concepts are sensible and the overall idea is interesting. The superior performances seen in the experiments further show that the paper is deserving.\n\nI am satisfied with the rebuttal by the authors.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. The final loss function proposed is the summation of three different losses, namely the state transition loss ($L_{f_\\theta}$), end-to-end loss ($L_{e2e}$), and latent observation conditioning loss ($L_s$). L2 loss between the decoded latent and the ground truth state, $\\mathbf{x}_t$ is used for the three sub-losses. While the first two losses are sensible (i.e., matching the decoded prediction with the ground-truth and matching the decoded estimation with the ground-truth), the final sub-loss seems perplexing. By using the loss $L_s = ||\\mathcal{D}(s^m(\\mathbf{y}_t^m)) - \\mathbf{x}_t||_2^2$, the neural network attempts to match the (decoded) latent observation $\\mathcal{D}(s^m(\\mathbf{y}_t^m))$ to the ground-truth state (e.g., matching a latent state derived from depth image to the ground-truth state). Is such a loss possible? For example, in the multimodal manipulation task when $\\mathbf{y}^4$ is the force/torque sensor reading. Does the observation necessarily contain sufficient information to derive ground-truth state?\n\n2. In the attention gain module, a matrix $\\tilde{\\mathbf{M}}$ is introduced to retain only the diagonal elements of the $d_x \\times d_x$ attention map in support of the argument that \"within each latent vector, every index is probabilistically independent, and index $i$ of a latent state should only consider index $i$ of each latent observation\". Is there any evidence or prior works to support this claim? The claim that \"$\\tilde{\\mathbf{M}}$ can be disabled based on different input sensor modalities, thus improving the model's resilience to missing modalities\" also seem unconvincing as it is difficult to relate how the removal of $\\tilde{\\mathbf{M}}$ makes the model resilient to missing modalities.\n\n3. In the multimodal manipulation task, $\\alpha$-MDF outperforms baselines dEFK and DPF in the estimation of the end-effector positions. It seems that only the RGB modality is used for dEKF since the DF baselines only take one modality as stated and the experiment (Table 1) does not consider sensor fusion techniques. Does $\\alpha$-MDF only consider the RGB modality in this case? In the  subsection on task setup and data, it seems to imply that $\\alpha$-MDF takes in more than just the RGB observations. If that is the case, then the comparison of results will not be a fair one since the $\\alpha$-MDF takes in additional information when making the estimation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is generally well written and clear in its presentation of concepts. While the use of attention mechanism is common in many recent machine learning literature, its use in replacing the gain matrix in the filtering process is a simple yet clever idea.\n\nStrengths:\n1. The introduction of the attention mechanism to replace traditional gains is refreshing. It allows efficient context-dependent prediction of the gain directly (during inference) without explicit understanding of the system-specifics (e.g., observation matrix, noise covariance matrix) since the system is modeled as a neural network in a data-driven manner.\n\n2. Additionally, the design of the attention mechanism which allows variable input length means that for different tasks and system of interest, the operators have the flexibility to consider different input modalities using the same architecture. Such a consideration is not viable for the baselines presented in the paper, and an additional fusion strategy is necessary.\n\n3. The empirical results presented in the paper shows that the approach is vastly superior to existing state-of-the-art and the simplicity of the proposed neural network model makes it even more attractive.\n\nWeakness:\n1. While the author considers operating within a latent space as a primary contribution of the paper (contribution 2), it is hard to claim so. Arguably, every neural network model, whether it is an expansion from a low to high dimensional space or a compression from high to low dimensional space, works with a latent space and the use of neural networks to model nonlinear relationships is not unique to this work.\n\n2. Little information is given for the transformer-style neural network $f_\\theta$, presented in the prediction step. Additional information can be provided to allow a better understanding of its implementation especially when $f_\\theta$ takes in not only a sequence of previous states $\\mathbf{X}_{t|t-N:t-1}$ but also an optional action $\\mathbf{a}_t$ which could be of a different dimension as seen in the experiments presented. While $f_\\theta$ is not a main focus in the paper, the prediction step remains a crucial component of the filtering process and its implementation should be clearly presented. The authors can provide a more detailed implementation in the appendix if the length of the paper does not permit its inclusion.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0NIzC1Ti1u4", "reviewer": "Reviewer_zXri", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a differentiable Bayesian framework for robot state estimation. The key contributions include attention gain with a learnable queue vector and the formulation of latent space filtering. The proposed method is formulated in an ensemble fashion, where multiple states are propagated and corrected at once to fuse for the best result. It is evaluated on 3 different robotic applications, including visual odometry, manipulation state estimation, and soft robotic state estimation. The experiments show the proposed method achieves higher accuracy than the baselines. \n", "summary_of_recommendation": "Overall, this paper makes the following two contributions: 1) a differentiable Bayesian filter that operates on the latent feature space. 2) A attention-based gain module that allows fusions of features on different spaces. The experimental results show the proposed method achieves higher accuracy than the baselines. As a result, I’m suggesting a weak accept. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Issues\n1. I might have missed something, but for the manipulation task, why do joint angles need to be estimated as a state? Especially the joint angle measurements are used as an input to the network. The encoder measurements are usually fairly accurate already. I’m confused about why this is needed.\n2. In addition to the above point, the end-effector position can be computed through the forward kinematics function, although the kinematic modeling can sometimes be inaccurate. I wonder how the performance of the proposed method compares to pure forward kinematics. \n3. Since one of the main contributions of the proposed method is the attention gain, I’m interested to see how the proposed method works compared to a non-learning Kalman filter in cases where a traditional formulation is feasible. The Kalman gain is supposed to be the optimal gain under the Gaussian assumption and linear models. I wonder if the proposed attention gain can achieve similar optimal results when the problem is less complicated and well-constrained. \n4. Just out of curiosity, how are the sensors disabled for the experiment shown in Figure 5 (b)? It looks like the sensors were disabled on the fly during continuous inferencing. Was it done by setting the specific input matrix/vector to zero?\n5. In Kitti odometry benchmark, it’s a common practice to estimate the 6D motion instead of x, y, and yaw, as the car might go up and down hills. I’m curious to learn how the proposed method works when the search space becomes larger. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n1. The proposed method operates on the latent space, which opens up the possibility of fusing sensors where the measurement model is obscure (e.x. RGB images). \n2. The attention-based gain enables the filter to adjust the gain even when features projected from the measurements live in different spaces. \n3. The proposed method was evaluated on 3 distinct applications, and all obtained better results than the baselines. \n4. Figure 5 (b) shows interesting qualitative results on the proposed attention value. The network is able to reduce the attention value when a specific sensor modality is disabled. \n\n\n## Weaknesses\n1. The proposed method seems heavily relying on vision sensors. From the ablation study, the IMU doesn’t seem to add much value to the system. \n2. The organization of the paper can be improved. Currently, it’s a bit difficult to get a sense of the experimental setup without heavily referring to the appendix. \n3. In section 3.1, lines 134-137, the paper states that the traditional Kalman gain is designed to handle a single observation space. I believe this is a bit inaccurate and an abuse of notation. How is the observation space defined here? The traditional Kalman filter does not restrict the measurements to be in the same measurement space (ex., We can simultaneously use velocity measurements and angular velocity measurements to correct the filter.) However, it is required that the function that maps from the state space to the measurement space be explicit. (As a result, some sensor measurements like images or deep-learned features are unable to be used in the formulation directly.) The proposed attention gain relaxes the need for an explicit measurement model.\n4. Although mentioned in the limitation section, there’re no experiments demonstrating how well the proposed method works in cases of concept drifts. What happens if the surrounding environment changes drastically for the tensegrity robot experiments?\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an attention-based methodology for robot estimation that eliminates the need for an explicit analytical model of the system. 

The reviewers have found that the approach is interesting and valid, and also the articila is well written. The rebuttal has better clarified on the points the reviewer pointed out for improvement.
Decision: Accept (Poster)