Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL
Authors: Yangru Huang
Peixi Peng
Yifan Zhao
Guangyao Chen
Yonghong Tian
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Accurate environment dynamics modeling is crucial for obtaining effective state representations in visual reinforcement learning (RL) applications. However, when facing multiple input modalities, existing dynamics modeling methods (e.g., DeepMDP) usually stumble in addressing the complex and volatile relationship between different modalities. In this paper, we study the problem of efficient dynamics modeling for multi-modal visual RL. We find that under the existence of modality heterogeneity, modality-correlated and distinct features are equally important but play different roles in reflecting the evolution of environmental dynamics. Motivated by this fact, we propose Dissected Dynamics Modeling (DDM), a novel multi-modal dynamics modeling method for visual RL. Unlike existing methods, DDM explicitly distinguishes consistent and inconsistent information across modalities and treats them separately with a divide-and-conquer strategy. This is done by dispatching the features carrying different information into distinct dynamics modeling pathways, which naturally form a series of implicit regularizations along the learning trajectories. In addition, a reward predictive function is further introduced to filter task-irrelevant information in both modality-consistent and inconsistent features, ensuring information integrity while avoiding potential distractions. Extensive experiments show that DDM consistently achieves competitive performance in challenging multi-modal visual environments.

Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL
Yangru Huang', Peixi Peng2.3, Yifan Zhao Guangyao Chen', Yonghong Tian 1,2,+
"Scoool of Computer Science, Peking University
Schooo of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University $Peng Cheng Laboratory
4 School of Computer Science and Engineering, Beihang University,
yrhuangdatu, pku. edu.edd..com en, fpxpeng, gy. ghen, yhtian)&pkuu edu, cn, zhaoyf @ buaa.edu.cn
Abstract
Accurate environment dynamics modeling is crucial for obtaining effective state representations in visual reinforcement learning (RL) applications. However, when facing multiple input modalities, existing dynamics modeling methods (e.g., Deep- MDP) usually stumble addressing the complex and volatile relationship between different modalities. In this paper, we study the problem of efficient dynamics modeling for multi-modal visual RL. We find that under the existence of modality heterogeneity, modality-correllter correlated and distinct features are equally important but play different roles in reflecting the evolution of environmental dynamics. Mo- tivated by this fact, we propose Dissected Dynamics Modeling (DDM), a novel multi-modal dynamics modeling method for visual RI.. Unlike existing methods, DDM explicitly distinguishes consistent and inconsistent information across modal- ities and treats them separately with a divide-and-conge conquer strategy. This is done by dispatching the features carrying different information into distinct dynamics mod- eling pathways, which naturally form series implicit regularizations along the learning trajectories. In addition, reward predictive function is further introduced to filter task-irrelevant information in both modalltt-cooniiseen and inconsistent features, ensuring information integrity while avoiding potential distractions. Ex- tensive experiments show that DDM consistently achieves competitive performance in challenging multi-modal visual environments. The code is available in this link: https://github.com Yara- HRR-HHYR/DDM
Introduction
Recent years have witnessed growing interest in visual reinforcement learning (RL). By leveraging raw pixel inputs directly from sensory data, visual RL allows agents to navigate and interpret their
environments with unprecedented precision and adaptability [33 24 5T] Since raw pixels are usually redundant and suffer from expanded data dimensionality, a key aspect of visual RL is to acquire typical approach involves modeling environmental
efficient and compact state representations. dynamics in an abstracted state space [T] where the state representations are supposed to mimic the actual environmental changes but operate on sufficiently low dimension. One classical solution is DeepMDP (9), which leverages neural networks to compress high-dimensional input data. Given consecutive observations, DeepMDP predicts reward and state transitions within the deep latent space, enabling more effective policy optimization over complex, dynamic environments.
"Corresoodingg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RGB frames
signals
Modality 1
Mi
Red light
RGB Incousistency
Event Inconsistency
ac
Consistent Context
M
1841
Observed Informaion Ation Takse
T
Consistancy Information Only-Tum Jeft X
Modality
RE13
Consistency Inconsistency
Brake
Con.
Incon.
->
Predict +- rthogunalize
(a) Complex modality relationships visual RL
(b) Comperison between DeepMDP (left) and our method (right)
Figure I: Motivation of our method and comparison with DeepMDP g designed for single modality. Ot, Zt, at and r, represent the observation, latent feature, action, and reward at time 4, respectively. zM, and 2n are the decoupled consistent (Con.) and inconsistent (Incon.) features for i-th modality. 2t
Although methods like DeepMDP have proven effective for environmental dynamics modeling, research on this subject has been mostly restricted to the single visual modality case (e.g., with only RGB frame pixels as input). However, under multi-modal scenarios, different modalities may share correlated scene descriptions but meanwhile preserve their own unique observations. In addition, such a modality relationship is not stable but constantly evolves with the highly dynamic environment. An intuitive example is given in Fig. [ (a). where we show the observations of both RGB frames and event camera signals (2 3 on a night-time driving scene. It can be seen that the two modalities only exhibit partially consistent context. Meanwhile, their inconsistencies include critical visual clues, such as traffic lights (in RGB frames) and cars hidden in the dark (in event signals). Existing approaches often overlook this complex and volatile status caused by modality heterogeneity. which may have detrimental impacts on learning correct environmental dynamics. straightforward solution might be to model the dynamics of each modality separately. However, this approach neglects the complementarity between different data sources, which could reduce its effectiveness.
To achieve efficient dynamics modeling in multi-modal scenarios, decoupling and understanding the interplay between modalities and environmental dynamics is necessary. Firstly, modality-correlate correlatee features provide foundational perspective by capturing shared and common information across different sensory inputs. These commonalities are critical for building cohesive description of the environment's 's overall behaviour. Secondly, the distinct features in each modality offer inconsistent side views of the environment. In conventional multi-modal visual tasks, these inconsistencies are typically deemed less critical and are mitigated through modality alignment [5] 40 27 However, in modeling RL dynamics, the inconsistent features are equally important as they 1) carry critical task- related clues and 2) are also the key ingredient in deducing the modality-speciff state changes. This importance also verified by our investigation presented in Sec. 4.4 where we find that incorporating modality inconsistencies leads to more accurate mimicry of the environment rewards.
Based on the above analysis, we propose Dissected Dynamics Modeling (DDM), a method that neatly integrates the decomposition of commonalities and differences across modalities with the dynamies modeling process. As shown Fig. bb). DDM seeks a mutual predictive property between modalities to establish an implicit regularization and isolate Furthermore, it extends this consistency across different temporal locations, ensuring that the learned common information resonates with environmental dynamics. Regarding modality inconsistencies, unlike standard practice that enforces mutual exclusivity with consistent features of the same modality, DDM imposes orthogonal constraints between different modalities to highlight their unique contents. This avoids overly strong regularization, which leads to improved feature quality. The decomposed inconsistent features are only used to establish the dynamics of each modality itself, which creates a more complete environment dynamics without causing interference across modalities. Despite dissecting modality content and model dynamics separately, DDM further introduces reward predictive function to ensure that both consistent and inconsistent features focus on relk-atl information. This approach preserves information integrity while avoiding potential distractions, which ensures optimized decision- making accuracy and responsiveness.
consistent features.
In summary, our work contributes threefold: 1) We analyze the challenges of modeling multi-modal environment dynamics for visual RI., offering insights into the key factors that influence modeling robustness. 2) We present DDM, novel method that seeks commonality but preserves differences

---

between modalities for enhanced dynamics modeling. 3) Our experimental results validate the effectiveness of our method, demonstrating its strength under complex environmental conditions.
2
Related Work
Visual Reinforcement Learning, Visual reinforcement learning (RL) aims to enable agents to make decisions based on raw visual inputs. Existing methods typically revolve around strategies such as data augmentation to enhance input diversity 50
29
30
31], incorporating auxiliary tasks for richer
representation supervision 23
53
19
35
46),
employing world models for behavior prediction [14
13 38 [49] or pre-training encoders to improve state representation compression (28 (9 [1]. Despite
these advances, current visual RL, methods predominantly rely on the RGB modality, which could potentially limit the agent's holistic understanding of the environment.
Multi-modal Reinforcement Learning. Handling multi-modal input is important for various control tasks [8] 44 22 but remains as an under-explored research area in the field of RL. Despite the advancements in multi-modal machine learning methods, applying these techniques to RL faces several challenges due to the highly dynamic RL environment and heterogeneous modalities, A few pioneering works have focused explicitly on multi-modal RI. methods primarily focus on aligning different modalities, using techniques like mutual information optimization or imposing consistency constraints [32]. Such a strategy overlooks the impact of modality-inconsistent aspects on decision-making, which may hinder its effectiveness.
20 However, these
Dynamies Modeling in Visual RL. Learning model the environment dynamics in visual RL is critical for obtaining efficient state representations and reducing computational complexity. Earlier Subsequent studies have
works involve reconstructing input observations at the pixel level 48
45
focused on dynamics modeling by predicting future latent states 10 9
14
37
25
2 411, showing
promising results. Although these methods have advanced the field, they are not explicitly designed for multi- scenarios and may face challenges under complex modality relationships. Unlike existing methods, our approach uniquely dissects modality contents based on their relationships, treating consistent and inconsistent information separately within the dynamics modeling process.
3 Methodology
We consider the standard RI. setting where an agent interacts with the environment consecutive time steps, At each time step. the agent has access to observations from d distinct modalities, where d >> 1. Fig.P illustrates the proposed Dissected Dynamies Modeling (DDM) method. DDM structured around two tightly entangled elements: 1) the decomposition of commonalities and differences across modalities and 2) modeling modality-aware environmental dynamics. Through this modeling process, the decomposed features of each modality are optimized to retain highly abstract and comprehensive task-related information. These features are then applied to learn policies, which promote both socision-making precision and sample efficiency. We now describe our overall approach in detail.
3.1 Preliminaries
Multi-modal Markov Decision Process. The task of multi-modal visual RL can be formulated as
Markov Decision Process with the tuple (S, A. T, RR,)), 7), where the state space is S I (=1 O. represented by the combined observation space comprising d different modalities 20 A is the
action space, T(s+11 |s, 04) is the transition function, R(s, a) is the reward function, and y E [0. 1)
the discount factor [42]. At every time step I, the agent observes all modality data to take action according to policy π. The environment then returns a reward to the agent. The goal is to optimize the policy by maximizing the expected cumulative reward.
Soft Actor Critic. Our approach is based on Soft Actor-Critic (SAC) 12) which iteratively
refines policy function æ and critic function Q. SAC seeks to maximize the expected cumulative reward while simultaneously encouraging exploration by an a-discounted maximum entropy:
L. Ea, ~ [Q oz) a log (aaloo)],
(1)
The parameters of the value function are updated by the Bellman equation:
LQ E ((o.)))>DD (aaei)iDD((Q( [(Q(ot: ox) (re (V(o++1))))
(2)
3

---

Obscrvation Encoder
Modality Dynamics Modeling
Policy Learning
RGBB-
O dlc
Value network
RGB
a2
Mi
rnines
ROBl-Con
Every: Con
0.00
Policy network
livert
Signals
Everl Inoon.
Figure 2: The proposed Dissected Dynamics Modeling (DDM) method. The input visual modalities are first passed through separate observation encoders to get partitioned poddaltty-oonsiseen and inconsistent features. These features then undergo modality-aware dynamics modeling to model accurate RL dynamics and promote feature quality. Finally, the optimized modality features are merged as state representations, which are then used for robust policy learning.
where D the replay buffer, and the soft state value function V(o++1) is calculated as follows:
V(0+1) Rayssw [Q(0++1: ā(+1) - a log 1 (0c++100++)1
where Q is the exponential moving average of the critic network Q, and ăe+1 is the action produced by the current policy.
3.2 Modality Decomposition
Feature Encoding and Partitioning. Given the significant disparities in low-level information inputs across different modalities, we utilize d modality encoders to transform high-dimension observations into deep latent features. These encoders possess identical network architectures, except for the initial input layer, which is adjusted to match the channel number of the corresponding input data shapes. The extracted feature of the i-th modality Mj at time step t is denoted as 2 Since our method aims decompose each modality into common and unique components, two separate network branches as information filters for feature partitioning. These two branches process the extracted modality feature 2; MM; producing two distinct outputs: 2, as the modality- consistent feature, and a.M. as the boodality-inconsistee feature. In the subsequent decomposition phase, these features undergo a series of regularizations ensure they accurately convey the intended information. For clarity, we mainly introduce the case involving two modalities. The extension to multiple modalities will be discussed subsequently.
further adopt
Extracting Consistencies Between Modalities. Consistent information across modalities provides foundational knowledge about general environmental behavior. To extract such information, we focus on enhancing the mutual predictive properties between different modalities. Given two modalities Mj and M2, we first impose mutual prediction (mp) constraint the partitioned modality-consisten features Mi and zyMa at the current time step t:
Emp = [1 MM |||| zMaa
where Fi and F2 are prediction heads implemented as Multi-Layer Perceptrons (MLP). Such prediction between modalities triggers a form of content alignment, offering immediate common environmental insights. However, merely imposing the consistency constraint at a static step may not fully leverage the pivotal temporal dynamics of the environment. Intuitively, observations in adjacent timesteps often preserve similar contexts. For example, autonomous driving, a car that exists in one timestep will likely remain in the next. Meanwhile, there may be subtle variations in details, such as the car's location, distance, and size. These temporal differences can serve as augmented views that facilitate the discrimination of modality commonalities. Therefore, we create an additional transition prediction constraint across different modalities. Given ZiMi zM, and the zt

---

action taken in time step t, we set up two transition prediction (tp) heads G1. G2 to predict the My Mg of the next time step: 2 1. 2x41
consistent features
Lp = -b (Z or) 2 (G((z,)),,)) =
(5)
This approach creates synergistic effect on both consistency extraction and dynamics modeling: on one hand, the expanded view in the subsequent time step offers enhanced guidance for learning to identify task-relevant common objects. On the other hand, predicting environmental transitions improves feature quality related to modeling scene dynamics. This interconmection tightly couples the two tasks, allowing them reinforce each other effectively.
Identifying Inconsistencies. Extracting inconsistent information between different modalities is a subtler task than consistency extraction. This subtleness is caused by the absence of clear constraints on feature content: unlike consistency extraction, which involves pulling features closer with concrete objective functions, inconsistency extraction lacks such definitive goal. For initial consideration, any content that is not consistent across modalities should be deemed as inconsistencies. Therefore, a common practice is to enforce mutual exclusivity between the consistent and inconsistent features of the same modality [21]. Taking modalities M1 and M2 as examples, the orthogonality objective can be applied to impose this mutual exclusivity follows:
C=t 2/ (z 22
where (.) denotes the dot product operation. However, this objective might be overly strong, Eq. forces the feature pairs i 2 and [z 2aM²] to be completely uncorrelated. Since each pair of features originated from the same modality encoder, they are likely to share similar encoding patterns. Therefore, enforeing orthogonality may lead to excessively departed features and decrease feature expressiveness. To alleviate this issue, we take an alternative solution. Given two modalities M, and M2, instead optimize the following objective:
Corth (2 2Ma)2
Eq. 7 directly matches our intention of highlighting different contexts across modalities. During optimization, the similar contexts in M2 and M2 are gradually excluded aM and & due to the strict constraint of orthogonality. Meanwhile, this optimization process is less aggressive than Eq. since 2,Mi and 2,Ma come from different modality encoders and minimizing their correlation does not affect learning effective representations within each encoder.
Extension to Multiple Modalities. So far we have been working the two-modality case. The extension to more modalities is also straightforward. For consistency extraction, we modify the prediction heads F; and G, for modality M, to take all other modalities as input. The objectives in Eq. and 5 then become:
Lmp = Σ 11 11
z,Mi 11 Ltp = Σ |IG.(fz) 1 a), - -
(8)
where is the modality number and { | + i] denotes all modality-consistent features except for M.. These - features are concatenated and sent into F; and G; for prediction. Eq. is inspired by set calculation, where each feature is regarded
set of contexts. It states that for sets, the union of any sets equals the remaining set, then all d sets must be identical and consistent. For inconsistency extraction, we simply extend Eq. 7 as:
d-1 d
Corth
(&
2. )2,
j-141
which covers all pair-wise modality combinations.
3.3
Modality-aware Dynamics Modeling
After decomposing modalities into consistent and inconsistent contexts, our approach treats them separately in the overall dynamics modeling process. The consistency features are used to predict cross-modality future states to ensure they capture common scene dynamics, as formulated by the
5

---

Table 1: Comparison with state-of-the-arr methods on CARLA benchmark. S-RL denotes single- modality RL methods, M-CV denotes multi-modal methods for conventional computer vision tasks, and M-RL denotes multi-modal RL methods. ER represents the episode return, and D(m) is the driving distance in meters. The best results are bolded.
Normal
Nidnight
Bainy
Average
Mathoda
Typa
Dim)
ER
Dim)
ER
Dim)
ER
Dim)
ER
Dim)
SAC
S-RL
204 49
253 35
1.54 58
207 +
125 57
181 39
174 #:42
217 = 34
164.25
214.50
179 4:42
DespMDDP
162
267
SPR
S-RL
256 4:5
297 53
205 67
252 48
165 62
220:1449
261 1:33
280
221.75
257.25
TransFuser
M-CV
23
286
124 58
20
57
2
70
274
224.75
259.00
EFNet
212
MuMMI
M-RL
233 70
58
29
258 3.
159 6:
206
228 =61
261 49
208.75
253.50
MAIE
M.RI
241 + 58
291 45
217 61
242 411
163 58
210++45
242 52
274 +0
215.75
254.25
HAVE
237
Ours DDM
M.R
28
33
27
300
229
4.77
25
314
272.75
304.75
Lip in Eq. 5 and 8 The inconsistent contents are sent into a different pathway, which is used to
deduce the full modality-speciffe evolution. In particular, for each modality M,, we first merge its consistent and inconsistent features into a unified modality representation:
(10)
the merged 8/44. contains a comprehensive description of M, Then, we set up a full-state predictive (fp) head P; for each M,, which takes both zMM and action ar to forecast the next state:
fi Σ (PP(E a.)
MM, -
2++1ll
(11)
Such a full-state prediction ensures the combination decomposed modality features faithfully captures the modality-speciff dynamics. To accurately reflect environmental rewards, reward predictive function R is further introduced to filter task irrelevnnt information, optimized by:
C. [R ae) T++1 ||,
(12)
where 2, s.M zMaa
is the ultimate state representation obtained by concatenating features from all modalities, and r++1 is the actual reward returned by the environment.
2, My
3.4 Policy-learnine with DDM
Given the feature decomposition and dynamics modeling processes, the policy-learning of our dissected dynamics modeling (DDM) method involves the joint optimization of multiple objectives that are divided into three groups:
LDDM fq + Lx + Lmp + Lorth + Ltp + Lip + L.
(13)
SAC
Feat. Decomposition
Dynamics Modeling
where Ltp is used in both feature decomposition and dynamics modeling, and the SAC takes the state representation 2; input to replace the observation or. During training, the DDM learns to extract effective state representations that faithfully reflect the environmental behaviour, yielding improved policy in various modal visual RL tasks.
4 Experiments
4.1 Experiment Settings
Environments. To evaluate the effectiveness of our approach in complex multi-modal scenarios, we conducted tests on two widely used RL benchmarks: the CARLA simulator (7) for autonomous driving, and 2) the DeepMind Control (DMControl) suite 43 for robotic control. For CARLA, we as- sessed our method under four different environmental conditions: "Normal", "Midnight", "Dazzling". and "Rainy". The latter three conditions provide challenging driving scenarios characterized by

---

Table 2: Comparing with state-of-the-an methods on DMControl. The best results are bolded.
Nethods
Type
Cartpole Swingop
Reacher Easy
Cheetah Run
Average
Cartpole Seingup_ Epprse
Reacher Hard
Quadruped Run
Average
SAC
S-RL
742
307 279
201 137
320.00
DrQ
S-RL
815 =58
913 +54
239 #75
655.67
674 =67
105 145
469 #46
416.00
DeepMDP
S-RL
827 266
823 489
524 140
724.67
713 =51
128 1:32
453 1:2
431.33
SPR
S-RI.
845 +49
905
777.00
+:88
TransFuser
M-CV
845 #36
906 +48
518 =52
756.35
728 =42
169 +51
211 #75
369.33
EFNNt
M-CV
824 143
900 235
555 434
759.67
769 =55
125 157
183 181
359.00
MuMMI
802 +59
895 +57
742.33
712
152
328
MATE
M-RL
829 #00
852 +52
535 #61
738.67
750 =37
147 +58
402 +69
433.00
HAVE
M-RI
835 452
867 #65
560 +59
754.00
732 #45
151 +69
426 +58
436.33
Ours DDM
M-RL
844
903 342
587 232
781.67
813
224 164
489 256
508.67
extreme lighting and heavy rainfall, closely mimicking real-world driving scenes. Various modalities provided by the benchmark are utilized, including RGB frames, event signals, depth images, and LiDAR images in bird's s-yye view (BEV) perspective. For DMControl, we conduct evaluations on six tasks, which are partitioned into two groups based on their difficulty levels. The modalities used are RGB frames and depth images. However, given the relatively clean backgrounds of tasks in DMControl and the high consistency between the two modalities, we additionally introduce inconsistencies by randomly selecting one modality and masking out 20% of its image contents. This intentional introduction of inconsistencies simulates the occlusion issues commonly encountered during wal-world machine operations, thereby increasing the task difficulty.
Compared Methods. We compare our DDM method with various types of methods, including single- modal baseline RL algorithms SAC [12] and DrQ [50], two competitive dynamics modeling methods DeepMDP [9] and SPR (37), conventional multi-modal methods EFNet (40) and Trans [5], and multi-modal RL methods MUMMI 4. MATE (32), and HAVE (0) For EFNet and TransFuser, since they were initially designed for traditional computer vision tasks, we mainly evaluate their core modality fusion and alignment modules. These modules are integrated with the SAC algorithm for decision-making which is the same base RL algorithm used in our method.
Implementation Details. Our approach is implemented with Pytorch. To ensure fair comparison, we use the same encoder network architectures and training hyperparameters for all methods being evaluated. In line with established practices [50 [33 we convert data from each modality into image- based representations and stack multiple sequential images to capture temporal dynamics. Methods on the CARLA benchmark and DMControl are trained for 100k and 500k frames, respectively. To obtain reliable results, we train each task-method pair for runs and report the mean values and standard deviations after 20 evaluations. More detailed experimental setups are given in the Appendix.
4.2 Comparison with State-of-the-art
Results on CARLA. Table I presents the performance of various methods on the CARLA benchmark using two modalities: RGB frames and event signals. The results demonstrate that our method outperforms others in all four environmental conditions. The table shows that the performances of the SAC and DrQ baselines are less satisfactory, especially in complex conditions such as "Dazzling" and "Midnight". While conventional dynamics modeling methods like DeepMDP and SPR do enhance performance, their improvements are modest compared to our DDM method. For example, DDM obtains an average episode return improvement of approximately 24.4% and 21.3% over DeepMDP and SPR, respectively. This substantial enhancement is attributed to the explicit modality decomposition and processing in DDM, which significantly boosts robustness across diverse multi- modal environments. The table also indicates that existing multi- modal methods achieve decent performance on CARLA. However, these methods primarily focus on modality alignment and fusion, neglecting inconsistency contexts critical for decision-makin the efficiency of our DDM, which effectively leverages both consistent and inconsistent modality contexts for robust decision-making, The results with all four modalities are given in the Appendix.
Consequently, they do not match
Results on D.Controol. We further evaluate different methods on the DMControl in Table Similar to the results on the CARLA benchmark, our method demonstrates significant performance improve- ments on DMControl. It apparent that conventional multi-modal methods such as TransFuser and EFNet exhibit limited performance, likely due to the increased inconsistency brought by random masking. contrast, our DDM method consistently enhances the performance of the SAC baseline by a large margin, further verifying its effectiveness.
7

---

Table 3: Ablation study of different components in DDM on CARLA benchmark.
Mathoda
Normal
Midnight
Damzling ing
Rainy
Average
ER
DXm)
D(m)
ER
[Xm)
DXm)
ER
D(m)
RGB
06
211 49
121 + 54
187 +:1
84 02
12
159 58
194 4:1
133.25
179.25
Baseline
35
125
214.50
+h
24
275 4:22
198 63
232 4:88
157 4:22
199
241 1:14:53
274 2 45
207.50
245.00
+.r
48
54
212
164
2
254
271 39
219.75
257.50
+ +mm
262 4:
312 39
227 4 54
2
184 5:
243
258 4 54
282
2 45
252.75
274.25
277 + 59
324 4
248
272 52
208 46
254
279
298 40
253.00
287.00
Lurch
289
= 61
279 41
300 + 46
229 x 42
+ 47
204 35
314 42
272.75
304.73
Normol
Midvight
Nerma
Midright
Rovont www.tttmmattooooom eroe
15
100
100
300
200
400
200
400
Roward
diffcrest methods
difieen methods
DDMG6T)
DDM Fyras)
DeepMDP
DOMURAD teed
DDM Furca)
DeapMDIP
Figure 3: The reward estimation and state prediction errors of different methods.
4.3
Ablation Study
To systematically verify the effectiveness of each component in DDM, we conduct series of ablation experiments the CARLA benchmark. The experiments conducted by progressively integrating different training objectives Table Compared with the baseline, incorporating the full-state prediction objective Lyp and reward prediction objective C, notably enhances performance. However, this enhancement does not bring a significant advantage
DDM into the baseline SAC algorithm. The results are given in
and conventional methods such DeepMDP and SPR reported in Table since the modality relationship is not explicitly modeled. Further applying Cmp and Ltp to model modality consistencies provides additional performance gains, showing the importance of extracting common contexts in multi-modal RI. Finally, the integration of Lorth to model modality inconsistencies yields substantial improvement, confirming the effectiveness of our modality decomposition strategy.
4.4 Discussion and Analysis
The Importance of Modality Commonality and Differences. key finding of our study is the crucial role of both modality commonality and differences in multi-modal dynamics modeling. To investigate this, we leverage the bisimulation metrics [9] for MDPs assess the dynamics modeling performance of different methods. Specifically, the bisimulation metrics define two states to be behaviourally similar if they have close rewards along the transition trajectory (9) Therefore, the difference between the estimated and true rewards can reflect how closely the learned states mimic the actual environmental states. Fig. a)), we plot the absolute reward difference [[ (() rt+1| for three methods: DeepMDP, DDM without using Lorth (i.e. only modeling modality consistency), and the full DDM model. The reward difference plotted for each time step of the same test sequence on the CARLA benchmark. From the figure, it is clear that compared to DeepMDP, modeling modality consistency can reduce reward estimation errors. Additionally, incorporating modality inconsistency further significantly enhances estimation accuracy. These results confirm the importance of both common and differing modality contexts in accurately learning dynamics.
Besides rewards, we also examine the state prediction accuracy of different methods. In Fig. 3b)) -|=(- for each method. For DDM methods, we plot the normalized latent state prediction error |ar
z! is defined as iP (2,M) a4), (2,Ma at)...., Pa(z Pa(2, Ma a.)), representing the predicted states from multiple modalities, for DeepMDDP, 2, represents the predicted single latent state. The figure shows that DeepMDP exhibits significantly higher state prediction errors, highlighting the benefits of decomposing modality contexts and treating each modality separately. Among the two DDM variants, the full DDM model with Lorth has less prediction error, again demonstrating the advantages of incorporating task-related inconsistencies.
8

---

Nornal
Midnight
338
DDM
300
Deiiing distamoo
251
24
22
26
DepMDD
155
Driving Distance
Driving Distance
Narnal
Midnight
Original
|Random Noise
+Randon Mask
+
20K
Steps
G
L0OK Stepe
+
DeepMiDD
S
DDM
(a) Policy rebusinecs on CARLA
Sample efficieeey on CARLA
Figure 4: Comparison between DeepMDP and DDM policy robustness and sample efficiency.
Nommal
Midnight
Dazzling
Rainy
RGB Con.
Event
RGB Incon.
Event Incon.
Figure 5: t-SNE visualization of consistent and inconsistent features.
Policy Robustness and Sample Efficiency. The modality decomposition and reward prediction in DDM efficiently extract task-related contexts and filter out irrelevant content, potentially enhancing policy robustness against distractions like data noise. To verify this, evaluate DDM and Deep- MDP on the CARLA benchmark with random Gaussian noises or random masks added to the input modality data. The results Fig. (a) show that DDM experiences minimal performance degrada- tion, highlighting its robustness. In contrast, DeepMDP exhibits a significant performance decline, particularly when faced with random input noises. Despite policy robustness, further investigate the sample efficiency of the two methods. The results in Fig. 4i) show that DDM converges faster and achieves better accuracy than DeepMDP various environmental conditions. These advantages highlight the superiority of DDM, demonstrating its potential for learning diverse visual control tasks.
Comparison of Different Approaches for Inconsistency Extrac- tion. In Sec. we have discussed two different approaches for in- consistency extraction, i.e., applying the mutual exclusive constraint within the same modality (Eq. (6) or across different modalities (Eq. [7). We conjecture that Eq. 66 might be overly strong, poten- tially affecting feature expressiveness. To verify this conjecture, we compare the performance of the two inconsistency extraction approaches on the CARLA benchmark. The results in Fig. 7 show that forcing orthogonality within the same modality using Eq, 6 consistently obtains worse performance, indicating that learns less expressive features compared to
OCo. (Fq Iscon. (E..7)
299
300
291
267
249
Nornal
Midnight
Dazeling
Rainy
Figure 7: Comparing different inconsistency constraints.
our approach.
Visualization of Decomposed Modality Features. To confirm that the modality features decomposed
by DDM accurately capture the intended contexts, we use t-SNE to visualize both modality-consiste consistent and inconsistent features for RGB and event signals on the CARLA benchmark. As shown in Fig DDM successfully aligns the conssitt-consistent features, resulting in closer feature distances, while features are distinctly separated. Further visualizations in reveal that the consistent features primarily focus common objects (e.g., road fences), whereas the inconsistent features highlight objects visible in one modality but not in others (e.g., lane lines RGB frames). In contrast, DeepMDP tends to indiscriminately focus discriminativity. These visualizations demonstrate the efficacy of the modality decomposition in DDM, facilitating more precise dynamics modeling and efficient state representation.
the condality-inconsine
all scene objects, lacking contextual

---

RGB frames
Event signals
DeepMDP
Consistency
Inconsistency
Figure 6: Visualization of feature heatmaps in DDM under normal and midnight conditions.
5
Limitations and Future Works
Our current DDM method has two primary limitations. First, while it effectively simulates multi- modal dynamics and enhances representation learningg lacks integrated planning mechanisms similar to those in MuZero [36] and Dreamer [3
Research has shown that planning mechanisms can effectively model the impact of various strategies in data-rich scenarios, thus minimizing trial and error in real-world applications. In future works, we plan to integrate long-term action consequences into our decision-make making processes by developing decomposed multi-modal world models. Second, our approach focuses primarily vision-based data inputs. However, recent studies suggest that integrating diverse modal inputs, such as textual and audio data, can yield adaptable representations [47]. Therefore, extending approach to include additional types of modal inputs represents another promising future direction to enhance its scope and performance.
15
16
34
6
6
Conclusion
We have studied the impact of modality heterogeneity on dynamics modeling in multi-modal visual reinforcement learning (RL) tasks. Our investigation reveals that both the commonalities and differences each modality are crucial for accurate dynamics modeling, albeit playing distinct roles. Based on this insight, we have presented Dissected Dynamics Modeling (DDM), a novel method designed to enhance dynamics modeling and representation learning for multi-modal visual RL. Our approach represents one of the first explorations of modeling environmental dynamics in multi-modal scenarios, providing a fresh context decomposition perspective based on modality relationships. Extensive experiments demonstrate the effectiveness of our approach, which significantly improves representation quality and decision-making performance in diverse environmental conditions.
7
Acknowledgment
The study funded by the National Natural Science Foundation of China under contracts No. 62332002, No. 62425101, No. 62422602, No. 62372010, No. 62202010, No. 62402015, and the China Postdoctoral Science Foundation under grant 2024M750100.
References
[1] David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning, pages 2915-2923, PMLR, 2016.
[2] Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2022.
[3] Guang Chen, Hu Cao, Jorg Conradt, Huajin Tang, Florian Rohrbein, and Alois Knoll. Event- based neuromorphic vision for autonomous driving: paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine, 37(4):34 2020.
10

---

[4] Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust supervied deep reinforcement learning. In IEEE Proceedings of the International Conference Robotics and Automation, page 4274-4280, 2021.
[5] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 45(1 45(11):12878-12899 2023.
[6] Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-fies model- reinforcement learning with prototypical representations. In Proceedings of the International Conference Machine Learning. 2022.
[7] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 1-16. PMLR, 13-15 Nov 2017.
[8] Norm Ferns, Prakash Panangaden, and Doina Precup. Metries for finite markov decision processes. In UAI, volume pages 162-169, 2004.
[9] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In Proceedings of the International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 2170-2179, 09-15 Jun 2019.
[10] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Ad- vances in neural information processing systems, 31, 2018.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings the International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870, 10-15 Jul 2018.
[12] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprini
2018.
[13] Danijar Hafner, Timothy Lillierap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. arXiv preprint arXiv: 19/2.0160 2019.
[14] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas. David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. Proceedings of the International Conference on Machine Learning, volume 97, pages 2555-2565, 09-15 Jun 2019.
[15) Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[16] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 2023.
[17] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Proceedings of the Advances in Neural Information Processing Systems, volume 34, pages 3680-3693, 2021.
[18] Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey. arXiv preprint arXiv:2202.02700 2022.
[19] Sili Huang, Yanchao Sun, Jifeng Hu, Siyuan Guo, Hechang Chen, Yi Chang, Lichao Sun, and Bo Yang. Learning generalizable agents via saliency-guider features decorrelation. Advances in Neural Information Processing Systems, 36, 2024.
[20] Yangru Huang, Peixi Peng, Yifan Zhao, Haoran Xu, Mengyue Geng, and Yonghong Tian. Hierarchical adaptive value estimation for multi-modal visual reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
11

---

[21] Xiaodong Jia, Xiao- Jing, Xiaoke Zhu, Songcan Chen, Bo Du, Ziyun Cai, Zhenyu He, and Dong Yue. supervise multi-view deep discriminant representation learning. IEEE transactions on pattern analysis and machine intelligence, 43(7):2496-5500 2020.
[22] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Proceedings of the International Conference on Machine Learning, volume 202, pages 14975-15022 2023.
[23] Michael Laskin, Aravind Srinivas, and Picter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 5639-56500 13-18 Jul 2020.
[24] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto. Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Proceedings of the Advances in Neural Information Processing Systems, volume 33, pages 19895, 2020.
[25] Kimin Lee, Younggyo Sco, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning, In International Conference on Machine Learning, pages 5757-5766. PMLR, 2020.
[26] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128x 120 db 15 μs latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits, 43(2):566-576, 2008.
[27] Kaiyi Lin, Xing Xu, Lianli Gao, Zheng Wang, and Heng Tao Shen. Learning cross-aligned latent embeddings for zero-shot cross-modal retrieval. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11515-11522, 2020.
[28) Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Proceedings of the Advances in Neural Information Processing Systems, volume 34, pages 18459-18473, 2021.
[29] Siao Liu, Zhaoyu Chen, Yang Liu, Yuzheng Wang, Dingkang Yang. Zhile Zhao, Ziqing Zhou, Xie Yi, Wei Li, Wenqiang Zhang, al. Improving generalization in visual reinforcement learning via conflict-aware gradient agreement augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23436-23446, 2023.
[30] Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning. arXiv preprint arXiv:2210.04561, 2022.
[31] Guozheng Ma, Linrui Zhang. Haoyu Wang, Lu Li, Zilin Wang, Zhen Wang. Li Shen, Xueqian Wang, and Dacheng Tao. Learning better with less: Effective augmentation for sample-efficient visual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[32] Jinming Ma, Feng Wu, Yingfeng Chen, Xianpeng Ji, and Yu Ding. Effective multimodal reinforeement learning with modality aligmment and importance enhancement. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, page 1684-1686, 2023.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. nature, 518(7540):529-332 2015.
[34] Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
[35] Rudra PK Poudel, Harit Pandya, Stephan Liwicki, and Roberto Cipolla. Recore: Regularized contrastive representation learning of world model. arXiv preprint arXhv:2312.00056, 2023.
12

---

[36] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839)6604 609, 2020.
[37] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philipp Bachman. Data-efficient reinforcement learning with self-predictive representations. In Pro- ceedings of the International Conference Leurning Representations, 2021.
[38] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of the Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1332-1344. PMLR, 14-18 Dec 2023.
[39] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the International Conference Machine Learning, volume 139, pages 9870-9879, PMLR, 18-24 Jul 2021.
[40] Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-baset fusion for motion deblurring with cross-modal attention. In Proceedings of European Conference on Computer Vision, volume 13678, pages 412-428, 2022.
[41] Ruixiang Sun, Hongyu Zang, Xin Li, and Riashat Islam. Learning latent dynamic robust representations for world models. arXiv preprint arXiv:2405.06263. 2024.
[42] Richard Sutton and Andrew G Barto. Reinforcement learningg An introduction. MIT press, 2018.
[43] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801.000000000000 2018
[44] Tasbolat Taunyazov, Luar Shui Song, Eugene Lim, Hian Hian See, David Lee, Benjamin CK Tee, and Harold Soh. Extended tactile perception: Vibration sensing through tools and grasped objects. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1755-1762. IEEE, 2021.
[45] Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint vrXiv:1502.022555 2015.
[46] Shuo Wang, Zhihao Wu, Xiaobo Hu, Jinwen Wang, Youfang Lin, and Kai Lv. What effects the generalization in visual reinforcement learning: policy consistency with truncated return prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 5590-5598, 2024.
[47] Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey, Machine Intelligence Research, 20(4):447-482, 2023.
[48] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
[49] Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
[50] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. Proceedings of the International Conference on Learning Representations, 2020.
[51] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 13022-13037, 2022.
13

---

[52] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In Proceedings of the International Conference on Learning Representations, 2020.
[53] Yue Zhao, Chenzhuang Du, Hang Zhao, and Tiejun Li. Intrinsically motivated self-supervised learning in reinforcement learning. In Proceedings of the International Conference on Robotics and Automation, page 6605-3615. IEEE, 2022.
14

---

Appendix
The contents of this Appendix are presented follows:
Section A Joffers extra experiment findings, including performance on the CARLA benchmark with more modalities, experiments for visual inputs from multiple camera positions, sample efficiency and different masking ratios on the DeepMind Control Suite.
Section B describes the experimental setups, including the details of network structures, environment conditions, training hyper-parameters, and hardware details respectively.
Section
C
discusses the potential socictal impacts of our method.
Table A1: Comparison with state-of saee--ff-hheeaaee art methods on CARLA benchmark with four modalities.
Mathoda
Type
ER
Nornll
D(m)
ER
Midnight
Dim)
ER
Dazzi ing
D(m)
ER
Bainy
D(m)
Average
Dim)
SAC
S-RL
182
283
198
236
174 1
211 1:78
194 x
250
187.00
247.25
DIQ
8-RI.
229
329
218 +:70 76
262 +:76
206 +
231 +:80
239 76
287 72
223.000 (K)
277.25
DeepMDP
S-RL
244
327 5.22
135 6.7
285 62
215 1:50770
2
248 61
272 58
235.5
281.50
SPR
83
250
TransFuser
M-CV
244 80
249 74
234 76
255 6:
264 72
308 =
247.75
304.50
EFNet
M-CV
268 74
356 60
240 69
312+66
225 78
249
259 74
301 72
248.00
304 304.50
MuMMI
254
299
00
MAIE
311+
238
300 73
246-50
HAVE
M-RL
295 72
365 70
375 81
334 76
258 81
267 76
307 72
357 75
283.75
325 75
Ours DDM
M-RL
316
378
296 75
27
276 72
282
76
353
69
303,00
342.00
Table A2: Results CARLA benchmark with different perspectives RGB+Lidar-BEV.
Nornal
diigtth
Dazzling
Rainy
Avaragee
Nathoda
ER
Dim)
D(m)
ER
DKm)
ER
D(m)
ER
D(m)
HAVE
241 64
2
2
221 0
244 2 74
272
281
6767
246.15 268.25
273T0I
Ours DDD
268 = 72
313
4
282 2 63
308 46
239
268 a 59
284 55
305
a 60
298.50
Table A3: Results on DMControl with different perspectives: RGB and depth modality with different
camera perspectives.
Cartpele
Reacher Easy
Cheetah
Cartpole Swingup_spaae sparse
Reacher
Quadruped
thode
Type
Swingup
Average
Average
SPR
R
792 =57
839 242 887 +47
542 149
724.33
704 168
134 182 +64
401 258 452 +65
413.00 470.00
Our--DDM
M.RI.
820 #40
555 +61
754.00
776 +57
A Additional Experimental Results
Results with More Modalities CARLA. Table Al presents the results of various methods on the CARLA benchmark, utilizing four modalities: RGB frames, event signals, depth images, and LiDAR BEV images. Compared to the results in Table nearly all methods show improved performance with the increased number of modalities. Additionally, the trends observed in Table A follow those in Table I with single-modal methods typically underperforming their mudti-modal counterparts. Among all the multi-modal methods, our DDM method consistently maintains the best performance due to its modality decomposition and integration. This comprehensive modeling allows DDM to effectively utilize additional information provided by multiple modalities, leading to robust decision-making compared other methods.
Experiments for visual inputs from multiple camera positions. We have further verified the ability of our model multiple camera positions. First, switch the camera view of RGB modality in DMControl. Second, we test on CARLA with RGB and LiDAR BEV as input modalities. LiDAR BEV is a bird view map, whose perspective is very different from RGB. For simplicity, only compare our method against the most competitive baseline methods on these two environments (i.e. HAVE CARLA and SPR on DMControl). The results and illustrations of different camera views are presented in Table A2 Table A3 and Fig. Al respectively. These results show that our method also works multiple camera positions.
Sample efficiency on DMControl. In Fig. 22 we evaluate the sample efficiency of different methods on DMControl. Compared to DeepMDP and EFNet, our method demonstrates the highest sample
15

---

(a) RGB and Depth modalities with different camera perspectives.
(b) RGB and Lidar-BEV modalities with different perspectives on Carla.
Figure Al: Illustration of visual inputs from multiple camera positions.
efficiency across five tasks, shown in the figure. This superior performance is attributed to method's ability to capture a more comprehensive representation of the environment. By accurately modeling the dynamics of each modality and their interactions, method can make more informed decisions with fewer samples, enhancing learning efficiency and overall performance.
1000
90-1
DeepMDP
900
854
813
EFNet
7622
500K steps
DDM
600
524
555
587 (322)
467
435
100K steps
489 A13
4000
226
224
200
133
TBR
100
Campole, swingop
Reacher, easy
Cheerah, nun
Cartpole, swingup_ spanse
Rencher, hard
Quadruped, nun
Figure A2: Sample efficiency on DMControl. For each task, we provide the task returns of three methods at 100K and 500K steps, respectively.
Table A4: Different mask ratio on DMControl.
Methods
Masking Ratio
Cartpole Swingup
Reacher
Easy
Cheetah Run
Average
Cartpole Boingup_spares
Reacher
Hard
Quadruped Run
Average
870
956 E:0
662 249
829.33
S02 161
564 263
337 258
634.33
SPR
0.2
845 1:4
905 :37
581 245
777.00
720 154
187 248
468 #39
458.33
0.6
813 +63
824 +59
525 +65
720.67
628 +59
82 +72
403 +68
371.00
0)
868 2:12
935 1:7
676 162
833.00
839 152
597
552 259
659.33
DDM
0.4
838 +50
852 242
585 =70
758.33
795 +35
211 264
437 =62
481.00
0.6
845 475
818 168
524 272
729.00
761 480
185 267
446 263
464.00
Different masking ratios on DMControl. Our masking operation is performed independently at each timestamp. Therefore, both the masked modality type and the masking locations vary randomly across different timestamps, which simulates a challenging occlusion scenario. For other masking ratios, we further test the ratio of 0%, 40%, 60%, and compare our method with SPR, the most competitive baseline DMControl. As shown in A4] the results reveal that as the masking ratio increases, both SPR and our DDM experience performance drops. However, DDM still outperforms SPR at different ratios.
16

---

Normal
Midnight
Dazzling
Rainy
Figure A3: Demonstration of RGB frames and event signals on the CARLA benchmark.
B Detailed Experimental Setup
In this section, we provide comprehensive description of the experimental setup, including network architectures, environment configurations, and typer-paramedt settings.
B.1 Network Architecture
Modality Encoders. Individual feature encoders are employed for each modality to handle the input data. Each input observation provided the encoder consists of 3 consecutive frames returned by the environment. This results in input channel numbers of 9 for RGB frames, 15 for event signals, for LiDAR BEV images, and for depth images. All the feature extractors share a common network architecture that comprises four convolutional layers, each followed by ReLU activations. The final layer produces 64 x 6 feature maps, which serves as the individual modality features = These features are then passed into two separate network branches get 50-dimensional consistent and inconsistent features z and e,M respectively.
Actor and Critic Networks. In our method, the actor network, which is composed of three fully connected (FC) layers with ReLU activations placed after the first two layers, produces parameters for a Gaussian distribution over actions. The critic network has similar structure but outputs an estimated Q-value for a specific state-action pair.
Prediction Heads and Reward Prediction Network. The prediction heads, as described in (9] take the input latent vector and the corresponding action inputs. They generate the inferred latent vectors through two fully connected (FC) layers, with ReLU activation following the first layer. Similarly, the reward prediction network uses the same architecture but outputs single-dimensional predicted reward. This network estimates the expected reward for specific state-action pair, offering critical feedback for dynamics modeling.
B.2
Environment Details
B.2.1
Environmental Conditions in CARLA
The four environmental conditions (Normal, Midnight, Dazzling, and Rainy) employed in our study are predefined with specific weather hyper-parameters, as detailed (Able[A5] These conditions provide comprchensive evaluation of the RL algorithms under various situations.
17

---

Table A5: The hyper-parametenn of weather conditions CARLA.
Weather Parameters
Normal
Midnight
Duzzlngg
Rainy
Cloudiness
5
30
D
Precipitation
0
0
100
Precipitation deposits
0
100
wind intensity
10
fog. density
20
fog distance
0.75
1000
wetness
0
50
sun azimuth aaggle
-I
270
sun altiituteeeeeii angle
45
-90
10
B.2.2 Modality Settings
Modalities in CARLA. RGB frames are the default input modality in CARLA, which capture rich texture and color information but may face challenges like motion blur and limited dynamic range under extreme light conditions, Event signal is a new modality provided in CARLA, mimicking the output of neuromorphic event cameras [26] 3. As shown Fig. A3 event signals focus on changes in brightness, offering a wider dynamic range. no motion blur, and high temporal resolution, making them ideal for dynamic scenes. However, they do not generate signals at static regions when no pixel differences occur. In addition, event signals may experience hot spot noises. We simulate this noise to replicate real event signal characteristics. Some key parameters related to these two modalities are provided in Table A6 Besides RGB frames and event signals, the depth images in CARLA provide distance maps, while LiDAR BEV creates scanning of the vehicle surroundings. Both depth and LiDAR adopt the default simulation parameters in CARLA.
Modalities in DMControl. In DMControl, the RGB frame modality captures direct visual informa- tion from the environment, providing detailed texture and color data essential for recognizing objects and their spatial relationships. Complementing this, the depth modality generates a depth map where each pixel value represents the distance to the corresponding point in the scene, offering crucial 3D spatial information. The settings of these two modalities are based on the default parameters.
Table A6: The hyper-parameter settings of the experiments on CARLA.
Expostre spord up Exposre__peed spesd_e__eeed_do
Blir amoun
intensiyy
Motion abject
intmetty
Shutter
0.2 1.20Mmon
Negaiis Bacshold
0.2 (MiimidnMM.
Mipnt_posthie Sreshald
Sigmi, threshold
Num
B.2.3 Reward Setting of CARLA
In the CARLA autonomous driving task, the objective of the agent is to travel far it highway without colliding in various weather conditions. Therefore, the reward function is designed similarly to [52] and [20
(14) The first term is devised to motivate the vehicle cover as much distance as possible along the highway. Vego indicates the velocity vector of the agent vehicle, Ühigkway represents the unit direction vector of the highway, and At 0.1 stands for the discretized simulation time. The final three terms are incorporated to guarantee that the vehicle evades collisions, reduces excessive steering, and avoids sudden braking, where Aes A., and A are assigned values of 0.001, 0.1, and 0.1 respectively.
Ft Ve Whighway At de collision ^s |steer| D brake
B.3 The parammees of RL training
The hyper-parameters RL. training are elaborated in Tables A7 and A8 and they are same to those in and (20].
18

---

Table A7: RL training hyper- syper-parameters CARLA.
Hyperparameter
Value
Image size
128
128
Stacked frames
Action repeat
1
Discount factor
0.99
Init steps
Episode length
1,000
Leurning algorithm
Soft Actor-Critic (SAC)
Number of Trames
100,000
Replay bufler size
100,000
Optimizer (encuder, acter, critic)
Adam (9) 0.9, 0.999)
Optimizer (transition and reward prediction network)
Adam
Learning (encoder, actor, critic)
Ic-3
Learning rate (transition and reward prediction network)
Ie-3
Learing
Transition and reward prediction network update frequency
Actor update frequency
Critic target update frequency
Table A8: RL training hyper- parameters on DMControl.
Hyperparameter
Value
Image size
84 x 84
Stacked frames
Action repeat
4
Batch size
128
Discount factor
0.99
Init steps
1,000
Episode length
Learning algorithm
Soft Actor-Critic (SAC)
Number frames
500,000
Replay buffer size
100,000
Optimizer (encoder, actor, critic)
Adam (3, 0.9, 0.999)
Optimizer (transition and reward prediction network)
Adam
Learning rale (encnder, actor, critic)
le.3
Learning (transition and reward prediction network)
le-3
Learning rate (o in SAC)
le-3
Transition reward prediction network update frequency
Actor update frequency
target update frequency
B.3.1 Hardware Details
Computing Resources. All models are trained using a server that is equipped with 4 NVIDIA GeForce RTX 3090 GPUs and a 64-core AMD EPYC 7H12 2.6GHz CPU Processor.
C Potential Negative Societal Impacts
While DDM offers significant advancements in reinforcement learning by effectively integrating multi- modal data, is essential to consider and address some potential negative societal impacts. DDM models could be susceptible to adversarial attacks, where malicious inputs are designed to deceive the system into making incorrect decisions. Such attacks could lead to failures in critical applications, such as autonomous vehicles or healthcare systems, potentially causing harm to individuals.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Limitations and Future Works section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
Please refer to the code link in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwti//gguie///eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and the research conducted in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve Institutional Review Board (IRB) approvals or equivalent for research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "ZUtEI39vEU", "reviewer": "Reviewer_Msqh", "summary": "The paper proposes Dissected Dynamics Modeling (DDM), a dynamics modeling framework for learning latent features in multi-modal visual RL. The methodology focuses on capturing both the shared and distinct information contained across input modalities. The paper presents a multi-modal architecture and training loss designed to accomplish this, and experimentally demonstrates the benefits of DDM on tasks from CARLA and the DeepMind Control Suite.", "questions": "**Methodology:**\n\n**[Q1]** Are gradients with respect to the latent feature encodings $z$ stopped in any components of the loss function in (13), or are they propagated through the loss function everywhere that $z$ appears?\n\n**[Q2]** Are the next-step target values $z_{t+1}$ in (5) and (11) calculated using the same feature encoder used for $z_t$? It would be helpful to make this more clear. Are gradients taken through $z_{t+1}$ in (5) and (11)?\n\n---\n\n**Experiments:**\n\n**[Q3]** What do the single modality baseline results represent? Do they only use RGB images, or do they consider a combined input that incorporates information from all modalities? If these baselines are not restricted to RGB images, it would be useful to include results (baselines or ablation of DDM) using only RGB inputs to demonstrate that the use of multi-modal inputs leads to improved performance over standard visual RL.\n\n**[Q4]** DreamerV3 [e] and DrQ-v2 [f] are strong model-based and model-free visual RL algorithms, respectively. Why were DeepMDP and SPR chosen as baselines instead of DreamerV3 (which has reported better head-to-head performance vs. SPR in Atari 100k)? Why was DrQ chosen as a baseline instead of DrQ-v2 (which has reported significant improvements over DrQ on DeepMind Control Suite)? \n\n**[Q5]** Has performance converged by the end of training for the experimental results, given the number of training steps used (100k and 500k in CARLA and DeepMind Control Suite, respectively)? In prior works on visual RL, DeepMind Control Suite tasks are typically trained for longer than this (1M steps in DreamerV3 [e], 3M steps in DrQ-v2 [f]), and these baselines had not converged to final performance after 500k steps.\n\n---\n\n**Minor:**\n\nSpelling: Inon. --> Incon. (p. 2, Figure 1 caption)\n\n---\n\nReferences:\n\n[e] Hafner et al. (2023). Mastering diverse domains through world models.\n\n[f] Yarats et al. (2021). Mastering visual continuous control: Improved data-augmented reinforcement learning.", "limitations": "The authors have addressed the limitations of the current work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**[S1] Novel approach to multi-modal RL:** The paper proposes a novel methodology for multi-modal visual RL that accounts for both the similarities and differences across input modalities, compared to prior works that largely focus only on the similarities across input modalities. Experiments demonstrate the benefits of this novel approach.\n\n**[S2] Detailed experimental analysis:** The paper provides detailed experimental analysis of the proposed DDM method. This includes comparisons across several types of baselines (although some choices of baselines are not state-of-the-art), ablation studies that justify design choices, robustness analysis, and visualizations for qualitative understanding.\n\n**[S3] Important problem:** The paper focuses on how to effectively leverage information from multiple input modalities for decision making. This addresses an important consideration for the successful deployment of RL in real-world systems where multi-modal data is common.", "weaknesses": "**[W1] Clarity of implementation / experimental details:** The high-level idea of the DDM framework is clear, but some of the implementation details are not clearly described. Some components of the experiments were also not described in detail, which made it difficult to interpret some of the results. Please see the questions below.\n\n**[W2] Connections to related work:** DDM appears to build upon previous works in visual / multi-modal RL for its dynamics modeling and consistency extraction components, but these connections are not made clear. I think it would be useful to better emphasize what parts of DDM build upon existing works vs. what parts are novel contributions. (i) The dynamics modeling approach looks similar to reconstruction-free approaches in visual RL [a,b,c,d], but the similarities / differences are not discussed. (ii) It is mentioned that existing multi-modal approaches have focused on aligning modalities, but the paper does not discuss how the proposed consistency extraction method relates to these approaches. \n\n**[W3] Heuristic design choices:** Design choices of the proposed architecture are justified through experimental results, but no theoretical support is provided (or connections to existing approaches that provide theoretical support).\n\n---\n\nReferences:\n\n[a] Gelada et al. (2019). DeepMDP: Learning continuous latent space models for representation learning.\n\n[b] Schwarzer et al. (2021). Data-efficient reinforcement learning with self-predictive representations.\n\n[c] Okada et al. (2021). Dreaming: Model-based reinforcement learning by latent imagination without reconstruction.\n\n[d] Zhang et al. (2021). Learning invariant representations for reinforcement learning without reconstruction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for the insightful comments. We also deeply appreciate the suggestions regarding presentation and experimentation. Below are our responses to your concerns:\n\n> [W1] Some implementation and experiment details are not clear.\n\nPlease refer to our responses to [Q1-Q5] for detailed clarification.\n\n\n> [W2] Connections to related work need to be emphasized.\n\nThank you for the suggestion. Here we discuss the connections in detail:\n\n**(i) Connections with dynamics modeling methods [a-d]**\n\nThe primary similarity between [a-d] and our DDM is the state and reward prediction strategy commonly used in RL dynamics modeling. However, DDM differs as it is tailored for multi-modal RL, introducing novel approaches in both modality decomposition and dynamics modeling.\n\nSpecifically, both DeepMDP [a] and DDM predicts latent states and rewards. However, DDM does not address modality relationships or learn state transitions in a modality-aware manner. SPR [b] uses multi-step state prediction but it overlooks modality interactions. Dreaming [c] mainly focuses on learning over different timesteps and samples. Differently, DDM aims at dissecting features across different modalities. DBC [d] employs a bisimulation metric to model dynamics, which differs from our dissected modeling strategy.\n\n\n**(ii) Relation to existing consistency extraction methods**\n\nAlthough the goal of drawing modality features closer for consistency extraction is similar between DDM and existing methods, the techniques vary significantly. MAIE [32] directly minimizes modality features at the same timestep. In contrast, DDM predicts cross-modality transition across adjacent timesteps. TransFuser [5] and EFNet [40] utilize attention, MUMMI [4] optimizes mutual information, and HAVE [20] employs attention and hypernetworks. Like MAIE, these methods do not create a synergy between dynamics modeling and modality alignment, which is a unique feature of the cross-modality transition prediction in our DDM.\n\n(Reference numbers are the same as in our paper)\n\n> [W3] No theoretical support is provided.\n\nThank you for emphasizing the need for a theoretical analysis. We have conducted initial research to find theoretical insights that support our approach.\n\nOur investigation suggests that information theory [1] might provide a suitable theoretical framework for our study. For instance, research in [2] shows how multiple sources provide structured multivariate information about a target variable. Analogously, in multi-modal RL, each modality can be treated as a source and the agent's action as the target. Based on this, we aim to establish a lower bound on the information gap between using only modality-consistent features versus all available information. Deriving this lower bound will underscore the importance of both modality consistencies and inconsistencies in decision-making, thus supporting our method design and empirical findings.\n\nHowever, despite our earnest efforts on this idea, establishing a definitive result remains challenging for us within the limited rebuttal timeline. We sincerely value your feedback and will deepen this theoretical analysis in future work.\n\n[1] The Mathematical Theory of Communication, Bell System Technical Journal, 1948\n\n[2] Nonnegative Decomposition of Multivariate Information, arXiv, 2010\n\n\n> [Q1] Are gradients of $z$ stopped in (13)?\n\nSorry for not clarifying this. We stop the gradient of all prediction target features in (4), (5), (8), and (11). That is, for each loss term of the form $||X(z_a)-z_b||^2$, where $X$ represents prediction head and $z_a$, $z_b$ are modality features, we stop the gradient of $z_b$.\n\n> [Q2] 1. Are $z_{t+1}$ in (5) and (11) used the same encoder as $z_t$? 2. Are gradients taken through $z_{t+1}$?\n\n1. Yes, they are. We do not utilize a moving average encoder like in SPR [b] but rather use the same online encoder while stopping the gradient of the target values like in DBC [d].\n\n2. The gradients are stopped as explained in Q1. This avoids potential model collapse, which is observed in SPR paper when no stop gradient operation is applied to the target values.\n\n([b,d] are the same as in [W2])\n\n> [Q3] What are single modality baseline results? Any results using only RGB inputs?\n\nThe results consider a combined input of all modalities. Following the advice, we have tested the case where only RGB inputs are used. The results are in Table 1 in the rebuttal PDF, which shows that the baseline with multi-modal input indeed outperforms RGB.\n\n> [Q4] Why use DeepMDP, SPR and DrQ as baselines instead of DreamerV3 and DrQ-v2?\n\nFor single-modality baselines, we would like a direct comparison of the dynamics modeling technique itself and reduce the impact of other factors (e.g., the RL algorithm and the use of planning mechanism or not). Therefore, we choose DeepMDP, SPR and DrQ, all use SAC as RL algorithm without planning, just like our method. We appreciate the recommendation of DreamerV3 and DrQ-v2 and have conducted experiments with them. As shown in Table 2 of the rebuttal PDF, DrQ-v2 obtains stable performance, while DreamerV3 gets inferior results on the Cartpole swingup_sparse task. This is probably caused by the sparse reward and modality inconsistencies, which hampers the learning of its transition model.\n\n> [Q5] Has the performance converged?\n\nOur setting of training steps follows DBC [d] and SECANT [1]. To verify the convergence, we further train different methods with double steps (i.e., 200K/1M for CARLA/DMControl). The results are given in Fig.1 (a)-(d) in the rebuttal PDF. The figure shows that the improvement beyond 100K/500k steps becomes exceedingly marginal. Therefore, our current training can achieve a satisfactory level of performance.\n\n[d] is the same as in [W2]\n\n[1] SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies, ICML 2021\n\n> Minor: Inon.-> Incon.\n\nThanks for spotting this, we will correct it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7UJuq7qbTS", "reviewer": "Reviewer_dbSG", "summary": "The paper presents a solution for better multimodal dynamic modeling in visual RL. The paper claims that existing works only emphasize consistent (aligned) information across modalities, leaving out the opportunity for the model to benefit from the inconsistent features. The work introduces a new consistency loss, where the (cross-modal) mutual prediction happens dynamically, meaning a modality has to infer the other modality's features for the next step rather than the current one. Furthermore, the work introduces a \"soft objective\" for cross-modal feature orthogonalization. The work shows consistent improvement over the existing state-of-the-art on CARLA and DMControl benchmarks.", "questions": "1) Do the authors think that incorporating a component of their modeling, e.g., $ \\mathcal{L_{tp}} $, into existing models could result in a boost? \n\n2) In Section 3.2 [L166], the authors suggest replacing a stronger objective (Eq. 6) with Eq. 7. Did you observe in your experiments that, indeed, a model trained with Eq.7 is better than Eq.6?", "limitations": "The authors discussed potential limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-motivated, well-written, and easy to follow. The state-of-the-art results provide stats with several runs. The design of each loss component is straightforward.\n2) The paper achieves significant improvement over the reported state-of-the-art results. \n3) The cross-modal transition prediction loss is interesting and results in the most significant performance boost, as shown in Table 3. Overall, each loss component is shown to benefit the model's performance.\n4) The method is generalizable beyond two modalities.", "weaknesses": "1) The authors report the episode to return and driving distance but do not report DS/RC/IP, which is reported in other methods mentioned in the papers and could be more informative for evaluating driving performance. Is there any reason why the authors do not report those metrics? \n\n2) L265-267 states that   $ \\mathcal{L_{fp}} $ and $ \\mathcal{L}_{r} $ do not benefit much to the existing DeepMDPandSPR. Are there any supporting references/results for this claim?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the time and effort invested in reviewing our work. The positive remarks are truly appreciated, and we feel encouraged by the feedback. We address the points raised in the comments as follows:\n\n> The authors report the episode to return and driving distance but do not report DS/RC/IP, is there any reason why the authors do not report those metrics?\n\nThanks for the thoughtful question on the metrics. Our evaluation protocol on CARLA follows the commonly adopted one in many existing RL works [1,2,3,4]. Different from the setting in TransFuser [5] (which reports DS/RC/IP), the RL agent's goal in this protocol is to drive as far as possible in limited timesteps without colliding into other moving vehicles or barriers. Each episode (i.e., evaluation trial) immediately ends when the agent vehicle collides or reaches the timestep limit. There is no predefined route during evaluation, so RC cannot be calculated. Because each episode immediately ends after a collision, the calculation of IP also becomes impossible. Since RC and IP are both impractical to be obtained, DS cannot be provided. Instead, the RL agent’s performance is primarily evaluated by Episode Reward (ER), which accounts for driving distance and driving stability, such as less abrupt steering.\n\nNote that although we compare with TransFuser in our paper, we evaluate it under the RL protocol rather than its original protocol. As we explained in the paper, this involves integrating the modality fusion and alignment modules of TransFuser with a baseline RL algorithm (SAC). The original setting in TransFuser and the RL setting in our work represent distinct approaches to training autonomous driving agents. The evaluation protocols and metrics for these two settings are also different, which is why we did not report DS/RC/IP scores.\n\n[1] SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies, ICML 2021\n\n[2] Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning, NeurIPS 2023\n\n[3] Model-Based Reinforcement Learning with Isolated Imaginations, TPAMI 2023\n\n[4] Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning, NeurIPS 2023\n\n[5] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving, TPAMI 2023\n\n> L265-267 states that $\\mathcal{L}\\_{fp}$ and $\\mathcal{L}\\_r$ do not benefit much to the existing DeepMDP and SPR. Are there any supporting references/results for this claim?\n\nThanks for identifying this issue. There might be a misunderstanding here. Specifically, we are not trying to use $\\mathcal{L}\\_{fp}$ and $\\mathcal{L}\\_r$ to benefit DeepMDP and SPR. In L265-267, we state “However, this enhancement does not bring a significant advantage over conventional methods such as DeepMDP and SPR”. We mean that although $\\mathcal{L}\\_{fp}$ and $\\mathcal{L}\\_r$ can improve the performance of the baseline model, the improved results are still not significantly better than the results of DeepMDP and SPR (reported in Table 1 of our paper). So, we are describing a cross-table comparison, by comparing the $+\\mathcal{L}\\_{fp}$ and $+\\mathcal{L}\\_r$ rows in Table 3 of our paper and the DeepMDP and SPR rows in Table 1 of our paper. We did not mention Table 1 in L264-267, which may cause confusion. We appreciate your detailed examination and will correct this issue in the revised paper.\n\n> Do the authors think that incorporating a component of their modeling, e.g., $\\mathcal{L}\\_{tp}$, into existing models could result in a boost?\n\nWe are confident that our modeling such as cross-modality transition prediction $\\mathcal{L}\\_{tp}$ can boost existing methods. This is because our method is flexible and is not limited to any particular RL model or network architecture. To verify the effectiveness of our method on other models, we further apply $\\mathcal{L}\\_{tp}$ to DrQ [1] and evaluate its performance. The results are given in Table 2 of the rebuttal PDF, which show that $\\mathcal{L}\\_{tp}$ can also boost the performance of existing RL models.\n\n[1] Image augmentation is all you need: Regularizing deep reinforcement learning from pixels, ICLR 2020\n\n> In Section 3.2 [L166], the authors suggest replacing a stronger objective (Eq. 6) with Eq. 7. Did you observe in your experiments that, indeed, a model trained with Eq.7 is better than Eq.6?\n\nYes, we have compared Eq.6 and Eq.7 in Sec.4.4 (L304-L313) of our paper. The results are presented in Fig.7 of our paper, which show that models trained with Eq.7 consistently perform better than models trained with Eq.6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xTN9V10nuB", "reviewer": "Reviewer_xBoX", "summary": "The paper proposed a method, namely Dissected Dynamics Modeling (DDM), for multi-modal environment dynamics modeling in visual RL. The core idea is to adopt additional modules to extract separate modality-consistent and modality-inconsistent features in each modality stream with designated losses as the regularization. During training, the model tries to maximize mutual information between modality-consistent features from different modalities at the current and the next timestamp. For modality-inconsistent features from, the model tries to enforce orthogonality. Experiments on CARLA and DMControl indicated the superiority of DDM over other state-of-the-arts and further analyses suggested the effectiveness of DDM v.s. DeepMDP. Ablation studies supported some of the important design choices.", "questions": "Please refer to the above Weaknesses part and address my concern accordingly", "limitations": "Yes, limitations are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The general idea of attempting to decouple modality-consistent and modality-inconsistent features to improve environment modeling makes sense, which could provide some insights to future work to this particular field\n\n- Experiment results show good improvements over some state-of-the-arts, suggesting DDM's effectiveness for environment modeling in visual RL\n\n- Good ablations, analyses and visualizations provided a comprehensive understanding of the method to some extent justifying DDM's soundness and supporting its superiority for environment modeling in visual RL", "weaknesses": "- The proposed method is only showed to be working on a subset of modalities that are visual only, which greatly limits its generalizability and contribution as a method for multi-modal learning. However, visual RL is not my expertise and thus I cannot evaluate the contribution on this particular field.\n\n- Following above, from the experiments it seems that the method also only works on visual inputs from the same camera perspective. i wonder if there are experiments demonstrating the model also works for visual inputs from multiple camera positions.\n\n- For experiments on DMControl, how is the masking done for frames at different timestamps? Also, besides 20% masking, do the authors have results with other masking ratio?\n\n- Some places are not written in a technically accurate way, though it's hard to tell whether it is because of technical misunderstanding or simply misuse of words. For example:\n  - Line 46 - 48, \"Firstly, modality-correlated features provide a foundational perspective by capturing shared and complemented information across different sensory inputs.\" The modality-consistent features that are shared among modalities does not COMPLEMENT each other. Instead, those modality-inconsistent features that are unique for each modality themselves complement other modalities.\n  - Line 50-51, \"these inconsistencies are typically deemed less critical and are filtered out through modality alignment\". Modality alignment should not have the effect of filtering out modality-inconsistent features, in my understanding. If the authors have some evidence, I'd like to see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for your thoughtful remarks and experimental suggestions. These remarks shed light on what we can improve and are crucial for refining our work. We address your main concerns as follows:\n\n> The method is only tested on visual modalities, limiting its generalizability and contribution.\n\nWe appreciate the comment and recognize the importance of generalizing to different modality types. We think our method’s contributions are twofold: the fundamental idea and the method itself. While our method mainly focuses on visual modalities, the core idea—that modality-correlated and distinct features are both crucial—might also provide initial spark of inspiration in non-visual fields. For instance, consider a photo of a sea and a corresponding text description like “Vibrant sea under a clear blue sky, with fluffy white clouds.” The photo may show details that the text misses, such as distant ships. Similarly, the text may also capture high-level concepts that lack in the image feature, such as the sensory concept of “fluffy”. These distinct features between modalities can be valuable for the task at hand, which is worth exploring and might motivate further research on image-text learning.\n\nWe hope the potential inspiration brought by our central idea can benefit the non-visual fields and help address some of the concerns. We are thankful for your reflective remark, which suggests a promising extension of our work to fit other modalities (e.g., audio, text) and benefit the entire multi-modal research community. We sincerely appreciate this insight and are committed to work on these extensions.\n\n\n> Experiments for visual inputs from multiple camera positions?\n\nThanks for the experiment advice. We have further verified the ability of our model on multiple camera positions. First, we switch the camera view of RGB modality in DMControl. Second, we test on CARLA with RGB and LiDAR BEV as input modalities. LiDAR BEV is a bird view map, whose perspective is very different from RGB. Due to the limited time budget, we only compare our method with the most competitive methods on these two environments.\n\nThe results and illustrations of different camera views are presented in Table 1,2 and Fig. 1(e),(f) in the rebuttal PDF. These results show that our method also works on multiple camera positions.\n\n> How is the masking done? Do the authors have results with other masking ratio?\n\nOur masking operation is performed independently at each timestamp. Therefore, both the masked modality type and the masking locations vary randomly across different timestamps, which simulates a challenging occlusion scenario.\n\nFor other masking ratios, we further test the ratio of 0%, 40%, 60%, and compare our method with SPR, the most competitive baseline on DMControl. The new results are in Table 2 of the rebuttal PDF. The results show that as the masking ratio increases, both SPR and our DDM experience performance drops. However, DDM still outperforms SPR at different ratios.\n\n> Some places are not written accurately, such as Line 46 - 48 and Line 50-51.\n\nWe apologize for the confusion caused. For the first issue in Line 46 – 48, we confirm it was a misuse of the word \"complemented,\" which inaccurately described modality consistencies. We meant to convey that the modality-consistent features contain shared and common information and create a unified description for the environment. We appreciate the detailed attention to this error and will correct it.\n\nFor the second issue in Line 50-51, our use of the phrase “filter out” may have been overly definitive. We intended to convey that modality alignment aims to encourage consistency and, consequently, mitigate inconsistencies. To clarify this, we provide evidence both in the literature and with experiments. Specifically, we discuss several alignment methods as follows:\n\n**Imposing consistency constraints** is a straightforward method for modality alignment, typically achieved by minimizing cross-modality feature distances [1,2]. Ideally, if two modality features have zero distance, it means perfect consistency and the absence of inconsistencies. We reference multiple works as evidence to support this design goal, as quoted below:\n\n“... the states expressed by different modalities *can be consistent* at the same time... to achieve this, we use a similarity measurement...” (Sec. IV.A in [1])\n\n“...to *ensure the consistency* of the latent embeddings of different modalities in the shared latent space, we develop two parallel cross-alignment schemes...” (Sec.3.2 in [2])\n\n**Mutual information optimization [3]** is also designed to enhance feature consistency (and thereby decrease inconsistencies), we quote [3] as follows:\n\n“The key innovation is a specially-designed density ratio estimator that *encourages consistency* between the latent codes of each modality.” (Abstract in [3])\n\nFor experimental evidence, we calculate the CKA index [4] between modality features. CKA is a metric to quantify similarity between network features for the same input. A higher CKA indicates greater feature correlation. Specifically, for RGB modality on CARLA, we extract features $z$ from a baseline SAC model without modality alignment, and the modality-consistent feature $\\overline{z}$ and inconsistent feature $\\hat{z}$ from our DDM. We Then compare the CKA between these features across 1K input samples. The results are as follows:\n\n① CKA($z$,$\\overline{z}$)=0.865\n\n② CKA($z$,$\\hat{z}$) = 0.897\n\n③ CKA($\\overline{z}$,$\\hat{z}$) = 0.561\n\nIt can be seen the both ① and ② are relatively high, showing that without modality alignment, $z$ retains both common and unique modality information. Further, ③ is notably lower than ②, indicating that compared with the non-aligned feature $z$, the inconsistent information in the aligned $\\overline{z}$ is indeed reduced.\n\nReferences:\n\n[1,2,3] correspond to [32,27,4] in our paper \n\n[4] Similarity of neural network representations revisited, ICML 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for the valuable time and effort dedicated to reviewing our work. The comments have been highly constructive, and the positive evaluations from all reviewers are immensely encouraging. We have carefully considered each remark and have responded accordingly. For a quick and convenient overview, we briefly outline our responses as follows:

**1. Additional experiments**. Following the reviewers’ suggestions, we have conducted additional experiments, such as evaluating more baselines, employing our design to other RL models, and test on different camera perspectives.
 
**2. Clarification of details**. We have clarified several details regarding our technical design and paper presentation, such as the gradient flow during training and the explanations of certain words and phrases in our paper.

**3. Further discussions**. Based on the reviewers’ comments, we have conducted additional discussions on several key aspects, such as the relationship of our method with existing methods and the analysis of modality alignment methods.

Please see our detailed responses below and the attached rebuttal PDF for addressing the individual comments from each reviewer. In addition to the rebuttal, we will incorporate these responses into our revised paper to further enhance its clarity. We thank you once again for your insightful feedback.

Meta-review: This paper proposes a multimodal Visual RL setting where the input modalities contain similar informative cues in both modalities while may also have inconsistent cues, and the goal is how to capture these disparate features for effective policy learning. The paper approaches this problem from a disentanglement perspective, and proposes to project the consistent cues to a common subspace, while the inconsistent cues between the modalities are disentangled using a mutual orthogonality constraint.

The paper received positive reviews overall, with three weak accept recommendations. While, there have been some concerns initially on i) the lack of theoretical foundations for the approach, ii) lack of clarity in implementation details, iii) experimental analysis, and iv) relations to prior works, authors provided a strong rebuttal addressing the concerns. AC concurs with the reviewers judgement that paper is well-crafted and demonstrates promising results and recommends acceptance. Authors should include the additional technical details that came up in the rebuttal into the final paper. 

Further, the current literature review in the paper is quite limited and is specific to visual RL. There is a large body of works on disentangled representation learning, which is the central idea in this paper and that must be reviewed in detail, especially in the context of cross-modal alignment. Some pointers are given below. Authors should contrast their work to such literature and clarify how the proposed model stands out. 
1. Learning to Decompose and Disentangle Representations for Video Prediction, Hsieh et al. NeurIPS 2018
2. Vector-Decomposed Disentanglement for Domain-Invariant Object Detection, Wu et al., CVPR 2021	
3. Semantics Disentangling for Generalized Zero-Shot Learning, Chen et al., ICCV 2021
4. SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders, Li et al, ECCV 2024
Decision: Accept (poster)