Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL

Authors: yangru  huang
peixi  peng
yifan  zhao
guangyao  chen
yonghong  tian
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 07:43:40

Abstract: Accurate environment dynamics modeling is crucial for obtaining effective state representations in visual reinforcement learning (RL) applications. However, when facing multiple input modalities, existing dynamics modeling methods (e.g., DeepMDP) usually stumble in addressing the complex and volatile relationship between different modalities. In this paper, we study the problem of efficient dynamics modeling for multi-modal visual RL. We find that under the existence of modality heterogeneity, modality-correlated and distinct features are equally important but play different roles in reflecting the evolution of environmental dynamics. Motivated by this fact, we propose Dissected Dynamics Modeling (DDM), a novel multi-modal dynamics modeling method for visual RL. Unlike existing methods, DDM explicitly distinguishes consistent and inconsistent information across modalities and treats them separately with a divide-and-conquer strategy. This is done by dispatching the features carrying different information into distinct dynamics modeling pathways, which naturally form a series of implicit regularizations along the learning trajectories. In addition, a reward predictive function is further introduced to filter task-irrelevant information in both modality-consistent and inconsistent features, ensuring information integrity while avoiding potential distractions. Extensive experiments show that DDM consistently achieves competitive performance in challenging multi-modal visual environments.

Main paper: 
 introduction: 
 Recent years have witnessed a growing interest in visual reinforcement learning (RL). By leveraging raw pixel inputs directly from sensory data, visual RL allows agents to navigate and interpret their environments with unprecedented precision and adaptability [33, 24, 51] . Since raw pixels are usually redundant and suffer from expanded data dimensionality, a key aspect of visual RL is to acquire efficient and compact state representations. A typical approach involves modeling environmental dynamics in an abstracted state space [1] , where the state representations are supposed to mimic the actual environmental changes but operate on a sufficiently low dimension. One classical solution is DeepMDP [9] , which leverages neural networks to compress high-dimensional input data. Given consecutive observations, DeepMDP predicts reward and state transitions within the deep latent space, enabling more effective policy optimization over complex, dynamic environments.Figure 1 : Motivation of our method and comparison with DeepMDP [9] designed for single modality. o t , z t , a t , and r t represent the observation, latent feature, action, and reward at time t, respectively. z Mi t and ẑMi t are the decoupled consistent (Con.) and inconsistent (Incon.) features for i-th modality.Although methods like DeepMDP have proven effective for environmental dynamics modeling, research on this subject has been mostly restricted to the single visual modality case (e.g., with only RGB frame pixels as input). However, under multi-modal scenarios, different modalities may share correlated scene descriptions but meanwhile preserve their own unique observations. In addition, such a modality relationship is not stable but constantly evolves with the highly dynamic environment. An intuitive example is given in Fig. 1 (a) , where we show the observations of both RGB frames and event camera signals [26, 3] on a night-time driving scene. It can be seen that the two modalities only exhibit partially consistent context. Meanwhile, their inconsistencies include critical visual clues, such as traffic lights (in RGB frames) and cars hidden in the dark (in event signals). Existing approaches often overlook this complex and volatile status caused by modality heterogeneity, which may have detrimental impacts on learning correct environmental dynamics. A straightforward solution might be to model the dynamics of each modality separately. However, this approach neglects the complementarity between different data sources, which could reduce its effectiveness.To achieve efficient dynamics modeling in multi-modal scenarios, decoupling and understanding the interplay between modalities and environmental dynamics is necessary. Firstly, modality-correlated features provide a foundational perspective by capturing shared and common information across different sensory inputs. These commonalities are critical for building a cohesive description of the environment's overall behaviour. Secondly, the distinct features in each modality offer inconsistent side views of the environment. In conventional multi-modal visual tasks, these inconsistencies are typically deemed less critical and are mitigated through modality alignment [5, 40, 27] . However, in modeling RL dynamics, the inconsistent features are equally important as they 1) carry critical taskrelated clues and 2) are also the key ingredient in deducing the modality-specific state changes. This importance is also verified by our investigation presented in Sec. 4 . 4 , where we find that incorporating modality inconsistencies leads to a more accurate mimicry of the environment rewards.Based on the above analysis, we propose Dissected Dynamics Modeling (DDM), a method that neatly integrates the decomposition of commonalities and differences across modalities with the dynamics modeling process. As shown in Fig. 1(b) , DDM seeks a mutual predictive property between modalities to establish an implicit regularization and isolate modality-consistent features. Furthermore, it extends this consistency across different temporal locations, ensuring that the learned common information resonates with environmental dynamics. Regarding modality inconsistencies, unlike standard practice that enforces mutual exclusivity with consistent features of the same modality, DDM imposes orthogonal constraints between different modalities to highlight their unique contents. This avoids overly strong regularization, which leads to improved feature quality. The decomposed inconsistent features are only used to establish the dynamics of each modality itself, which creates a more complete environment dynamics without causing interference across modalities. Despite dissecting modality content and model dynamics separately, DDM further introduces a reward predictive function to ensure that both consistent and inconsistent features focus on task-relevant information. This approach preserves information integrity while avoiding potential distractions, which ensures optimized decision-making accuracy and responsiveness.In summary, our work contributes threefold: 1) We analyze the challenges of modeling multi-modal environment dynamics for visual RL, offering insights into the key factors that influence modeling robustness. 2) We present DDM, a novel method that seeks commonality but preserves differences between modalities for enhanced dynamics modeling. 3) Our experimental results validate the effectiveness of our method, demonstrating its strength under complex environmental conditions. 
related work: 
 Visual Reinforcement Learning. Visual reinforcement learning (RL) aims to enable agents to make decisions based on raw visual inputs. Existing methods typically revolve around strategies such as data augmentation to enhance input diversity [50, 29, 30, 31] , incorporating auxiliary tasks for richer representation supervision [23, 53, 19, 35, 46] , employing world models for behavior prediction [14, 13, 38, 49] , or pre-training encoders to improve state representation compression [28, 39, 51] . Despite these advances, current visual RL methods predominantly rely on the RGB modality, which could potentially limit the agent's holistic understanding of the environment.Multi-modal Reinforcement Learning. Handling multi-modal input is important for various control tasks [18, 44, 22] but remains as an under-explored research area in the field of RL. Despite the advancements in multi-modal machine learning methods, applying these techniques to RL faces several challenges due to the highly dynamic RL environment and heterogeneous modalities. A few pioneering works have focused explicitly on multi-modal RL [4, 32, 20] . However, these methods primarily focus on aligning different modalities, using techniques like mutual information optimization [4] or imposing consistency constraints [32] . Such a strategy overlooks the impact of modality-inconsistent aspects on decision-making, which may hinder its effectiveness.Dynamics Modeling in Visual RL. Learning to model the environment dynamics in visual RL is critical for obtaining efficient state representations and reducing computational complexity. Earlier works involve reconstructing input observations at the pixel level [48, 45] . Subsequent studies have focused on dynamics modeling by predicting future latent states [10, 9, 14, 37, 25, 2, 41] , showing promising results. Although these methods have advanced the field, they are not explicitly designed for multi-modal scenarios and may face challenges under complex modality relationships. Unlike existing methods, our approach uniquely dissects modality contents based on their relationships, treating consistent and inconsistent information separately within the dynamics modeling process. 
methodology: 
 We consider the standard RL setting where an agent interacts with the environment in consecutive time steps. At each time step, the agent has access to observations from d distinct modalities, where d > 1. Fig. 2 illustrates the proposed Dissected Dynamics Modeling (DDM) method. DDM is structured around two tightly entangled elements: 1) the decomposition of commonalities and differences across modalities and 2) modeling modality-aware environmental dynamics. Through this modeling process, the decomposed features of each modality are optimized to retain highly abstract and comprehensive task-related information. These features are then applied to learn policies, which promote both decision-making precision and sample efficiency. We now describe our overall approach in detail.Multi-modal Markov Decision Process. The task of multi-modal visual RL can be formulated as a Markov Decision Process with the tuple (S, A, T , R, γ), where the state space is S = d i=1 O Mi represented by the combined observation space comprising d different modalities [20] . A is the action space, T (s t+1 |s t , a t ) is the transition function, R(s, a) is the reward function, and γ ∈ [0, 1) is the discount factor [42] . At every time step t, the agent observes all modality data to take action according to policy π. The environment then returns a reward to the agent. The goal is to optimize the policy by maximizing the expected cumulative reward.Soft Actor Critic. Our approach is based on Soft Actor-Critic (SAC) [11, 12] , which iteratively refines a policy function π and a critic function Q. SAC seeks to maximize the expected cumulative reward while simultaneously encouraging exploration by an α-discounted maximum entropy:EQUATIONThe parameters of the value function are updated by the Bellman equation: where D is the replay buffer, and the soft state value function V (o t+1 ) is calculated as follows:EQUATIONEQUATION)where Q is the exponential moving average of the critic network Q, and ãt+1 is the action produced by the current policy.Feature Encoding and Partitioning. Given the significant disparities in low-level information inputs across different modalities, we utilize d modality-specific encoders to transform high-dimensional observations into deep latent features. These encoders possess identical network architectures, except for the initial input layer, which is adjusted to match the channel number of the corresponding input data shapes. The extracted feature of the i-th modality M i at time step t is denoted as z Mi t . Since our method aims to decompose each modality into common and unique components, we further adopt two separate network branches as information filters for feature partitioning. These two branches process the extracted modality feature z Mi t , producing two distinct outputs: z Mi t as the modalityconsistent feature, and ẑMi t as the modality-inconsistent feature. In the subsequent decomposition phase, these features undergo a series of regularizations to ensure they accurately convey the intended information. For clarity, we mainly introduce the case involving two modalities. The extension to multiple modalities will be discussed subsequently.Extracting Consistencies Between Modalities. Consistent information across modalities provides foundational knowledge about general environmental behavior. To extract such information, we focus on enhancing the mutual predictive properties between different modalities. Given two modalities M 1 and M 2 , we first impose a mutual prediction (mp) constraint on the partitioned modality-consistent features z M1 t and z M2 t at the current time step t:EQUATIONwhere F 1 and F 2 are prediction heads implemented as Multi-Layer Perceptrons (MLP). Such a prediction between modalities triggers a form of content alignment, offering immediate common environmental insights. However, merely imposing the consistency constraint at a static step t may not fully leverage the pivotal temporal dynamics of the environment. Intuitively, observations in adjacent timesteps often preserve similar contexts. For example, in autonomous driving, a car that exists in one timestep will likely remain in the next. Meanwhile, there may be subtle variations in details, such as the car's location, distance, and size. These temporal differences can serve as augmented views that facilitate the discrimination of modality commonalities. Therefore, we create an additional transition prediction constraint across different modalities. Given z M1 t , z M2 t and the action a t taken in time step t, we set up two transition prediction (tp) heads G 1 , G 2 to predict the consistent features z M1 t+1 , z M2 t+1 of the next time step:EQUATIONThis approach creates a synergistic effect on both consistency extraction and dynamics modeling: on one hand, the expanded view in the subsequent time step offers enhanced guidance for learning to identify task-relevant common objects. On the other hand, predicting environmental transitions improves feature quality related to modeling scene dynamics. This interconnection tightly couples the two tasks, allowing them to reinforce each other effectively.Identifying Inconsistencies. Extracting inconsistent information between different modalities is a subtler task than consistency extraction. This subtleness is caused by the absence of clear constraints on feature content: unlike consistency extraction, which involves pulling features closer with concrete objective functions, inconsistency extraction lacks such a definitive goal. For initial consideration, any content that is not consistent across modalities should be deemed as inconsistencies. Therefore, a common practice is to enforce mutual exclusivity between the consistent and inconsistent features of the same modality [21] . Taking modalities M 1 and M 2 as examples, the orthogonality objective can be applied to impose this mutual exclusivity as follows:EQUATIONwhere (•) denotes the dot product operation. However, this objective might be overly strong, as Eq. 6 forces the feature pairs [z M1 t , ẑM1 t ] and [z M2 t , ẑM2 t ] to be completely uncorrelated. Since each pair of features originated from the same modality encoder, they are likely to share similar encoding patterns. Therefore, enforcing orthogonality may lead to excessively departed features and decrease feature expressiveness. To alleviate this issue, we take an alternative solution. Given two modalities M 1 and M 2 , we instead optimize the following objective:EQUATIONEq. 7 directly matches our intention of highlighting different contexts across modalities. During optimization, the similar contexts in M 1 and M 2 are gradually excluded in ẑM1 t and ẑM2 t due to the strict constraint of orthogonality. Meanwhile, this optimization process is less aggressive than Eq. 6 since ẑM1 t and ẑM2 t come from different modality encoders and minimizing their correlation does not affect learning effective representations within each encoder.Extension to Multiple Modalities. So far we have been working on the two-modality case. The extension to more modalities is also straightforward. For consistency extraction, we modify the prediction heads F i and G i for modality M i to take all other modalities as input. The objectives in Eq. 4 and 5 then become:EQUATION)where d > 2 is the modality number and {z| j ̸ = i} denotes all d -1 modality-consistent features except for z Mi t . These d -1 features are concatenated and sent into F i and G i for prediction. Eq. 8 is inspired by set calculation, where each feature is regarded as a set of contexts. It states that for d sets, if the union of any d -1 sets equals the remaining set, then all d sets must be identical and consistent. For inconsistency extraction, we simply extend Eq. 7 as:EQUATIONwhich covers all pair-wise modality combinations.After decomposing modalities into consistent and inconsistent contexts, our approach treats them separately in the overall dynamics modeling process. The consistency features are used to predict cross-modality future states to ensure they capture common scene dynamics, as formulated by the L tp in Eq. 5 and 8. The inconsistent contents are sent into a different pathway, which is used to deduce the full modality-specific evolution. In particular, for each modality M i , we first merge its consistent and inconsistent features into a unified modality representation:EQUATIONthe merged zMi t contains a comprehensive description of M i . Then, we set up a full-state predictive (fp) head P i for each M i , which takes both zMi t and action a t to forecast the next state:EQUATION)Such a full-state prediction ensures the combination of decomposed modality features faithfully captures the modality-specific dynamics. To accurately reflect environmental rewards, a reward predictive function R is further introduced to filter task-irrelevant information, optimized by:EQUATIONwhere z t = zM1 t , zM2 t , . . . , zM d t is the ultimate state representation obtained by concatenating features from all d modalities, and r t+1 is the actual reward returned by the environment.Given the feature decomposition and dynamics modeling processes, the policy-learning of our dissected dynamics modeling (DDM) method involves the joint optimization of multiple objectives that are divided into three groups:EQUATION)where L tp is used in both feature decomposition and dynamics modeling, and the SAC takes the state representation z t as input to replace the observation o t . During training, the DDM learns to extract effective state representations that faithfully reflect the environmental behaviour, yielding improved policy in various multi-modal visual RL tasks. 
experiment settings: 
 Environments. To evaluate the effectiveness of our approach in complex multi-modal scenarios, we conducted tests on two widely used RL benchmarks: 1) the CARLA simulator [7] for autonomous driving, and 2) the DeepMind Control (DMControl) suite [43] for robotic control. For CARLA, we assessed our method under four different environmental conditions: "Normal", "Midnight", "Dazzling", and "Rainy". The latter three conditions provide challenging driving scenarios characterized by For DMControl, we conduct evaluations on six tasks, which are partitioned into two groups based on their difficulty levels. The modalities used are RGB frames and depth images. However, given the relatively clean backgrounds of tasks in DMControl and the high consistency between the two modalities, we additionally introduce inconsistencies by randomly selecting one modality and masking out 20% of its image contents. This intentional introduction of inconsistencies simulates the occlusion issues commonly encountered during real-world machine operations, thereby increasing the task difficulty.Compared Methods. We compare our DDM method with various types of methods, including singlemodal baseline RL algorithms SAC [12] and DrQ [50] , two competitive dynamics modeling methods DeepMDP [9] and SPR [37] , conventional multi-modal methods EFNet [40] and TransFuser [5] , and multi-modal RL methods MUMMI [4] , MAIE [32] , and HAVE [20] . For EFNet and TransFuser, since they were initially designed for traditional computer vision tasks, we mainly evaluate their core modality fusion and alignment modules. These modules are integrated with the SAC algorithm for decision-making, which is the same base RL algorithm used in our method.Implementation Details. Our approach is implemented with Pytorch. To ensure a fair comparison, we use the same encoder network architectures and training hyperparameters for all methods being evaluated. In line with established practices [50, 23] , we convert data from each modality into imagebased representations and stack multiple sequential images to capture temporal dynamics. Methods on the CARLA benchmark and DMControl are trained for 100k and 500k frames, respectively. To obtain reliable results, we train each task-method pair for 5 runs and report the mean values and standard deviations after 20 evaluations. More detailed experimental setups are given in the Appendix.Results on CARLA. Table 1 presents the performance of various methods on the CARLA benchmark using two modalities: RGB frames and event signals. The results demonstrate that our method outperforms others in all four environmental conditions. The table shows that the performances of the SAC and DrQ baselines are less satisfactory, especially in complex conditions such as "Dazzling" and "Midnight". While conventional dynamics modeling methods like DeepMDP and SPR do enhance performance, their improvements are modest compared to our DDM method. For example, DDM obtains an average episode return improvement of approximately 24.4% and 21.3% over DeepMDP and SPR, respectively. This substantial enhancement is attributed to the explicit modality decomposition and processing in DDM, which significantly boosts robustness across diverse multimodal environments. The table also indicates that existing multi-modal methods achieve decent performance on CARLA. However, these methods primarily focus on modality alignment and fusion, neglecting inconsistency contexts critical for decision-making. Consequently, they do not match the efficiency of our DDM, which effectively leverages both consistent and inconsistent modality contexts for robust decision-making. The results with all four modalities are given in the Appendix.Results on DMControl. We further evaluate different methods on the DMControl in Table 2 . Similar to the results on the CARLA benchmark, our method demonstrates significant performance improvements on DMControl. It is apparent that conventional multi-modal methods such as TransFuser and EFNet exhibit limited performance, likely due to the increased inconsistency brought by random masking. In contrast, our DDM method consistently enhances the performance of the SAC baseline by a large margin, further verifying its effectiveness. To systematically verify the effectiveness of each component in DDM, we conduct a series of ablation experiments on the CARLA benchmark. The experiments are conducted by progressively integrating different training objectives in DDM into the baseline SAC algorithm. The results are given in Table 3 . Compared with the baseline, incorporating the full-state prediction objective L f p and reward prediction objective L r notably enhances performance. However, this enhancement does not bring a significant advantage over conventional methods such as DeepMDP and SPR reported in Table 1 , since the modality relationship is not explicitly modeled. Further applying L mp and L tp to model modality consistencies provides additional performance gains, showing the importance of extracting common contexts in multi-modal RL. Finally, the integration of L orth to model modality inconsistencies yields substantial improvement, confirming the effectiveness of our modality decomposition strategy.The Importance of Modality Commonality and Differences. A key finding of our study is the crucial role of both modality commonality and differences in multi-modal dynamics modeling.To investigate this, we leverage the bisimulation metrics [8, 9] for MDPs to assess the dynamics modeling performance of different methods. Specifically, the bisimulation metrics define two states to be behaviourally similar if they have close rewards along the transition trajectory [9] . Therefore, the difference between the estimated and true rewards can reflect how closely the learned states mimic the actual environmental states. In Fig. 3 (a), we plot the absolute reward difference |R(z t , a t ) -r t+1 | for three methods: DeepMDP, DDM without using L orth (i.e., only modeling modality consistency), and the full DDM model. The reward difference is plotted for each time step of the same test sequence on the CARLA benchmark. From the figure, it is clear that compared to DeepMDP, modeling modality consistency can reduce reward estimation errors. Additionally, incorporating modality inconsistency further significantly enhances estimation accuracy. These results confirm the importance of both common and differing modality contexts in accurately learning dynamics.Besides rewards, we also examine the state prediction accuracy of different methods. In Fig. 3 (b), we plot the normalized latent state prediction error||z ′ t -zt+1||||zt+1|| for each method. For DDM methods, z ′ t is defined as [P 1 (z M1 t , a t ), P 2 (z M2 t , a t ), . . . , P d (z M d t , a t )], representing the predicted states from multiple modalities, for DeepMDP, z ′ t represents the predicted single latent state. The figure shows that DeepMDP exhibits significantly higher state prediction errors, highlighting the benefits of decomposing modality contexts and treating each modality separately. Among the two DDM variants, the full DDM model with L orth has less prediction error, again demonstrating the advantages of incorporating task-related inconsistencies. Comparison of Different Approaches for Inconsistency Extraction. In Sec. 3.2, we have discussed two different approaches for inconsistency extraction, i.e., applying the mutual exclusive constraint within the same modality (Eq. 6) or across different modalities (Eq. 7). We conjecture that Eq. 6 might be overly strong, potentially affecting feature expressiveness. To verify this conjecture, we compare the performance of the two inconsistency extraction approaches on the CARLA benchmark. The results in Fig. 7 show that forcing orthogonality within the same modality using Eq. 6 consistently obtains worse performance, indicating that it learns less expressive features compared to our approach.Visualization of Decomposed Modality Features. To confirm that the modality features decomposed by DDM accurately capture the intended contexts, we use t-SNE to visualize both modality-consistent and inconsistent features for RGB and event signals on the CARLA benchmark. As shown in Fig. 5 , DDM successfully aligns the modality-consistent features, resulting in closer feature distances, while the modality-inconsistent features are distinctly separated. Further visualizations in Fig. 6 reveal that the consistent features primarily focus on common objects (e.g., road fences), whereas the inconsistent features highlight objects visible in one modality but not in others (e.g., lane lines in RGB frames).In contrast, DeepMDP tends to indiscriminately focus on all scene objects, lacking contextual discriminativity. These visualizations demonstrate the efficacy of the modality decomposition in DDM, facilitating more precise dynamics modeling and efficient state representation.  
limitations and future works: 
 Our current DDM method has two primary limitations. First, while it effectively simulates multimodal dynamics and enhances representation learning, it lacks integrated planning mechanisms similar to those in MuZero [36] and Dreamer [13, 15, 16, 34, 6] . Research has shown that planning mechanisms can effectively model the impact of various strategies in data-rich scenarios, thus minimizing trial and error in real-world applications. In future works, we plan to integrate long-term action consequences into our decision-making processes by developing decomposed multi-modal world models. Second, our approach focuses primarily on vision-based data inputs. However, recent studies suggest that integrating diverse modal inputs, such as textual and audio data, can yield adaptable representations [47] . Therefore, extending our approach to include additional types of modal inputs represents another promising future direction to enhance its scope and performance. 
conclusion: 
 We have studied the impact of modality heterogeneity on dynamics modeling in multi-modal visual reinforcement learning (RL) tasks. Our investigation reveals that both the commonalities and differences in each modality are crucial for accurate dynamics modeling, albeit playing distinct roles.Based on this insight, we have presented Dissected Dynamics Modeling (DDM), a novel method designed to enhance dynamics modeling and representation learning for multi-modal visual RL. Our approach represents one of the first explorations of modeling environmental dynamics in multi-modal scenarios, providing a fresh context decomposition perspective based on modality relationships.Extensive experiments demonstrate the effectiveness of our approach, which significantly improves representation quality and decision-making performance in diverse environmental conditions. 
acknowledgment: 
 The  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Results with More Modalities on CARLA. Table A1 presents the results of various methods on the CARLA benchmark, utilizing four modalities: RGB frames, event signals, depth images, and LiDAR BEV images. Compared to the results in Table 1 , nearly all methods show improved performance with the increased number of modalities. Additionally, the trends observed in Table A1 follow those in Table 1 , with single-modal methods typically underperforming their multi-modal counterparts. Among all the multi-modal methods, our DDM method consistently maintains the best performance due to its modality decomposition and integration. This comprehensive modeling allows DDM to effectively utilize additional information provided by multiple modalities, leading to robust decision-making compared to other methods.Experiments for visual inputs from multiple camera positions. We have further verified the ability of our model on multiple camera positions. First, we switch the camera view of RGB modality in DMControl. Second, we test on CARLA with RGB and LiDAR BEV as input modalities. LiDAR BEV is a bird view map, whose perspective is very different from RGB. For simplicity, we only compare our method against the most competitive baseline methods on these two environments (i.e., HAVE on CARLA and SPR on DMControl). The results and illustrations of different camera views are presented in Table A2 , Table A3 and Fig. A1 respectively. These results show that our method also works on multiple camera positions.Sample efficiency on DMControl. In Fig. A2 , we evaluate the sample efficiency of different methods on DMControl. Compared to DeepMDP and EFNet, our method demonstrates the highest sample efficiency across five tasks, as shown in the figure. This superior performance is attributed to our method's ability to capture a more comprehensive representation of the environment. By accurately modeling the dynamics of each modality and their interactions, our method can make more informed decisions with fewer samples, enhancing learning efficiency and overall performance. Different masking ratios on DMControl . Our masking operation is performed independently at each timestamp. Therefore, both the masked modality type and the masking locations vary randomly across different timestamps, which simulates a challenging occlusion scenario. For other masking ratios, we further test the ratio of 0%, 40%, 60%, and compare our method with SPR, the most competitive baseline on DMControl. As shown in Table A4 , the results reveal that as the masking ratio increases, both SPR and our DDM experience performance drops. However, DDM still outperforms SPR at different ratios. In this section, we provide a comprehensive description of the experimental setup, including network architectures, environment configurations, and hyper-parameter settings.Modality Encoders. Individual feature encoders are employed for each modality to handle the input data. Each input observation provided to the encoder consists of 3 consecutive frames returned by the environment. This results in input channel numbers of 9 for RGB frames, 15 for event signals, 3 for LiDAR BEV images, and 3 for depth images. All the feature extractors share a common network architecture that comprises four convolutional layers, each followed by ReLU activations. The final layer produces 64 × × 6 feature maps, which serves as the individual modality features z Mi t . are then passed into two separate network branches to get 50-dimensional modality-consistent and inconsistent features z Mi t and ẑMi t , respectively. Actor and Critic Networks. In our method, the actor network, which is composed of three fully connected (FC) layers with ReLU activations placed after the first two layers, produces parameters for a Gaussian distribution over actions. The critic network has a similar structure but outputs an estimated Q-value for a specific state-action pair.Prediction Heads and Reward Prediction Network. The prediction heads, as described in [9] , take the input latent vector and the corresponding action as inputs. They generate the inferred latent vectors through two fully connected (FC) layers, with ReLU activation following the first layer. Similarly, the reward prediction network uses the same architecture but outputs a single-dimensional predicted reward. This network estimates the expected reward for a specific state-action pair, offering critical feedback for dynamics modeling.The four environmental conditions (Normal, Midnight, Dazzling, and Rainy) employed in our study are predefined with specific weather hyper-parameters, as detailed in Table A5 . These conditions provide a comprehensive evaluation of the RL algorithms under various situations. Modalities in CARLA. RGB frames are the default input modality in CARLA, which capture rich texture and color information but may face challenges like motion blur and limited dynamic range under extreme light conditions. Event signal is a new modality provided in CARLA, mimicking the output of neuromorphic event cameras [26, 3] . As shown in Fig. A3 , event signals focus on changes in brightness, offering a wider dynamic range, no motion blur, and high temporal resolution, making them ideal for dynamic scenes. However, they do not generate signals at static regions when no pixel differences occur. In addition, event signals may experience hot spot noises. We simulate this noise to replicate real event signal characteristics. Some key parameters related to these two modalities are provided in Table A6 . Besides RGB frames and event signals, the depth images in CARLA provide distance maps, while LiDAR BEV creates scanning of the vehicle surroundings. Both depth and LiDAR adopt the default simulation parameters in CARLA.Modalities in DMControl. In DMControl, the RGB frame modality captures direct visual information from the environment, providing detailed texture and color data essential for recognizing objects and their spatial relationships. Complementing this, the depth modality generates a depth map where each pixel value represents the distance to the corresponding point in the scene, offering crucial 3D spatial information. The settings of these two modalities are based on the default parameters. In the CARLA autonomous driving task, the objective of the agent is to travel as far as it can on a highway without colliding in various weather conditions. Therefore, the reward function is designed similarly to [52] and [20] :r t = v ⊤ ego ûhighway • ∆t -λ c • collision -λ s • |steer| -λ b • brake (14)The first term is devised to motivate the vehicle to cover as much distance as possible along the highway. v ⊤ ego indicates the velocity vector of the agent vehicle, ûhighway represents the unit direction vector of the highway, and ∆t = 0.1 stands for the discretized simulation time. The final three terms are incorporated to guarantee that the vehicle evades collisions, reduces excessive steering, and avoids sudden braking, where λ c , λ s , and λ b are assigned values of 0.001, 0.1, and 0.1 respectively.The hyper-parameters of RL training are elaborated in Tables A7 and A8 , and they are same to those in [17] and [20] .Answer: [Yes] Justification: Please refer to the code link in the abstract. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper proposes a multimodal Visual RL setting where the input modalities contain similar informative cues in both modalities while may also have inconsistent cues, and the goal is how to capture these disparate features for effective policy learning. The paper approaches this problem from a disentanglement perspective, and proposes to project the consistent cues to a common subspace, while the inconsistent cues between the modalities are disentangled using a mutual orthogonality constraint.

The paper received positive reviews overall, with three weak accept recommendations. While, there have been some concerns initially on i) the lack of theoretical foundations for the approach, ii) lack of clarity in implementation details, iii) experimental analysis, and iv) relations to prior works, authors provided a strong rebuttal addressing the concerns. AC concurs with the reviewers judgement that paper is well-crafted and demonstrates promising results and recommends acceptance. Authors should include the additional technical details that came up in the rebuttal into the final paper. 

Further, the current literature review in the paper is quite limited and is specific to visual RL. There is a large body of works on disentangled representation learning, which is the central idea in this paper and that must be reviewed in detail, especially in the context of cross-modal alignment. Some pointers are given below. Authors should contrast their work to such literature and clarify how the proposed model stands out. 
1. Learning to Decompose and Disentangle Representations for Video Prediction, Hsieh et al. NeurIPS 2018
2. Vector-Decomposed Disentanglement for Domain-Invariant Object Detection, Wu et al., CVPR 2021	
3. Semantics Disentangling for Generalized Zero-Shot Learning, Chen et al., ICCV 2021
4. SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders, Li et al, ECCV 2024

Reviews: review: paper_summary: paper_summary: The paper proposes Dissected Dynamics Modeling (DDM), a dynamics modeling framework for learning latent features in multi-modal visual RL. The methodology focuses on capturing both the shared and distinct information contained across input modalities. The paper presents a multi-modal architecture and training loss designed to accomplish this, and experimentally demonstrates the benefits of DDM on tasks from CARLA and the DeepMind Control Suite.
strength_weakness: strength_weakness: **[W1] Clarity of implementation / experimental details:** The high-level idea of the DDM framework is clear, but some of the implementation details are not clearly described. Some components of the experiments were also not described in detail, which made it difficult to interpret some of the results. Please see the questions below.

**[W2] Connections to related work:** DDM appears to build upon previous works in visual / multi-modal RL for its dynamics modeling and consistency extraction components, but these connections are not made clear. I think it would be useful to better emphasize what parts of DDM build upon existing works vs. what parts are novel contributions. (i) The dynamics modeling approach looks similar to reconstruction-free approaches in visual RL [a,b,c,d], but the similarities / differences are not discussed. (ii) It is mentioned that existing multi-modal approaches have focused on aligning modalities, but the paper does not discuss how the proposed consistency extraction method relates to these approaches. 

**[W3] Heuristic design choices:** Design choices of the proposed architecture are justified through experimental results, but no theoretical support is provided (or connections to existing approaches that provide theoretical support).

---

References:

[a] Gelada et al. (2019). DeepMDP: Learning continuous latent space models for representation learning.

[b] Schwarzer et al. (2021). Data-efficient reinforcement learning with self-predictive representations.

[c] Okada et al. (2021). Dreaming: Model-based reinforcement learning by latent imagination without reconstruction.

[d] Zhang et al. (2021). Learning invariant representations for reinforcement learning without reconstruction.
questions: questions: **Methodology:**

**[Q1]** Are gradients with respect to the latent feature encodings $z$ stopped in any components of the loss function in (13), or are they propagated through the loss function everywhere that $z$ appears?

**[Q2]** Are the next-step target values $z_{t+1}$ in (5) and (11) calculated using the same feature encoder used for $z_t$? It would be helpful to make this more clear. Are gradients taken through $z_{t+1}$ in (5) and (11)?

---

**Experiments:**

**[Q3]** What do the single modality baseline results represent? Do they only use RGB images, or do they consider a combined input that incorporates information from all modalities? If these baselines are not restricted to RGB images, it would be useful to include results (baselines or ablation of DDM) using only RGB inputs to demonstrate that the use of multi-modal inputs leads to improved performance over standard visual RL.

**[Q4]** DreamerV3 [e] and DrQ-v2 [f] are strong model-based and model-free visual RL algorithms, respectively. Why were DeepMDP and SPR chosen as baselines instead of DreamerV3 (which has reported better head-to-head performance vs. SPR in Atari 100k)? Why was DrQ chosen as a baseline instead of DrQ-v2 (which has reported significant improvements over DrQ on DeepMind Control Suite)? 

**[Q5]** Has performance converged by the end of training for the experimental results, given the number of training steps used (100k and 500k in CARLA and DeepMind Control Suite, respectively)? In prior works on visual RL, DeepMind Control Suite tasks are typically trained for longer than this (1M steps in DreamerV3 [e], 3M steps in DrQ-v2 [f]), and these baselines had not converged to final performance after 500k steps.

---

**Minor:**

Spelling: Inon. --> Incon. (p. 2, Figure 1 caption)

---

References:

[e] Hafner et al. (2023). Mastering diverse domains through world models.

[f] Yarats et al. (2021). Mastering visual continuous control: Improved data-augmented reinforcement learning.
limitations: limitations: The authors have addressed the limitations of the current work.
paper_summary: paper_summary: The paper presents a solution for better multimodal dynamic modeling in visual RL. The paper claims that existing works only emphasize consistent (aligned) information across modalities, leaving out the opportunity for the model to benefit from the inconsistent features. The work introduces a new consistency loss, where the (cross-modal) mutual prediction happens dynamically, meaning a modality has to infer the other modality's features for the next step rather than the current one. Furthermore, the work introduces a "soft objective" for cross-modal feature orthogonalization. The work shows consistent improvement over the existing state-of-the-art on CARLA and DMControl benchmarks.
strength_weakness: strength_weakness: 1) The authors report the episode to return and driving distance but do not report DS/RC/IP, which is reported in other methods mentioned in the papers and could be more informative for evaluating driving performance. Is there any reason why the authors do not report those metrics? 

2) L265-267 states that   $ \mathcal{L_{fp}} $ and $ \mathcal{L}_{r} $ do not benefit much to the existing DeepMDPandSPR. Are there any supporting references/results for this claim?
questions: questions: 1) Do the authors think that incorporating a component of their modeling, e.g., $ \mathcal{L_{tp}} $, into existing models could result in a boost? 

2) In Section 3.2 [L166], the authors suggest replacing a stronger objective (Eq. 6) with Eq. 7. Did you observe in your experiments that, indeed, a model trained with Eq.7 is better than Eq.6?
limitations: limitations: The authors discussed potential limitations.
paper_summary: paper_summary: The paper proposed a method, namely Dissected Dynamics Modeling (DDM), for multi-modal environment dynamics modeling in visual RL. The core idea is to adopt additional modules to extract separate modality-consistent and modality-inconsistent features in each modality stream with designated losses as the regularization. During training, the model tries to maximize mutual information between modality-consistent features from different modalities at the current and the next timestamp. For modality-inconsistent features from, the model tries to enforce orthogonality. Experiments on CARLA and DMControl indicated the superiority of DDM over other state-of-the-arts and further analyses suggested the effectiveness of DDM v.s. DeepMDP. Ablation studies supported some of the important design choices.
strength_weakness: strength_weakness: - The proposed method is only showed to be working on a subset of modalities that are visual only, which greatly limits its generalizability and contribution as a method for multi-modal learning. However, visual RL is not my expertise and thus I cannot evaluate the contribution on this particular field.

- Following above, from the experiments it seems that the method also only works on visual inputs from the same camera perspective. i wonder if there are experiments demonstrating the model also works for visual inputs from multiple camera positions.

- For experiments on DMControl, how is the masking done for frames at different timestamps? Also, besides 20% masking, do the authors have results with other masking ratio?

- Some places are not written in a technically accurate way, though it's hard to tell whether it is because of technical misunderstanding or simply misuse of words. For example:
  - Line 46 - 48, "Firstly, modality-correlated features provide a foundational perspective by capturing shared and complemented information across different sensory inputs." The modality-consistent features that are shared among modalities does not COMPLEMENT each other. Instead, those modality-inconsistent features that are unique for each modality themselves complement other modalities.
  - Line 50-51, "these inconsistencies are typically deemed less critical and are filtered out through modality alignment". Modality alignment should not have the effect of filtering out modality-inconsistent features, in my understanding. If the authors have some evidence, I'd like to see.
questions: questions: Please refer to the above Weaknesses part and address my concern accordingly
limitations: limitations: Yes, limitations are discussed.

Comments: title: Appreciation for the supportive feedback and recommendation
Thank you for the positive support
Thank you for the supportive feedback
Response
comment: We are truly grateful for your positive comments and for recognizing the efforts in our response and additional results. Thank you once again for your support and the time you invested in reviewing our work thoroughly.
Dear authors, 
I greatly appreciate your responses and the additional results presented in the PDF.
I think the authors addressed all my comments and I think this work should be accepted.
We appreciate the comments that provide new perspectives on both experimental design and future research directions. We are glad to hear that your concerns have been addressed satisfactorily. We will ensure that the discussed changes and results are integrated into the revised manuscript to enhance its clarity and robustness. Thank you again for your positive support.
I appreciate the authors' responses, which addressed my concerns well. After reading other reviews and responses, I decided to update my rating from Borderline Accept to Weak Accept.

Please make necessary modification and add the relevant clarification, discussion, and experiment results to your final paper.
We appreciate the guidance provided in your detailed and informative review. We will incorporate these discussions and results into our revised manuscript. Thank you again for your supportive feedback.
Thank you for the detailed responses and additional experimental results. Incorporating some of these discussions / results into the paper will improve clarity and further strengthen the experimental analysis. My main questions and concerns have been addressed, and I have increased my overall review score to reflect this.