Title: Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL
Venue: NeurIPS 2024
Accepted: yes
Keywords: Visual Reinforcement Learning
Abstract: Accurate environment dynamics modeling is crucial for obtaining effective state representations in visual reinforcement learning (RL) applications. However, when facing multiple input modalities, existing dynamics modeling methods (e.g., DeepMDP) usually stumble in addressing the complex and volatile relationship between different modalities. In this paper, we study the problem of efficient dynamics modeling for multi-modal visual RL. We find that under the existence of modality heterogeneity, modality-correlated and distinct features are equally important but play different roles in reflecting the evolution of environmental dynamics. Motivated by this fact, we propose Dissected Dynamics Modeling (DDM), a novel multi-modal dynamics modeling method for visual RL. Unlike existing methods, DDM explicitly distinguishes consistent and inconsistent information across modalities and treats them separately with a divide-and-conquer strategy. This is done by dispatching the features carrying different information into distinct dynamics modeling pathways, which naturally form a series of implicit regularizations along the learning trajectories. In addition, a reward predictive function is further introduced to filter task-irrelevant information in both modality-consistent and inconsistent features, ensuring information integrity while avoiding potential distractions. Extensive experiments show that DDM consistently achieves competitive performance in challenging multi-modal visual environments.

Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL

Abstract

Accurate environment dynamics modeling is crucial for obtaining effective state representations in visual reinforcement learning (RL) applications. However, when facing multiple input modalities, existing dynamics modeling methods ( e.g. , DeepMDP) usually stumble in addressing the complex and volatile relationship between different modalities. In this paper, we study the problem of efficient dynamics modeling for multi-modal visual RL. We find that under the existence of modality heterogeneity, modality-correlated and distinct features are equally important but play different roles in reflecting the evolution of environmental dynamics. Motivated by this fact, we propose Dissected Dynamics Modeling (DDM), a novel multi-modal dynamics modeling method for visual RL. Unlike existing methods, DDMexplicitly distinguishes consistent and inconsistent information across modalities and treats them separately with a divide-and-conquer strategy. This is done by dispatching the features carrying different information into distinct dynamics modeling pathways, which naturally form a series of implicit regularizations along the learning trajectories. In addition, a reward predictive function is further introduced to filter task-irrelevant information in both modality-consistent and inconsistent features, ensuring information integrity while avoiding potential distractions. Extensive experiments show that DDM consistently achieves competitive performance in challenging multi-modal visual environments. The code is available in this link: https://github.com/Yara-HYR/DDM

1 Introduction

Recent years have witnessed a growing interest in visual reinforcement learning (RL). By leveraging raw pixel inputs directly from sensory data, visual RL allows agents to navigate and interpret their environments with unprecedented precision and adaptability [33, 24, 51]. Since raw pixels are usually redundant and suffer from expanded data dimensionality, a key aspect of visual RL is to acquire efficient and compact state representations. A typical approach involves modeling environmental dynamics in an abstracted state space [1], where the state representations are supposed to mimic the actual environmental changes but operate on a sufficiently low dimension. One classical solution is DeepMDP [9], which leverages neural networks to compress high-dimensional input data. Given consecutive observations, DeepMDP predicts reward and state transitions within the deep latent space, enabling more effective policy optimization over complex, dynamic environments.

ҧ

ҧ

𝑜

𝑡

𝑀𝑗

𝑜

𝑡+1

𝑀𝑗

Ƹ 𝑧

𝑡

𝑀𝑗

𝑜 𝑡

𝑀𝑖

𝑜 𝑡+1

𝑀𝑖

𝑧

𝑡+1

𝑀𝑗

Ƹ 𝑧

𝑡

𝑀𝑖

𝑧

𝑡+1

𝑀𝑖

𝑎𝑡

𝑟 𝑡+1

𝑧

𝑡

𝑀𝑖

Ƹ 𝑧

𝑡+1

𝑀𝑖

𝑧

𝑡

𝑀𝑗

Ƹ 𝑧

𝑡+1

𝑀𝑗

Consistency Information Only→Turn left

Car

Car

RGB frames

Event signals

Consistency

+

Inconsistency   →   Brake

Car

Red light

𝑎𝑡

𝑟 𝑡+1

𝑧𝑡

𝑧𝑡+1

Consistent Context

RGB

Inconsistency

Event

Inconsistency

Observed Information

Modality

𝑖

Modality

𝑗

Action Taken

Con.

Incon.

Predict

Orthogonalize

𝑜𝑡

𝑜𝑡+1

(a) Complex modality relationships in visual RL

(b) Comparison between DeepMDP (left) and our method (right)

ҧ

Although methods like DeepMDP have proven effective for environmental dynamics modeling, research on this subject has been mostly restricted to the single visual modality case ( e.g. , with only RGB frame pixels as input). However, under multi-modal scenarios, different modalities may share correlated scene descriptions but meanwhile preserve their own unique observations. In addition, such a modality relationship is not stable but constantly evolves with the highly dynamic environment. An intuitive example is given in Fig. 1 (a), where we show the observations of both RGB frames and event camera signals [26, 3] on a night-time driving scene. It can be seen that the two modalities only exhibit partially consistent context. Meanwhile, their inconsistencies include critical visual clues, such as traffic lights (in RGB frames) and cars hidden in the dark (in event signals). Existing approaches often overlook this complex and volatile status caused by modality heterogeneity, which may have detrimental impacts on learning correct environmental dynamics. A straightforward solution might be to model the dynamics of each modality separately. However, this approach neglects the complementarity between different data sources, which could reduce its effectiveness.

To achieve efficient dynamics modeling in multi-modal scenarios, decoupling and understanding the interplay between modalities and environmental dynamics is necessary. Firstly , modality-correlated features provide a foundational perspective by capturing shared and common information across different sensory inputs. These commonalities are critical for building a cohesive description of the environment's overall behaviour. Secondly , the distinct features in each modality offer inconsistent side views of the environment. In conventional multi-modal visual tasks, these inconsistencies are typically deemed less critical and are mitigated through modality alignment [5, 40, 27]. However, in modeling RL dynamics, the inconsistent features are equally important as they 1) carry critical taskrelated clues and 2) are also the key ingredient in deducing the modality-specific state changes. This importance is also verified by our investigation presented in Sec. 4.4, where we find that incorporating modality inconsistencies leads to a more accurate mimicry of the environment rewards.

Based on the above analysis, we propose Dissected Dynamics Modeling (DDM) , a method that neatly integrates the decomposition of commonalities and differences across modalities with the dynamics modeling process. As shown in Fig. 1(b), DDM seeks a mutual predictive property between modalities to establish an implicit regularization and isolate modality-consistent features. Furthermore, it extends this consistency across different temporal locations, ensuring that the learned common information resonates with environmental dynamics. Regarding modality inconsistencies, unlike standard practice that enforces mutual exclusivity with consistent features of the same modality, DDMimposes orthogonal constraints between different modalities to highlight their unique contents. This avoids overly strong regularization, which leads to improved feature quality. The decomposed inconsistent features are only used to establish the dynamics of each modality itself, which creates a more complete environment dynamics without causing interference across modalities. Despite dissecting modality content and model dynamics separately, DDM further introduces a reward predictive function to ensure that both consistent and inconsistent features focus on task-relevant information. This approach preserves information integrity while avoiding potential distractions, which ensures optimized decision-making accuracy and responsiveness.

In summary, our work contributes threefold: 1) We analyze the challenges of modeling multi-modal environment dynamics for visual RL, offering insights into the key factors that influence modeling robustness. 2) We present DDM, a novel method that seeks commonality but preserves differences

ҧ

between modalities for enhanced dynamics modeling. 3) Our experimental results validate the effectiveness of our method, demonstrating its strength under complex environmental conditions.

2 Related Work

Visual Reinforcement Learning. Visual reinforcement learning (RL) aims to enable agents to make decisions based on raw visual inputs. Existing methods typically revolve around strategies such as data augmentation to enhance input diversity [50, 29, 30, 31], incorporating auxiliary tasks for richer representation supervision [23, 53, 19, 35, 46], employing world models for behavior prediction [14, 13, 38, 49], or pre-training encoders to improve state representation compression [28, 39, 51]. Despite these advances, current visual RL methods predominantly rely on the RGB modality, which could potentially limit the agent's holistic understanding of the environment.

Multi-modal Reinforcement Learning. Handling multi-modal input is important for various control tasks [18, 44, 22] but remains as an under-explored research area in the field of RL. Despite the advancements in multi-modal machine learning methods, applying these techniques to RL faces several challenges due to the highly dynamic RL environment and heterogeneous modalities. A few pioneering works have focused explicitly on multi-modal RL [4, 32, 20]. However, these methods primarily focus on aligning different modalities, using techniques like mutual information optimization [4] or imposing consistency constraints [32]. Such a strategy overlooks the impact of modality-inconsistent aspects on decision-making, which may hinder its effectiveness.

Dynamics Modeling in Visual RL. Learning to model the environment dynamics in visual RL is critical for obtaining efficient state representations and reducing computational complexity. Earlier works involve reconstructing input observations at the pixel level [48, 45]. Subsequent studies have focused on dynamics modeling by predicting future latent states [10, 9, 14, 37, 25, 2, 41], showing promising results. Although these methods have advanced the field, they are not explicitly designed for multi-modal scenarios and may face challenges under complex modality relationships. Unlike existing methods, our approach uniquely dissects modality contents based on their relationships, treating consistent and inconsistent information separately within the dynamics modeling process.

3 Methodology

Weconsider the standard RL setting where an agent interacts with the environment in consecutive time steps. At each time step, the agent has access to observations from d distinct modalities, where d > 1 . Fig. 2 illustrates the proposed Dissected Dynamics Modeling (DDM) method. DDM is structured around two tightly entangled elements: 1) the decomposition of commonalities and differences across modalities and 2) modeling modality-aware environmental dynamics. Through this modeling process, the decomposed features of each modality are optimized to retain highly abstract and comprehensive task-related information. These features are then applied to learn policies, which promote both decision-making precision and sample efficiency. We now describe our overall approach in detail.

3.1 Preliminaries

Multi-modal Markov Decision Process. The task of multi-modal visual RL can be formulated as a Markov Decision Process with the tuple ( S , A , T , R , γ ) , where the state space is S = ∏ d i =1 O M i represented by the combined observation space comprising d different modalities [20]. A is the action space, T ( s t +1 | s t , a t ) is the transition function, R ( s, a ) is the reward function, and γ ∈ [0 , 1) is the discount factor [42]. At every time step t , the agent observes all modality data to take action according to policy π . The environment then returns a reward to the agent. The goal is to optimize the policy by maximizing the expected cumulative reward.

Soft Actor Critic. Our approach is based on Soft Actor-Critic (SAC) [11, 12], which iteratively refines a policy function π and a critic function Q . SAC seeks to maximize the expected cumulative reward while simultaneously encouraging exploration by an α -discounted maximum entropy:

The parameters of the value function are updated by the Bellman equation:

Ƹ 𝑧

𝑡

𝑀𝑖

Modality-aware Dynamics Modeling

Observation Encoder

Value network

Policy network

Ƹ 𝑧

𝑡

𝑀𝑗

Ƹ 𝑧

𝑡

𝑀𝑖

𝑧

𝑡

𝑀𝑗

𝑧

𝑡

𝑀𝑖

Policy Learning

𝑎𝑡

𝑎𝑡

𝑎𝑡

𝑎𝑡

Ƹ 𝑧

𝑡

𝑀𝑗

Ƹ 𝑧

𝑡

𝑀𝑖

𝑧

𝑡

𝑀𝑗

𝑧

𝑡

𝑀𝑖

Ƹ 𝑧

𝑡+1

𝑀𝑗

Ƹ 𝑧

𝑡+1

𝑀𝑖

𝑧

𝑡+1

𝑀𝑗

𝑧

𝑡+1

𝑀𝑖

MLP

𝑧t

𝑎𝑡

𝑄

RGB-Con.

Event-Con.

RGB-Incon.

Event-Incon.

Ƹ 𝑧

𝑡

𝑀𝑗

Ƹ 𝑧

𝑡

𝑀𝑖

𝑧

𝑡

𝑀𝑗

𝑧

𝑡

𝑀𝑖

Time

Time

RGB

Frames

Event

Signals

𝑟 𝑡+1

𝑎𝑡

ҧ

ҧ

ҧ

ҧ

where D is the replay buffer, and the soft state value function V ( o t +1 ) is calculated as follows:

where ¯ Q is the exponential moving average of the critic network Q , and ˜ a t +1 is the action produced by the current policy.

3.2 Modality Decomposition

Feature Encoding and Partitioning. Given the significant disparities in low-level information inputs across different modalities, we utilize d modality-specific encoders to transform high-dimensional observations into deep latent features. These encoders possess identical network architectures, except for the initial input layer, which is adjusted to match the channel number of the corresponding input data shapes. The extracted feature of the i -th modality M i at time step t is denoted as z M i t . Since our method aims to decompose each modality into common and unique components, we further adopt two separate network branches as information filters for feature partitioning. These two branches process the extracted modality feature z M i t , producing two distinct outputs: z M i t as the modalityconsistent feature, and ˆ z M i t as the modality-inconsistent feature. In the subsequent decomposition phase, these features undergo a series of regularizations to ensure they accurately convey the intended information. For clarity, we mainly introduce the case involving two modalities. The extension to multiple modalities will be discussed subsequently.

Extracting Consistencies Between Modalities. Consistent information across modalities provides foundational knowledge about general environmental behavior. To extract such information, we focus on enhancing the mutual predictive properties between different modalities. Given two modalities M 1 and M 2 , we first impose a mutual prediction (mp) constraint on the partitioned modality-consistent features z M 1 t and z M 2 t at the current time step t :

where F 1 and F 2 are prediction heads implemented as Multi-Layer Perceptrons (MLP). Such a prediction between modalities triggers a form of content alignment, offering immediate common environmental insights. However, merely imposing the consistency constraint at a static step t may not fully leverage the pivotal temporal dynamics of the environment. Intuitively, observations in adjacent timesteps often preserve similar contexts. For example, in autonomous driving, a car that exists in one timestep will likely remain in the next. Meanwhile, there may be subtle variations in details, such as the car's location, distance, and size. These temporal differences can serve as augmented views that facilitate the discrimination of modality commonalities. Therefore, we create an additional transition prediction constraint across different modalities. Given z M 1 t , z M 2 t and the

ҧ

෩

෡

𝑀𝑖

𝑡

ҧ

ҧ

ҧ

action a t taken in time step t , we set up two transition prediction (tp) heads G 1 , G 2 to predict the consistent features z M 1 t +1 , z M 2 t +1 of the next time step:

This approach creates a synergistic effect on both consistency extraction and dynamics modeling: on one hand, the expanded view in the subsequent time step offers enhanced guidance for learning to identify task-relevant common objects. On the other hand, predicting environmental transitions improves feature quality related to modeling scene dynamics. This interconnection tightly couples the two tasks, allowing them to reinforce each other effectively.

Identifying Inconsistencies. Extracting inconsistent information between different modalities is a subtler task than consistency extraction. This subtleness is caused by the absence of clear constraints on feature content: unlike consistency extraction, which involves pulling features closer with concrete objective functions, inconsistency extraction lacks such a definitive goal. For initial consideration, any content that is not consistent across modalities should be deemed as inconsistencies. Therefore, a common practice is to enforce mutual exclusivity between the consistent and inconsistent features of the same modality [21]. Taking modalities M 1 and M 2 as examples, the orthogonality objective can be applied to impose this mutual exclusivity as follows:

where ( · ) denotes the dot product operation. However, this objective might be overly strong, as Eq. 6 forces the feature pairs [ z M 1 t , ˆ z M 1 t ] and [ z M 2 t , ˆ z M 2 t ] to be completely uncorrelated. Since each pair of features originated from the same modality encoder, they are likely to share similar encoding patterns. Therefore, enforcing orthogonality may lead to excessively departed features and decrease feature expressiveness. To alleviate this issue, we take an alternative solution. Given two modalities M 1 and M 2 , we instead optimize the following objective:

Eq. 7 directly matches our intention of highlighting different contexts across modalities. During optimization, the similar contexts in M 1 and M 2 are gradually excluded in ˆ z M 1 t and ˆ z M 2 t due to the strict constraint of orthogonality. Meanwhile, this optimization process is less aggressive than Eq. 6 since ˆ z M 1 t and ˆ z M 2 t come from different modality encoders and minimizing their correlation does not affect learning effective representations within each encoder.

Extension to Multiple Modalities. So far we have been working on the two-modality case. The extension to more modalities is also straightforward. For consistency extraction, we modify the prediction heads F i and G i for modality M i to take all other modalities as input. The objectives in Eq. 4 and 5 then become:

̸

̸

̸

where d > 2 is the modality number and { z M j t | j = i } denotes all d -1 modality-consistent features except for z M i t . These d -1 features are concatenated and sent into F i and G i for prediction. Eq. 8 is inspired by set calculation, where each feature is regarded as a set of contexts. It states that for d sets, if the union of any d -1 sets equals the remaining set, then all d sets must be identical and consistent. For inconsistency extraction, we simply extend Eq. 7 as:

which covers all pair-wise modality combinations.

3.3 Modality-aware Dynamics Modeling

After decomposing modalities into consistent and inconsistent contexts, our approach treats them separately in the overall dynamics modeling process. The consistency features are used to predict cross-modality future states to ensure they capture common scene dynamics, as formulated by the

L tp in Eq. 5 and 8. The inconsistent contents are sent into a different pathway, which is used to deduce the full modality-specific evolution. In particular, for each modality M i , we first merge its consistent and inconsistent features into a unified modality representation:

the merged ˜ z M i t contains a comprehensive description of M i . Then, we set up a full-state predictive (fp) head P i for each M i , which takes both ˜ z M i t and action a t to forecast the next state:

Such a full-state prediction ensures the combination of decomposed modality features faithfully captures the modality-specific dynamics. To accurately reflect environmental rewards, a reward predictive function R is further introduced to filter task-irrelevant information, optimized by:

where z t = [ ˜ z M 1 t , ˜ z M 2 t , . . . , ˜ z M d t ] is the ultimate state representation obtained by concatenating features from all d modalities, and r t +1 is the actual reward returned by the environment.

3.4 Policy-learning with DDM

Given the feature decomposition and dynamics modeling processes, the policy-learning of our dissected dynamics modeling (DDM) method involves the joint optimization of multiple objectives that are divided into three groups:

where L tp is used in both feature decomposition and dynamics modeling, and the SAC takes the state representation z t as input to replace the observation o t . During training, the DDM learns to extract effective state representations that faithfully reflect the environmental behaviour, yielding improved policy in various multi-modal visual RL tasks.

4 Experiments

4.1 Experiment Settings

Environments. To evaluate the effectiveness of our approach in complex multi-modal scenarios, we conducted tests on two widely used RL benchmarks: 1) the CARLA simulator [7] for autonomous driving, and 2) the DeepMind Control (DMControl) suite [43] for robotic control. For CARLA, we assessed our method under four different environmental conditions: 'Normal', 'Midnight', 'Dazzling', and 'Rainy'. The latter three conditions provide challenging driving scenarios characterized by

extreme lighting and heavy rainfall, closely mimicking real-world driving scenes. Various modalities provided by the benchmark are utilized, including RGB frames, event signals, depth images, and LiDAR images in a bird's-eye view (BEV) perspective. For DMControl, we conduct evaluations on six tasks, which are partitioned into two groups based on their difficulty levels. The modalities used are RGB frames and depth images. However, given the relatively clean backgrounds of tasks in DMControl and the high consistency between the two modalities, we additionally introduce inconsistencies by randomly selecting one modality and masking out 20% of its image contents. This intentional introduction of inconsistencies simulates the occlusion issues commonly encountered during real-world machine operations, thereby increasing the task difficulty.

Compared Methods. We compare our DDM method with various types of methods, including singlemodal baseline RL algorithms SAC [12] and DrQ [50], two competitive dynamics modeling methods DeepMDP [9] and SPR [37], conventional multi-modal methods EFNet [40] and TransFuser [5], and multi-modal RL methods MUMMI [4], MAIE [32], and HAVE [20]. For EFNet and TransFuser, since they were initially designed for traditional computer vision tasks, we mainly evaluate their core modality fusion and alignment modules. These modules are integrated with the SAC algorithm for decision-making, which is the same base RL algorithm used in our method.

Implementation Details. Our approach is implemented with Pytorch. To ensure a fair comparison, we use the same encoder network architectures and training hyperparameters for all methods being evaluated. In line with established practices [50, 23], we convert data from each modality into imagebased representations and stack multiple sequential images to capture temporal dynamics. Methods on the CARLA benchmark and DMControl are trained for 100k and 500k frames, respectively. To obtain reliable results, we train each task-method pair for 5 runs and report the mean values and standard deviations after 20 evaluations. More detailed experimental setups are given in the Appendix.

4.2 Comparison with State-of-the-art

Results on CARLA. Table 1 presents the performance of various methods on the CARLA benchmark using two modalities: RGB frames and event signals. The results demonstrate that our method outperforms others in all four environmental conditions. The table shows that the performances of the SAC and DrQ baselines are less satisfactory, especially in complex conditions such as 'Dazzling' and 'Midnight'. While conventional dynamics modeling methods like DeepMDP and SPR do enhance performance, their improvements are modest compared to our DDM method. For example, DDM obtains an average episode return improvement of approximately 24.4% and 21.3% over DeepMDP and SPR, respectively. This substantial enhancement is attributed to the explicit modality decomposition and processing in DDM, which significantly boosts robustness across diverse multimodal environments. The table also indicates that existing multi-modal methods achieve decent performance on CARLA. However, these methods primarily focus on modality alignment and fusion, neglecting inconsistency contexts critical for decision-making. Consequently, they do not match the efficiency of our DDM, which effectively leverages both consistent and inconsistent modality contexts for robust decision-making. The results with all four modalities are given in the Appendix.

Results on DMControl. We further evaluate different methods on the DMControl in Table 2. Similar to the results on the CARLA benchmark, our method demonstrates significant performance improvements on DMControl. It is apparent that conventional multi-modal methods such as TransFuser and EFNet exhibit limited performance, likely due to the increased inconsistency brought by random masking. In contrast, our DDM method consistently enhances the performance of the SAC baseline by a large margin, further verifying its effectiveness.

0

5

10

15

0

100

200

300

400

Reward estimation errors

0

5

10

15

20

25

0

100

200

300

400

Reward estimation errors

(b) State prediction errors of different methods

(a) Reward estimation errors of different methods

DDM(w/o

ℒ𝑜𝑟𝑡ℎ

)

DDM(full)

DeepMDP

DDM(full) trend

DDM(w/o

ℒ𝑜𝑟𝑡ℎ

) trend

DeepMDP trend

0

1

2

0

200

400

State prediction errors

0

1

2

0

200

400

State prediction errors

Normal

Midnight

Normal

Midnight

4.3 Ablation Study

To systematically verify the effectiveness of each component in DDM, we conduct a series of ablation experiments on the CARLA benchmark. The experiments are conducted by progressively integrating different training objectives in DDM into the baseline SAC algorithm. The results are given in Table 3. Compared with the baseline, incorporating the full-state prediction objective L fp and reward prediction objective L r notably enhances performance. However, this enhancement does not bring a significant advantage over conventional methods such as DeepMDP and SPR reported in Table 1, since the modality relationship is not explicitly modeled. Further applying L mp and L tp to model modality consistencies provides additional performance gains, showing the importance of extracting common contexts in multi-modal RL. Finally, the integration of L orth to model modality inconsistencies yields substantial improvement, confirming the effectiveness of our modality decomposition strategy.

4.4 Discussion and Analysis

The Importance of Modality Commonality and Differences. A key finding of our study is the crucial role of both modality commonality and differences in multi-modal dynamics modeling. To investigate this, we leverage the bisimulation metrics [8, 9] for MDPs to assess the dynamics modeling performance of different methods. Specifically, the bisimulation metrics define two states to be behaviourally similar if they have close rewards along the transition trajectory [9]. Therefore, the difference between the estimated and true rewards can reflect how closely the learned states mimic the actual environmental states. In Fig. 3(a), we plot the absolute reward difference | R ( z t , a t ) -r t +1 | for three methods: DeepMDP, DDM without using L orth ( i.e. , only modeling modality consistency), and the full DDM model. The reward difference is plotted for each time step of the same test sequence on the CARLA benchmark. From the figure, it is clear that compared to DeepMDP, modeling modality consistency can reduce reward estimation errors. Additionally, incorporating modality inconsistency further significantly enhances estimation accuracy. These results confirm the importance of both common and differing modality contexts in accurately learning dynamics.

Besides rewards, we also examine the state prediction accuracy of different methods. In Fig. 3(b), we plot the normalized latent state prediction error || z ′ t -z t +1 || || z t +1 || for each method. For DDM methods, z ′ t is defined as [ P 1 (˜ z M 1 t , a t ) , P 2 (˜ z M 2 t , a t ) , . . . , P d (˜ z M d t , a t )] , representing the predicted states from multiple modalities, for DeepMDP, z ′ t represents the predicted single latent state. The figure shows that DeepMDP exhibits significantly higher state prediction errors, highlighting the benefits of decomposing modality contexts and treating each modality separately. Among the two DDM variants, the full DDM model with L orth has less prediction error, again demonstrating the advantages of incorporating task-related inconsistencies.

(b) Sample efficiency on CARLA

152

189

108

212

261

254

138

280

286

244

223

267

338

300

267

314

Normal

Midnight

Dazzling

Rainy

20K Steps

100K Steps

DDM

DeepMDP

Driving distance (m)

238

294

66

305

286

338

Normal

187

252

42

275

244

300

Midnight

DDM

DeepMDP

Original

+Random Noise

+Random Mask

Driving Distance

Driving Distance

(a) Policy robustness on CARLA

Normal

Dazzling

Midnight

Rainy

Event Con.

RGB Incon.

40

20

20

30

15

10

10

10

20

0

10

0

-10

10

-10

02-

20

-10

15

02-

-30

20

30

40

20

0

20

40

-60

40

20

0

20

40

40

20

20

40

60

-30

20

10

0

10

20

30

Policy Robustness and Sample Efficiency. The modality decomposition and reward prediction in DDMefficiently extract task-related contexts and filter out irrelevant content, potentially enhancing policy robustness against distractions like data noise. To verify this, we evaluate DDM and DeepMDP on the CARLA benchmark with random Gaussian noises or random masks added to the input modality data. The results in Fig. 4(a) show that DDM experiences minimal performance degradation, highlighting its robustness. In contrast, DeepMDP exhibits a significant performance decline, particularly when faced with random input noises. Despite policy robustness, we further investigate the sample efficiency of the two methods. The results in Fig. 4(b) show that DDM converges faster and achieves better accuracy than DeepMDP in various environmental conditions. These advantages highlight the superiority of DDM, demonstrating its potential for learning diverse visual control tasks.

Comparison of Different Approaches for Inconsistency Extraction. In Sec. 3.2, we have discussed two different approaches for inconsistency extraction, i.e. , applying the mutual exclusive constraint within the same modality (Eq. 6) or across different modalities (Eq. 7). We conjecture that Eq. 6 might be overly strong, potentially affecting feature expressiveness. To verify this conjecture, we compare the performance of the two inconsistency extraction approaches on the CARLA benchmark. The results in Fig. 7 show that forcing orthogonality within the same modality using Eq. 6

Con.

⊥

Incon. (Eq.6)

Incon.

⊥ I

ncon . (Eq.7)

305

338

291

300

249

267

299

314

Driving distance

Normal

Midnight

Dazzling

Rainy

consistently obtains worse performance, indicating that it learns less expressive features compared to our approach.

Visualization of Decomposed Modality Features. To confirm that the modality features decomposed by DDM accurately capture the intended contexts, we use t-SNE to visualize both modality-consistent and inconsistent features for RGB and event signals on the CARLA benchmark. As shown in Fig.5, DDMsuccessfully aligns the modality-consistent features, resulting in closer feature distances, while the modality-inconsistent features are distinctly separated. Further visualizations in Fig.6 reveal that the consistent features primarily focus on common objects (e.g., road fences), whereas the inconsistent features highlight objects visible in one modality but not in others (e.g., lane lines in RGB frames). In contrast, DeepMDP tends to indiscriminately focus on all scene objects, lacking contextual discriminativity. These visualizations demonstrate the efficacy of the modality decomposition in DDM, facilitating more precise dynamics modeling and efficient state representation.

DeepMDP

Consistency

RGB frames

Event signals

Inconsistency

5 Limitations and Future Works

Our current DDM method has two primary limitations. First, while it effectively simulates multimodal dynamics and enhances representation learning, it lacks integrated planning mechanisms similar to those in MuZero [36] and Dreamer [13, 15, 16, 34, 6]. Research has shown that planning mechanisms can effectively model the impact of various strategies in data-rich scenarios, thus minimizing trial and error in real-world applications. In future works, we plan to integrate long-term action consequences into our decision-making processes by developing decomposed multi-modal world models. Second, our approach focuses primarily on vision-based data inputs. However, recent studies suggest that integrating diverse modal inputs, such as textual and audio data, can yield adaptable representations [47]. Therefore, extending our approach to include additional types of modal inputs represents another promising future direction to enhance its scope and performance.

6 Conclusion

We have studied the impact of modality heterogeneity on dynamics modeling in multi-modal visual reinforcement learning (RL) tasks. Our investigation reveals that both the commonalities and differences in each modality are crucial for accurate dynamics modeling, albeit playing distinct roles. Based on this insight, we have presented Dissected Dynamics Modeling (DDM), a novel method designed to enhance dynamics modeling and representation learning for multi-modal visual RL. Our approach represents one of the first explorations of modeling environmental dynamics in multi-modal scenarios, providing a fresh context decomposition perspective based on modality relationships. Extensive experiments demonstrate the effectiveness of our approach, which significantly improves representation quality and decision-making performance in diverse environmental conditions.

7 Acknowledgment

The study was funded by the National Natural Science Foundation of China under contracts No. 62332002, No. 62425101, No. 62422602, No. 62372010, No. 62202010, No. 62402015, and the China Postdoctoral Science Foundation under grant 2024M750100.

References

David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning , pages 2915-2923. PMLR, 2016.

Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In Proceedings of the International Conference on Learning Representations , 2022.

Guang Chen, Hu Cao, Jorg Conradt, Huajin Tang, Florian Rohrbein, and Alois Knoll. Eventbased neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine , 37(4):34-49, 2020.

Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. In IEEE Proceedings of the International Conference on Robotics and Automation , page 4274-4280, 2021.

Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(11):12878-12895, 2023.

Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations. In Proceedings of the International Conference on Machine Learning , 2022.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning , volume 78 of Proceedings of Machine Learning Research , pages 1-16. PMLR, 13-15 Nov 2017.

Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI , volume 4, pages 162-169, 2004.

Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In Proceedings of the International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2170-2179, 09-15 Jun 2019.

David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems , 31, 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1861-1870, 10-15 Jul 2018.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 , 2019.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning , volume 97, pages 2555-2565, 09-15 Jun 2019.

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 , 2020.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023.

Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Proceedings of the Advances in Neural Information Processing Systems , volume 34, pages 3680-3693, 2021.

Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, and Yikang Li. Multi-modal sensor fusion for auto driving perception: A survey. arXiv preprint arXiv:2202.02703 , 2022.

Sili Huang, Yanchao Sun, Jifeng Hu, Siyuan Guo, Hechang Chen, Yi Chang, Lichao Sun, and Bo Yang. Learning generalizable agents via saliency-guided features decorrelation. Advances in Neural Information Processing Systems , 36, 2024.

Yangru Huang, Peixi Peng, Yifan Zhao, Haoran Xu, Mengyue Geng, and Yonghong Tian. Hierarchical adaptive value estimation for multi-modal visual reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems , 2023.

Xiaodong Jia, Xiao-Yuan Jing, Xiaoke Zhu, Songcan Chen, Bo Du, Ziyun Cai, Zhenyu He, and Dong Yue. Semi-supervised multi-view deep discriminant representation learning. IEEE transactions on pattern analysis and machine intelligence , 43(7):2496-2509, 2020.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Proceedings of the International Conference on Machine Learning , volume 202, pages 14975-15022, 2023.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5639-5650, 13-18 Jul 2020.

Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Proceedings of the Advances in Neural Information Processing Systems , volume 33, pages 19884-19895, 2020.

Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning , pages 5757-5766. PMLR, 2020.

Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128 × 128 120 db 15 µ s latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits , 43(2):566-576, 2008.

Kaiyi Lin, Xing Xu, Lianli Gao, Zheng Wang, and Heng Tao Shen. Learning cross-aligned latent embeddings for zero-shot cross-modal retrieval. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11515-11522, 2020.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Proceedings of the Advances in Neural Information Processing Systems , volume 34, pages 18459-18473, 2021.

Siao Liu, Zhaoyu Chen, Yang Liu, Yuzheng Wang, Dingkang Yang, Zhile Zhao, Ziqing Zhou, Xie Yi, Wei Li, Wenqiang Zhang, et al. Improving generalization in visual reinforcement learning via conflict-aware gradient agreement augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 23436-23446, 2023.

Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning. arXiv preprint arXiv:2210.04561 , 2022.

Guozheng Ma, Linrui Zhang, Haoyu Wang, Lu Li, Zilin Wang, Zhen Wang, Li Shen, Xueqian Wang, and Dacheng Tao. Learning better with less: Effective augmentation for sample-efficient visual reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Jinming Ma, Feng Wu, Yingfeng Chen, Xianpeng Ji, and Yu Ding. Effective multimodal reinforcement learning with modality alignment and importance enhancement. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems , page 1684-1686, 2023.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In Proceedings of the Advances in Neural Information Processing Systems , 2022.

Rudra PK Poudel, Harit Pandya, Stephan Liwicki, and Roberto Cipolla. Recore: Regularized contrastive representation learning of world model. arXiv preprint arXiv:2312.09056 , 2023.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In Proceedings of the International Conference on Learning Representations , 2021.

Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of the Conference on Robot Learning , volume 205 of Proceedings of Machine Learning Research , pages 1332-1344. PMLR, 14-18 Dec 2023.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the International Conference on Machine Learning , volume 139, pages 9870-9879. PMLR, 18-24 Jul 2021.

Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention. In Proceedings of European Conference on Computer Vision , volume 13678, pages 412-428, 2022.

Ruixiang Sun, Hongyu Zang, Xin Li, and Riashat Islam. Learning latent dynamic robust representations for world models. arXiv preprint arXiv:2405.06263 , 2024.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Tasbolat Taunyazov, Luar Shui Song, Eugene Lim, Hian Hian See, David Lee, Benjamin CK Tee, and Harold Soh. Extended tactile perception: Vibration sensing through tools and grasped objects. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 1755-1762. IEEE, 2021.

Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251 , 2015.

Shuo Wang, Zhihao Wu, Xiaobo Hu, Jinwen Wang, Youfang Lin, and Kai Lv. What effects the generalization in visual reinforcement learning: policy consistency with truncated return prediction. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 5590-5598, 2024.

Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research , 20(4):447-482, 2023.

Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems , 28, 2015.

Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In Proceedings of the International Conference on Learning Representations , 2020.

Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems , volume 35, pages 13022-13037, 2022.

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In Proceedings of the International Conference on Learning Representations , 2020.

Yue Zhao, Chenzhuang Du, Hang Zhao, and Tiejun Li. Intrinsically motivated self-supervised learning in reinforcement learning. In Proceedings of the International Conference on Robotics and Automation , page 3605-3615. IEEE, 2022.

Appendix

The contents of this Appendix are presented as follows:

Section A offers extra experiment findings, including performance on the CARLA benchmark with more modalities, experiments for visual inputs from multiple camera positions, sample efficiency and different masking ratios on the DeepMind Control Suite.

Section B describes the experimental setups, including the details of network structures, environment conditions, training hyper-parameters, and hardware details respectively.

Section C discusses the potential societal impacts of our method.

A Additional Experimental Results

Results with More Modalities on CARLA. Table A1 presents the results of various methods on the CARLA benchmark, utilizing four modalities: RGB frames, event signals, depth images, and LiDAR BEV images. Compared to the results in Table 1, nearly all methods show improved performance with the increased number of modalities. Additionally, the trends observed in Table A1 follow those in Table 1, with single-modal methods typically underperforming their multi-modal counterparts. Among all the multi-modal methods, our DDM method consistently maintains the best performance due to its modality decomposition and integration. This comprehensive modeling allows DDMto effectively utilize additional information provided by multiple modalities, leading to robust decision-making compared to other methods.

Experiments for visual inputs from multiple camera positions. We have further verified the ability of our model on multiple camera positions. First, we switch the camera view of RGB modality in DMControl. Second, we test on CARLA with RGB and LiDAR BEV as input modalities. LiDAR BEV is a bird view map, whose perspective is very different from RGB. For simplicity, we only compare our method against the most competitive baseline methods on these two environments ( i.e. , HAVE on CARLA and SPR on DMControl). The results and illustrations of different camera views are presented in Table A2, Table A3 and Fig. A1 respectively. These results show that our method also works on multiple camera positions.

Sample efficiency on DMControl. In Fig. A2, we evaluate the sample efficiency of different methods on DMControl. Compared to DeepMDP and EFNet, our method demonstrates the highest sample

(a) RGB and Depth modalities with different camera perspectives.

(b) RGB and Lidar-BEV modalities with different perspectives on Carla.

efficiency across five tasks, as shown in the figure. This superior performance is attributed to our method's ability to capture a more comprehensive representation of the environment. By accurately modeling the dynamics of each modality and their interactions, our method can make more informed decisions with fewer samples, enhancing learning efficiency and overall performance.

750

861

472

467

43

108

824

900

555

769

125

183

0

100

200

300

400

500

600

700

800

900

1000

Cartpole, swingup

Reacher, easy

Cheetah, run

Cartpole,

swingup_sparse

Reacher, hard

Quadruped, run

Episode Reward

762

769

523

486

136

413

854

904

587

813

224

489

761

596

226

182

77

182

827

823

524

713

128

453

DeepMDP

EFNet

DDM

100K steps

500K steps

Different masking ratios on DMControl . Our masking operation is performed independently at each timestamp. Therefore, both the masked modality type and the masking locations vary randomly across different timestamps, which simulates a challenging occlusion scenario. For other masking ratios, we further test the ratio of 0%, 40%, 60%, and compare our method with SPR, the most competitive baseline on DMControl. As shown in Table A4, the results reveal that as the masking ratio increases, both SPR and our DDM experience performance drops. However, DDM still outperforms SPR at different ratios.

Normal

Midnight

Dazzling

Rainy

B Detailed Experimental Setup

In this section, we provide a comprehensive description of the experimental setup, including network architectures, environment configurations, and hyper-parameter settings.

B.1 Network Architecture

Modality Encoders. Individual feature encoders are employed for each modality to handle the input data. Each input observation provided to the encoder consists of 3 consecutive frames returned by the environment. This results in input channel numbers of 9 for RGB frames, 15 for event signals, 3 for LiDAR BEV images, and 3 for depth images. All the feature extractors share a common network architecture that comprises four convolutional layers, each followed by ReLU activations. The final layer produces 64 × 6 × 6 feature maps, which serves as the individual modality features z M i t . These features are then passed into two separate network branches to get 50-dimensional modality-consistent and inconsistent features z M i t and ˆ z M i t , respectively.

Actor and Critic Networks. In our method, the actor network, which is composed of three fully connected (FC) layers with ReLU activations placed after the first two layers, produces parameters for a Gaussian distribution over actions. The critic network has a similar structure but outputs an estimated Q-value for a specific state-action pair.

Prediction Heads and Reward Prediction Network. The prediction heads, as described in [9], take the input latent vector and the corresponding action as inputs. They generate the inferred latent vectors through two fully connected (FC) layers, with ReLU activation following the first layer. Similarly, the reward prediction network uses the same architecture but outputs a single-dimensional predicted reward. This network estimates the expected reward for a specific state-action pair, offering critical feedback for dynamics modeling.

B.2 Environment Details

B.2.1 Environmental Conditions in CARLA

The four environmental conditions (Normal, Midnight, Dazzling, and Rainy) employed in our study are predefined with specific weather hyper-parameters, as detailed in Table A5. These conditions provide a comprehensive evaluation of the RL algorithms under various situations.

B.2.2 Modality Settings

Modalities in CARLA. RGB frames are the default input modality in CARLA, which capture rich texture and color information but may face challenges like motion blur and limited dynamic range under extreme light conditions. Event signal is a new modality provided in CARLA, mimicking the output of neuromorphic event cameras [26, 3]. As shown in Fig. A3, event signals focus on changes in brightness, offering a wider dynamic range, no motion blur, and high temporal resolution, making them ideal for dynamic scenes. However, they do not generate signals at static regions when no pixel differences occur. In addition, event signals may experience hot spot noises. We simulate this noise to replicate real event signal characteristics. Some key parameters related to these two modalities are provided in Table A6. Besides RGB frames and event signals, the depth images in CARLA provide distance maps, while LiDAR BEV creates scanning of the vehicle surroundings. Both depth and LiDAR adopt the default simulation parameters in CARLA.

Modalities in DMControl. In DMControl, the RGB frame modality captures direct visual information from the environment, providing detailed texture and color data essential for recognizing objects and their spatial relationships. Complementing this, the depth modality generates a depth map where each pixel value represents the distance to the corresponding point in the scene, offering crucial 3D spatial information. The settings of these two modalities are based on the default parameters.

B.2.3 Reward Setting of CARLA

In the CARLA autonomous driving task, the objective of the agent is to travel as far as it can on a highway without colliding in various weather conditions. Therefore, the reward function is designed similarly to [52] and [20]:

The first term is devised to motivate the vehicle to cover as much distance as possible along the highway. v ⊤ ego indicates the velocity vector of the agent vehicle, ˆ u highway represents the unit direction vector of the highway, and ∆ t = 0 . 1 stands for the discretized simulation time. The final three terms are incorporated to guarantee that the vehicle evades collisions, reduces excessive steering, and avoids sudden braking, where λ c , λ s , and λ b are assigned values of 0 . 001 , 0 . 1 , and 0 . 1 respectively.

B.3 The hyper-parameters of RL training

The hyper-parameters of RL training are elaborated in Tables A7 and A8, and they are same to those in [17] and [20].

B.3.1 Hardware Details

Computing Resources. All models are trained using a server that is equipped with 4 NVIDIA GeForce RTX 3090 GPUs and a 64-core AMD EPYC 7H12 2.6GHz CPU Processor.

C Potential Negative Societal Impacts

While DDM offers significant advancements in reinforcement learning by effectively integrating multimodal data, it is essential to consider and address some potential negative societal impacts. DDM models could be susceptible to adversarial attacks, where malicious inputs are designed to deceive the system into making incorrect decisions. Such attacks could lead to failures in critical applications, such as autonomous vehicles or healthcare systems, potentially causing harm to individuals.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Limitations and Future Works section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please refer to the code link in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and the research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve Institutional Review Board (IRB) approvals or equivalent for research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a method, namely Dissected Dynamics Modeling (DDM), for multi-modal environment dynamics modeling in visual RL. The core idea is to adopt additional modules to extract separate modality-consistent and modality-inconsistent features in each modality stream with designated losses as the regularization. During training, the model tries to maximize mutual information between modality-consistent features from different modalities at the current and the next timestamp. For modality-inconsistent features from, the model tries to enforce orthogonality. Experiments on CARLA and DMControl indicated the superiority of DDM over other state-of-the-arts and further analyses suggested the effectiveness of DDM v.s. DeepMDP. Ablation studies supported some of the important design choices.\n\nSTRENGTHS:\n- The general idea of attempting to decouple modality-consistent and modality-inconsistent features to improve environment modeling makes sense, which could provide some insights to future work to this particular field\n\n- Experiment results show good improvements over some state-of-the-arts, suggesting DDM's effectiveness for environment modeling in visual RL\n\n- Good ablations, analyses and visualizations provided a comprehensive understanding of the method to some extent justifying DDM's soundness and supporting its superiority for environment modeling in visual RL\n\nWEAKNESSES:\n- The proposed method is only showed to be working on a subset of modalities that are visual only, which greatly limits its generalizability and contribution as a method for multi-modal learning. However, visual RL is not my expertise and thus I cannot evaluate the contribution on this particular field.\n\n- Following above, from the experiments it seems that the method also only works on visual inputs from the same camera perspective. i wonder if there are experiments demonstrating the model also works for visual inputs from multiple camera positions.\n\n- For experiments on DMControl, how is the masking done for frames at different timestamps? Also, besides 20% masking, do the authors have results with other masking ratio?\n\n- Some places are not written in a technically accurate way, though it's hard to tell whether it is because of technical misunderstanding or simply misuse of words. For example:\n  - Line 46 - 48, \"Firstly, modality-correlated features provide a foundational perspective by capturing shared and complemented information across different sensory inputs.\" The modality-consistent features that are shared among modalities does not COMPLEMENT each other. Instead, those modality-inconsistent features that are unique for each modality themselves complement other modalities.\n  - Line 50-51, \"these inconsistencies are typically deemed less critical and are filtered out through modality alignment\". Modality alignment should not have the effect of filtering out modality-inconsistent features, in my understanding. If the authors have some evidence, I'd like to see.\n\nQUESTIONS:\nPlease refer to the above Weaknesses part and address my concern accordingly\n\nLIMITATIONS:\nYes, limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a solution for better multimodal dynamic modeling in visual RL. The paper claims that existing works only emphasize consistent (aligned) information across modalities, leaving out the opportunity for the model to benefit from the inconsistent features. The work introduces a new consistency loss, where the (cross-modal) mutual prediction happens dynamically, meaning a modality has to infer the other modality's features for the next step rather than the current one. Furthermore, the work introduces a \"soft objective\" for cross-modal feature orthogonalization. The work shows consistent improvement over the existing state-of-the-art on CARLA and DMControl benchmarks.\n\nSTRENGTHS:\n1) The paper is well-motivated, well-written, and easy to follow. The state-of-the-art results provide stats with several runs. The design of each loss component is straightforward.\n2) The paper achieves significant improvement over the reported state-of-the-art results. \n3) The cross-modal transition prediction loss is interesting and results in the most significant performance boost, as shown in Table 3. Overall, each loss component is shown to benefit the model's performance.\n4) The method is generalizable beyond two modalities.\n\nWEAKNESSES:\n1) The authors report the episode to return and driving distance but do not report DS/RC/IP, which is reported in other methods mentioned in the papers and could be more informative for evaluating driving performance. Is there any reason why the authors do not report those metrics? \n\n2) L265-267 states that   $ \\mathcal{L_{fp}} $ and $ \\mathcal{L}_{r} $ do not benefit much to the existing DeepMDPandSPR. Are there any supporting references/results for this claim?\n\nQUESTIONS:\n1) Do the authors think that incorporating a component of their modeling, e.g., $ \\mathcal{L_{tp}} $, into existing models could result in a boost? \n\n2) In Section 3.2 [L166], the authors suggest replacing a stronger objective (Eq. 6) with Eq. 7. Did you observe in your experiments that, indeed, a model trained with Eq.7 is better than Eq.6?\n\nLIMITATIONS:\nThe authors discussed potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Dissected Dynamics Modeling (DDM), a dynamics modeling framework for learning latent features in multi-modal visual RL. The methodology focuses on capturing both the shared and distinct information contained across input modalities. The paper presents a multi-modal architecture and training loss designed to accomplish this, and experimentally demonstrates the benefits of DDM on tasks from CARLA and the DeepMind Control Suite.\n\nSTRENGTHS:\n**[S1] Novel approach to multi-modal RL:** The paper proposes a novel methodology for multi-modal visual RL that accounts for both the similarities and differences across input modalities, compared to prior works that largely focus only on the similarities across input modalities. Experiments demonstrate the benefits of this novel approach.\n\n**[S2] Detailed experimental analysis:** The paper provides detailed experimental analysis of the proposed DDM method. This includes comparisons across several types of baselines (although some choices of baselines are not state-of-the-art), ablation studies that justify design choices, robustness analysis, and visualizations for qualitative understanding.\n\n**[S3] Important problem:** The paper focuses on how to effectively leverage information from multiple input modalities for decision making. This addresses an important consideration for the successful deployment of RL in real-world systems where multi-modal data is common.\n\nWEAKNESSES:\n**[W1] Clarity of implementation / experimental details:** The high-level idea of the DDM framework is clear, but some of the implementation details are not clearly described. Some components of the experiments were also not described in detail, which made it difficult to interpret some of the results. Please see the questions below.\n\n**[W2] Connections to related work:** DDM appears to build upon previous works in visual / multi-modal RL for its dynamics modeling and consistency extraction components, but these connections are not made clear. I think it would be useful to better emphasize what parts of DDM build upon existing works vs. what parts are novel contributions. (i) The dynamics modeling approach looks similar to reconstruction-free approaches in visual RL [a,b,c,d], but the similarities / differences are not discussed. (ii) It is mentioned that existing multi-modal approaches have focused on aligning modalities, but the paper does not discuss how the proposed consistency extraction method relates to these approaches. \n\n**[W3] Heuristic design choices:** Design choices of the proposed architecture are justified through experimental results, but no theoretical support is provided (or connections to existing approaches that provide theoretical support).\n\n---\n\nReferences:\n\n[a] Gelada et al. (2019). DeepMDP: Learning continuous latent space models for representation learning.\n\n[b] Schwarzer et al. (2021). Data-efficient reinforcement learning with self-predictive representations.\n\n[c] Okada et al. (2021). Dreaming: Model-based reinforcement learning by latent imagination without reconstruction.\n\n[d] Zhang et al. (2021). Learning invariant representations for reinforcement learning without reconstruction.\n\nQUESTIONS:\n**Methodology:**\n\n**[Q1]** Are gradients with respect to the latent feature encodings $z$ stopped in any components of the loss function in (13), or are they propagated through the loss function everywhere that $z$ appears?\n\n**[Q2]** Are the next-step target values $z_{t+1}$ in (5) and (11) calculated using the same feature encoder used for $z_t$? It would be helpful to make this more clear. Are gradients taken through $z_{t+1}$ in (5) and (11)?\n\n---\n\n**Experiments:**\n\n**[Q3]** What do the single modality baseline results represent? Do they only use RGB images, or do they consider a combined input that incorporates information from all modalities? If these baselines are not restricted to RGB images, it would be useful to include results (baselines or ablation of DDM) using only RGB inputs to demonstrate that the use of multi-modal inputs leads to improved performance over standard visual RL.\n\n**[Q4]** DreamerV3 [e] and DrQ-v2 [f] are strong model-based and model-free visual RL algorithms, respectively. Why were DeepMDP and SPR chosen as baselines instead of DreamerV3 (which has reported better head-to-head performance vs. SPR in Atari 100k)? Why was DrQ chosen as a baseline instead of DrQ-v2 (which has reported significant improvements over DrQ on DeepMind Control Suite)? \n\n**[Q5]** Has performance converged by the end of training for the experimental results, given the number of training steps used (100k and 500k in CARLA and DeepMind Control Suite, respectively)? In prior works on visual RL, DeepMind Control Suite tasks are typically trained for longer than this (1M steps in DreamerV3 [e], 3M steps in DrQ-v2 [f]), and these baselines had not converged to final performance after 500k steps.\n\n---\n\n**Minor:**\n\nSpelling: Inon. --> Incon. (p. 2, Figure 1 caption)\n\n---\n\nReferences:\n\n[e] Hafner et al. (2023). Mastering diverse domains through world models.\n\n[f] Yarats et al. (2021). Mastering visual continuous control: Improved data-augmented reinforcement learning.\n\nLIMITATIONS:\nThe authors have addressed the limitations of the current work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}