Learning Fused State Representations for Control from Multi-View Observations
Authors: Zeyu Wang
Yao-Hui Li
Xin Li
Hongyu Zang
Romain Laroche
Riashat Islam
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
multi-view learning
Primary area: reinforcement_learning
TL;DR: We proposed a method to learn fused state representations for multi-view RL.

Abstract: Multi-View Reinforcement Learning (MVRL) seeks to provide agents with multi-view observations, enabling them to perceive environment with greater effectiveness and precision. Recent advancements in MVRL focus on extracting latent representations from multiview observations and leveraging them in control tasks. However, it is not straightforward to learn compact and task-relevant representations, particularly in the presence of redundancy, distracting information, or missing views. In this paper, we propose **M**ulti-view **F**usion **S**tate for **C**ontrol (**MFSC**), firstly incorporating bisimulation metric learning into MVRL to learn task-relevant representations. Furthermore, we propose a multiview-based mask and latent reconstruction auxiliary task that exploits shared information across views and improves MFSC’s robustness in missing views by introducing a mask token. Extensive experimental results demonstrate that our method outperforms existing approaches in MVRL tasks. Even in more realistic scenarios with interference or missing views, MFSC consistently maintains high performance. The project code is available at [https://github.com/zpwdev/MFSC](https://github.com/zpwdev/MFSC).

Learning Fused State Representations for Control from Multi-View Observations
Zeyu Wang" "1 Yao-Hui Li Xin Li 12
Hongyu Zang²
Romain Laroche Riashat Islam"
Abstract
necked by high data collection costs and increased computa- tional complexity. MVRL has recently emerged as promis- ing alternative, leveraging 2D observations from multiple perspective cameras to enhance spatial perception while reducing computational demands (Li et al., 2019). Despite these advantages, MVRL still struggles with two common challenges: i) the difficulty in learning and fusing com- pact representation from redundant and high-dimensional multi-view observations, and ii) the problem of dealing with missing or interfering views in real-world scenarios.
View Reinforcement Learning (MVRL) seeks to provide agents with multi-view obser- vations, enabling them to perceive environment with greater effectiveness and precision. Recent advancements in MVRL focus on extracting la- tent representations from multiview observations and leveraging them in control tasks. However, it is not straightforward to learn compact and task- relevant representations, particularly in the pres- ence of redundancy, distracting information, or missing views. In this paper, we propose Multi- view Fusion State for Control (MFSC), firstly incorporating bisimulation metric learning into MVRL to learn task-relevant representations. Fur- thermore, we propose
Chen et al. (2021) averages keypoint estimations from dif- ferent camera views to obtain a single world coordinate pre- diction. Hwang et al. (2023) learns distinct representations for each view using a VAF-based architecture and derives the final fused representation for MVRL via a weighted fu- sion mechanism. Jangir et al. (2022) introduces the use of cross-view attention mechanism, offering more flexible ap- proach for representation fusion. However, these multi-view representation learning methods lack explicit mechanisms to focus on task-relevant features, often incorporating irrele- vant information into the final representation. Recent works bisimulation metric learning (Castro al., 2021; Zhang et al., 2021; Zang et al., 2022) capture task-relevant rep- resentations by measuring behavioral similarities between states based on their rewards and transition distributions, and appear to be promising solution to mitigate this issue.
salttiview-bbseed mask and latent reconstruction auxiliary task that ex- ploits shared information across views and im- proves MFSC's robustness missing views by introducing a mask token. Extensive experimen- tal results demonstrate that our method outper- forms existing approaches in MVRL tasks. Even in more realistic scenarios with interference missing views, MFSC consistently maintains high performance. The project code is available at https:// //ttt//// com/ 2pwdev/MEBS /MFSC.
In this work, we integrate the self-attention mechanism with bisimulation metric, offering powerful approach to multi- view representation learning and fusion. Inspired by BERT (Devlin, 2018) and VIT (Dosovitskiy, 2020), learnable fu- sion [state] embedding for MVRL tasks is introduced to avoid biases from fixed views, and dynamically capture the optimal fusion of multi-view information. By aligning the fusion of multi-viww representations based on the similarity of task-relevant information, we enable robust multi-view integration and efficient control. However, bisimulation metric learning heavily depends representations, which may cause it to overlook fine-grained information or interactions unique to each view not directly reflected in the reward signal. Secondly, bisimulation as- sumes complete and high-quality observations, making it vulnerable to scenarios where observations are incomplete, as it lacks mechanisms compensate for missing views.
1. Introduction
In robotic manipulation tasks, acquiring accurate 3D scene information, including understanding target position, orien- tation, occlusions, and stacking relationships among objects in complex environments, is crucial for effective grasping and interaction with objects (Ze et al., 2023; Akinola et al., 2020). However, directly using 3D inputs is often bottle-
"quual contribution, order is determined by random drawing. Beijing Institute of Technology, Beijing, China Key Labora- tory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China Meituan, Beijing, China Waayee, London, UK Mila- Quebec Institute, Montreal, Canada. 'Correspondence to: Xin Li <xinli @ bit.edu.en>>
rewards fuse state
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Learning Fused State Representations for Control from Multi-Vieyy Observations
Table I: Comparison of different MVRL algorithms.
Method
Masking
Applying Demos Reconstruction
Framework
Multi-view Methods
Keypoint3D (Chen et al., 2021)
Pixel
Incoder-Decodee
3D keypoints Cross-attention
LookCloser (Jangir et al., 2022)
X
X
X
Encoder only
F2C (Hwang et al., 2023)
X
X
X
Encoder only
CVIBs
MV-MWM (Seo al., 2023b)
Feature+ View
V
Pixel
Self-attention
MVD (Dunion & Albrecht, 2024)
X Pixel
X
Encoder only
Contrastive learning
MFSC (ours)
x
Latent
Encoder only
To overcome these limitations, we propose an enhanced approach that integrates multiview-based masking and la- tent reconstruction along with bisimulation metric learning. The disimulatioondrriven reconstruction helps ensure that the learned state representations are not only aligned with the task's global objectives but also retain crucial details that are specific to each individual view. Furthermore, the reconstruction process encourages representation learning to leverage cross- view dependencies to recover the masked information, rather than simply depending on the reward signal alone for state representation learning. Through the auxiliary task of multiview-based masking and latent recon- struction, we naturally introduce a learnable mask token to replace missing views, bolstering the model's capability to handle missing views. The key contributions of our work are summarized below:
(2018) categorized existing methods into two approaches: representation alignment and representation fusion. Repre- sentation alignment aims capture the relationships among multiple different views through feature alignment, such as minimizing the distance between representations of differ- ent views (Feng et al., 2014; Li et al., 2003), maximizing similarity between views (Bachman et al., 2019; Frome et al., 2013), and maximizing the correlation of variables across views (Andrew et al., 2013). Representation fusion integrates representations from different views to form a compact representation for downstream tasks (Geng et al., 2022; Xie et al., 2020; Karpathy & Fei-Fei, 2015). Both strategies aim to harness the complementary information provided by multiple views for more comprehensive data representation. In this work, we propose a multi-viee repre- sentation fusion method designed for control. Our approach leverages the properties of sequential decision-making tasks and multi-view data achieve task-relevant and robust rep- resentation fusion.
We propose a novel MVRL algorithm MFSC, which is the first to incorporate bisimulation into the MVRL framework. This integration enables the extraction of task-relevant representations from redundant multi- view observations. Our method can be combined with any existing RL algorithms with minimal adjustments.
Multi- Reinforcement Learning. Effective state representation learning in MVRL aims to transform high- dimensional observations into a compact latent space. Li et al. (2019) propose a VAE-based algorithm that minimizes Euclidean distance between states encoded from different views, assuming consistent primary view. Keypoint3I) (Chen et al., 2021) uses 3D reconstruction to learn key- points from third-person views, requiring camera calibra- tion. Lookcloser (Jangir et al., 2022) introduces a cross-view attention mechanism for aggregating egocentric and third- person representations without calibration, though at higher computational cost. F2C (Hwang et al.. 2023) uses condi- tional variational information bottlenecks (CVIBs) model state space, showing robustness to missing views. MV- MWM (Seo et al., 2023b), based on Dreamer-v2 (Hafner et al., 2021), applies dual masking and pixel-level recon- struction tasks during pretraining. MVD (Dunion & AI- brecht, 2024) employs contrastive learning to disentangle multi-view representations for control tasks. MOSER (Wan et al., 2024) is a model-based approach that actively seeks the optimal perspective for learning task representations un- der multiple views to enhance performance. For detailed comparison of MFSC with other closely related algorithms, refer to Table I and Appendix B.2.
We propose multiview-based masking and latent re- construction auxiliary task that significantly enhances multi-view representation fusion. By introducing a learnable mask token, our method improves robustness against missing views.
Extensive experiments on motion and robotic manipu- lation tasks demonstrate that our method outperforms existing MVRL algorithms, consistently achieving su- perior performance even in challenging scenarios with missing or noisy views.
2. Related Work
Multi-View Representation Learning. Multi-view rep- resentation learning uses multiple sources (or views) of a shared context. These views may include multiple camera views of the same scene (as in our work), as well as multi- modal inputs (c.g., audio and video) or synthetic views of the unimoaal measurements (e.g., sime-sequenced data). Li et al.
2

---

Learning Fused State Representations for Control from Multi-Viewy Observations
3. Preliminaries
g. c controls the trade-off between prioritizing immediate reward differences and accounting for long-term future state dynamics.
This section establishes the foundational concepts under- lying our method, i.e. the Multi- View Markov Decision Process (MV-MDP) and the bisimulation metric.
Theorem 3.2. Castro (2020) The operator F has a unique fixed point 9 and 9 is a π TT-isimulatiio metric.
3.1. Multi- View Markov Decision Processes
4. Theoretical Analysis
Multi- View Markov Decision Process is defined as the following tuple M A, R), S represents the set of states in the environment, A is set of actions == {Ok
MVRL. introduces the challenge of dealing with observa- tions that may contain redundant or irrelevant information across different views. To address this, we employ bisim- ulation metrics to capture task-relevant information from multi-view observations. This section formalizes theo- retical framework, extending the bisimulation metric from single-view RL to multi-view setting and demonstrating its effectiveness in simplifying MVRL tasks.
j J==1 represents the set of K observations [o T==1" With the assumption that each multi-view observation Đe Ô uniquely determines its generating state S. we can obtain the latent state regarding its multi- view observation by projection function Φ (Ø) Ö S. Therefore, 8 and Φ (ō) can be used interchangeably. P(s'(s, a) = Pr(st+1 = s = s, a, = a) is the transi- tion dynamics distribution. The corresponding transition function under the multi-view observation space is defined d P(o o a). where (ō a)
4.1. Bisimulation Metric in Multi- View Setting
Formally, we define the bisimulation metric for policy on a multi-view setting as:
(d'1s') P (s'ls,a) a) and Q (o]s) = II=m Pr (r = |=|s = 5) is the probabil- ity distribution of joint observation. R(s, a) E is the immediate reward function for taking action at state 8, [0,1) is the discount factor. The goal of the agent is to find the optimal policy x(als) to maximize the expected 1 ar)(
(1) -cc) -rū FF 0,) =(1 -(1-
+ c. Ne-B,,n Boy 0f or,,i, (G(d, T,},
(I)
where o,, E O² 02
Ok and the operator FF has unique fixed point GT. Learning the exact bisimula- tion metric is computationally challenging. In this paper, we employ a "base metric" by setting GTT as a simple cosine distance and guide the representation learning, a learnable aggregator o, to align with the behavioral pattern demon- strated in (Q) Consequently, (q. rewritten as:
reward: Eo.......
3.2. Bisimulation
Bisimulation is used to formalize the concept of state sim- ilarity. The core principle is that two states are deemed equivalent if their future behaviors, i.e. state transitions and rewards given the same actions, are indistinguishable under the same policy. Conventional bisimulation met- ric needs to compute the Wasserstein distance over the transition distributions across all actions, which is com- putationally expensive. Instead, Castro (2020) developed π r-bisimulation metric which removes the requirement of considering all actions and only needs to consider the ac- tions induced by a policy æ. Castro et al. (2021) introduced the MICo update operator as a mathematical tool for con- structing a r-bisimulation metric. This metric evaluates state similarity by focusing exclusively on the actions dic- tated by specific policy π, rather than considering the entire action space.
FT (ō;) (ō,)) (1-c) c) 1 (ra(a) ا + c. E C. B. o. o (G (o,)) (d,)). (2)
The aggregator ø maps multi-view observations into a more compact space of summarizations Z, defined as: ó O x O ... x O* + Z E Rdd. By clustering observations to be similar under the bisimulation metric, the original MVRL problem can be approximated solving latent MDP .M = (Z, A, 7.po).
4.2. Latent MDP Construction Assumptions
We establish the following assumption to support the con- struction of the latent MDP:
Definition 3.1. (r-bisimulation metric (Castro, 2020)). Let M be the set of all measures over S. The operator F= M + M is defined as:
Assumption 4.1. There exists a latent space Z with dimen- sion lower than that of O x O x .... O² and a bounded approximation error 1 > 0, such that the induced latent MDP M (2, A, P. Ω R, y, Po) satisfies:
F" (g)(s) 84) (1 -cc..rr., - W(g) PT)),
where 8j E S. r = EnEa ((lls))r P EAEA (a.s and W(g) denotes the Wasserstein dis- tance between distributions, computed with cost function
Markovian Property: P(z'|z, S a,
for all ō = (01, 02. 0₂) E O x 02 ... x 0 such that ((ō) = = E 2, and same for R.
3

---

Learning Fused State Representations for Control from Multi- View Observations
Orginnal Multiviiw Observations
Masked Mult-viow Oissorvations
Self-Attention Fusion Module
Mork(()
Environment
O
OF
O+
O=
Multi-view Observations
CNN Encoder
Weight Sharing
CNN E Encoder
Norm
State Pusion Embeddiing =x
Position Embeskiing
0
O
0
D
Multi-Heed Allention
Multi-view
Disimulation Loss
Representation Learning
Self-Allention
Weightt Shertin
Self-Allention Fusion Module
Fusion Module
Reconstruction Loss
Norm
MLP
Fusion State
Fusion State
Prediction Head
Actor Network
Cosine Similarity
Critic Natwork
Figure I: Framework of MFSC. (a) The left part illustrates the process of MVRL, where the agent receives observations from multiple views, learns fused latent state, and interacts with the environment through an actor-critic framework. (b) The middle part provides a detailed overview of the MFSC architecture. Each view is encoded into latent embedding via a CNN, followed by state fusion using the mask-based self-supervised auxiliary task is employed to enhance the model's cross-view learning capabilities. (c) The right part presents the inner workings of the Self-Attention Fusion Module, which integrates embeddings from different views through attention mechanisms to produce a unified state representation.
Fusion Module. Metrie learning is guided by bisimulation loss, and
2. Value Function Approximation: For every ō Õ, there cxists 2 E Z such that (o)- S 17.
corresponding clusters in simplified MDP. By leveraging the learned aggregator, plexity of the multi-view MDP's state space while main- taining a predictable level of accuracy in value function estimation.
effectively reduce the com-
This assumption ensures that retains sufficient informa- tion for value estimation while reducing redundancy.
4.3. Bounding Value Function Differences
5. Learning Fused State Representations
To quantify the impact of the above transformation, we derive the following lemma:
In this section, we detail how the aggregator d is constructed to produce fused state representations. Fig.1 outlines the overall framework, which comprises two key components: (i) Self-Attention Fusion Module, which integrates bisimu- lation metrics to guide the aggregator in extracting task- relevant features from multi-view observations; and (ii) Multiview-basee Masking and Latent Resconstruction. which serves as auxiliary task to further promote cross- view state aggregation.
Lemma 4.2 (Value Function Bound), For a latent MDP M
constructed by an aggregator o O¹ x 0 x ....x x O - Z that clusters multi-view observations a e-neighborhood. The difference between the optimal value functions of the original and latent MDPs is bounded as:
2
[V) V (6(T)) < (1- (1-7)(1-c)
(3)
5.1. Self-Attention Fusion Module
where e [0. 1) is the trade-off parameter in the bisimula- tion metric, and 2 7.
In this section, we will introduce how bisimulation can be incorporated into multi-view representation learning. Specif- ically, our method comprises two core submodules: (i) Con- volutional Feature Embedding, generating embeddings
The proof can be found in the Appendix A. Lemma 4.2 serves to establish a bound on the difference between the optimal value functions of multi-view observations and their
4

---

Learning Fused State Representations for Control from Multi-View Observations
of the original high-dimensional multi-view observations; (ii) Self-Attention Fusion Module, integrating multi-view representations based on bisimulation metric.
mation to the original observation distance G*:
G (ō,, o5) = Go. (ōt, σ5) ;= 11- coS (ōs, (T,) ) (T)( 1 (o:)) (6,))
Convolutional Feature Embedding. The feature encod- ing module uses a Convolutional Neural Network (CNN) encoder to encode single-view image observations into fixed- dimensional embeddings. Given a multi- observations o- = {o o ..... o* ). where of #x the CNN encodes each image into a single-view representation x², where x² E Rd.
Therefore, the objective of state fusion with bisimulation metric is:
fus = E ( (Gow (0. Gz)
{
Go r r + + G
Self-Attention Fusion Module. Similar to the [class] token used in BERT (Devlin, 2018) and ViT (Dosovitskiy, 2020), we prepend learnable fusion [state] embedding r E Rd to the sequence of multi-view embedded repre- sentations. The state fusion representation z", trained via self-attention and the bisimulation metrie, serves as the final fused representation of multi-view observations and is used for downstream RL tasks. Additionally, akin to ViT, a train- able positional embedding ID tensor Epos is incorporated into the multi-view embeddings to encode view-specific information:
(8)
8 ~ P (. 1ow (o,) 80g ~ P (-10- (0))
P is latent state dynamies model and D is the replay buffer. For detailed explanations of the latent state dynamics model and the reward scaling mechanism, please refer to the Ap- pendices C.2 and C.3. By incorporating bisimulation met- rics during state aggregation, our method is able to focus on the causal features that directly influence rewards, effec- tively integrating information from multi-view observations.
5.2. Multiview-based Masking and Latent Reconstruction
20
[x"
1 Epos: Epos ([++))
(4)
Although bisimulation learning provides an effective ap- proach to aggregating task-relevant information across mul- tiple views, it overlooks the local details within each view, as it focuses global state transitions and task reward rather than the details of each individual view. To address these challenges, we introduce multiview -based masking and latent reconstruction auxiliary task, fostering the learning of cross-view dependencies. The masking and reconstruction mechanism strengthens bisimulation's 's capacity to capture richer multi-view information, enabling more robust fusion and control.
The embedded sequence is then fed into the Self-Attention Fusion Module. Specifically, the Self-Attention Fusion Module consists of I. attention layers. Each layer is com- posed of Multi-Headed Self-Attention (MHSA) layer, layer normalization (LN), and Multi-Layer Perceptron (MLP) blocks. The process can be described as follows:
a/ = MHSA(LN(ZE-D)) 2E-1, 1=1...I L
(5)
Ze MLP(LN(2/)) +44. f=1...I
(6)
Specifically, we randomly mask a portion of the origi- nal multi-view image observations o²......... o The masked observation sequences (ō) ,2.6... o+ are then pro- cessed through the CNN Encoder and the Self-Attention Fusion Module, resulting a set of masked state embed- dings {E), 2] i,,,, ..., =1 Motivated by the success of Sim- Siam (Chen & He, 2021) in self self-super learning, we an asymmetric architecture to calculate the distance between the reconstructed latent states and the target states. The state embeddings of the masked observations are passed through a prediction head to get the final reconstructed/prediitte states (#). #1,,............ ...; = We construct the reconstruction loss using cosine similarity, ensuring that the final predicted re- sult closely approximates its corresponding target. The final objective function of masking and latent reconstruction can be formulated as:
The output after L attention layers is ZL = f2f, 2/. ) ...,, where =f, represents the final state fusion embed- ding. To extract task-relevant representations from multi- view observations, bisimulation metric learning is integrated into the state fusion process. Specifically, the bisimulation metric for given policy π (Eq.2) employs the measure- ment G, as defined in SimSR (Zang et al., 2022). Here, G is based on cosine distance, which offers lower compu- tational complexity compared to the Wasserstein distance while effectively mitigating representation collapse.
In RL, the critic in actor-critic algorithms like SAC (Haarnoja et al., 2018), can be decomposed into two function approximators, za and o. with parameters 0 and w respec- tively: Qa = vho (ow (o)). Here, vin serves as the value function approximator, while d. is the state aggregator, with the goal of aligning the distances between representations to match the cosine distance. Therefore, the parameterized representation distance Go,, can be defined as an approxi-
k
1
Lree =
(9)
k
||
5

---

Learning Fused State Representations for Control from Multi- Observations
Close
X:22
Drawer
+10
Open
Pog Unplig
-100
Push W/ Vall
Ed0odd Rehurm
00
0.5
1.
24
0.2
3.5
1.1
2.
322
4.0
2.5
0.5
1.5
2.4
3.2
6.0
1.0
0.0
3.5
3.2
0.0
0.5
16
24
32
Ennirommet Steps
MFSC(ours)
Keypaint3D
LookCleser
Vanilla PPO
Figure 2: Performance comparison on five robotic arm manipulation tasks from Meta-World. All curves show the mean and its 95% Confidence Intervals (CIs) of performance across four independent seeds. The black dashed line represents the final convergence result of F2C in Meta-World.
The masking and latent reconstruction serves as an auxiliary task and is optimized together with the multi-view state fusion module. Thus, the overall loss function of MFSC is:
the agent's fine-grained understanding of movable joints and components. To further validate whether our algorithm can still capture task-relevant information, we also evaluate in more practical 'Ant No Color' task, where no explicit visual cues are provided.
Lmfse Luss + ACrec
(10)
represents the weighting coefficient for the two loss func- tions.
6. Experiment
Through our experiments, we aim to investigate the follow- ing questions: (I) How does the performance of MFSC compare to existing MVRL algorithms?
(2) Has MFSC successfully captured task-relevant representations? And, how does the Soff-Attention Fusion Module operate across different views? (3) To what extent can MFSC handle tasks with missing views? (4) How does the MFSC perform in the presence of noise interference? (5) What is the relative importance and contribution of each part of MFSC?
2.0 2201
20 X20
VPSCloubst
Web
14
222
0.4
0.4
6.1. Setup
Figure 3: Performance on PyBullet's Ant.
Experimental Setup. We evaluate our method on set of 3D manipulation environments Meta-World (Yu et al., 2020) and high degree of freedom 3D locomotion environment PyBullet's Ant (Coumans & Bai, 2022). These environ- ments originally designed for state-based RL, posing significant challenges for pixel-based RL. Furthermore, we also evaluate MFSC on the widely used single-view con- tinuous control benchmark DMControl (Tassa et al., 2018), comparing with recent visual RL baselines, as well as in the more realistic multi-view highway driving scenario, CARLA (Dosovitskiy et al., 2017). More details about the experiments can be found in the Appendix E.
Baselines, We compare MFSC with several baseline meth- ods. All baselines, including MFSC, are implemented with PPO (Schulman al., 2017). The baselines include: (I) Keypoint3ID (Chen et al., 2021), which learss 3D) keypoints from multiple third-person view cameras; (2) LookCloser (Jangir et al., 2022), which applies cross-attention between pairs of views to integrate multi-view information; (3) Fuse2Control (F2C) (Hwang et al., 2023), which employs an iiformaationthhoreett approach to learn a state space model and extract information independently from each view; (4) MVD (Dunion & Albrecht, 2024), contrastive learning- based algorithm for disentangling multi- rep- resentations. Additionally, for common RL algorithms, we stack images from all three views to form the observation: (5) RAD (Laskin et al., 2020a), which achieves high sample efficiency through data augmentation; (6) Vanilla PPO, the original PPO algorithm (Schulman al., 2017).
For fair comparison, we adopt the same Meta-World exper- imental setup as described in Chen et al. (2021). Details regarding the specific task settings of Meta- World can be found in the Appendix D. PyBullet's 's Ant is highly dynamic, with different colors assigned adjacent limbs enhance
6

---

Learning Fused State Representations for Control from Multi- View Observations
Frame
Frame
Frame
Frame 4
Frame
Frame
View View Views
Declusion
Viow View 2 View
X
Figure 4: Visualization of multi-view fusion for task-relevant representations. The first column of each frame represents snapshots from different views. The second column of each frame displays Grad-CAM heatmaps for each view, generated from the task loss and normalized via min-max scaling. In the third column of each frame, unify the gradient maps from all three views. We select the top 2.5% of gradient values across all views, highlight those corresponding pixels in white within each view, and show all other pixels in black.
6.2. Evaluation on Meta-World and PyBullet's Ant
6.3. Visualization and Analysis
Meta-World. As shown in Fig.2, our method consis- tently outperforms other MVRL algorithms across all five tasks, exhibiting significantly higher sample efficiency and more stable performance. particular, Vanilla PPO shows scarcely any indication of performance improvement, indi- cating the difficulty in extracting meaningful fused represen- tations without any special designs. While RAD integrates data augmentation and performs well in simpler tasks, it struggles to learn effective fused representations in more complex tasks, such as 'Open Window' and 'Close Box". Keypoint3D demonstrates competitive performance in cer- tain tasks, particularly in 'Close Box', but its overall training efficiency and final performance remain suboptimal. Look- Closer, which is also based on Transformer architecture, is also effective. However, its overall performance has not yet exceeded that of MFSC. Moreover, our method outperforms the previous SOTA algorithm, F2C, in terms of convergence performance across the majority of tasks.
To verify that MFSC is indeed capable of capturing task- relevant features, we utilize the Gradient-weighted Class Ac- tivation Mapping (Grad-Ca -AMM) technique (Selvaraju et al., 2017) to visualize the learned representations of MFSC. As shown in Fig.4 4 (middle column of each frame), MFSC consistently focuses on task-relevant features, such as the target position, robotic arm, or ant's 's legs, while minimizing focus on less relevant elements, like window edges or the ant's body. This analysis demonstrates MFSC's ability to effectively identify and extract task-relevant representations from each view.
In the 'Open Window' task (first row). during the initial stage where the goal is to position the robotic arm correctly, all three views provide critical information, and the model distributes its attention accordingly. However, when the goal shifts to opening the window, occlusion in View leads the model focus more on View and View 2, which contain more relevant details. In the 'Ant No Color' task, the importance of information from all three views is rel- atively balanced, shown by the uniform distribution of top-grade gradient pixels, indicating that MFSC allocates attention more evenly.
PyBullet's Ant. As shown in Fig.3, the performance of our method in the 'Ant' environment is comparable to Key- point3D. During training, our algorithm exhibits stable and consistent performance improvement. Even in the more practical 'Ant No Color' task, our method exhibits the high- est performance, demonstrating its ability to effectively cap- ture and fuse tesk-relevant information from multi-view ob- servations. In contrast, methods based on contrastive learn- ing and pixel-level reconstruction tend to overemphasize local variations, making it difficult to capture task-relevan information and leading to significant performance degrada- tion when key color cues are removed.
The visualization above demonstrates that MFSC can effec- tively extract and fuse task-relevant representations from multiple views. The fused representations enhance the per- formance of downstream RL tasks by offering a more robust and comprehensive understanding of the environment. By integrating information from diverse observational sources, MFSC enables more efficient policy learning and decision- making complex control scenarios.
7

---

Learning Fused State Representations for Control from Multi- Observations
30
Pog Unplug
Close Droiwer
Open Window
450
+1.
12
Ephode
Mocia Laaccoer ranig
0.0
00-
00
24
0.0
0.6
2-4
0.0
24
166
Figure 5: (a) Performance comparison of MFSC under full-view and missing-view. The black and gray lines represent the final convergence results of F2C. (b) Performance of ablation study.
6.4. Robustness Against Missing Views
tions from different views (RAD) and fused representations derived through attention mechanisms (LookCloser) prove susceptible to the disruptive effects of noise. In contrast, MFSC preserves its high performance despite the introduc- tion of irrelevant information.
To explore whether MFSC has the ability to counteract miss- ing views, we systematically evaluate the performance of MFSC under conditions of missing views across three tasks from Meta- World. During training, we explicitly introduced a mask token and leveraged the masking and latent recon- struction to learn shared representations across views. In the testing phase, we simulate real-world scenarios with incomplete view information by randomly omitting obser- vations from one view and replacing the missing view with the mask token. Fig.5(a) summarizes the performance com- parison between MFSC and two other competitive MVRL algorithms, LookCloser and F2C, under both missing view and full view conditions. The results indicate that, although missing views can affect certain task-speeific details, MFSC still exhibits decent performance in some tasks. Compared to LookCloser, the performance degradation of MFSC is relatively limited, and it even achieves performance com- parable to the previous SOTA algorithm for missing views, F2C. This robustness to missing views is largely attributed to the introduction of multiview-based masking and latent reconstruction, which leverages cross-view -iews dependencies to promote robust shared representation learning.
Push wi Wall
Clssa Drawer
Figure 7: Partial visualization under the noisy view.
From the visualization analysis in Fig.7, by incorporating bisimulation into the representation learning process, MFSC continues to capture task-relevant features, thereby signifi- cantly enhancing its robustness interference views. More details on visualization and ablation study are in the Ap- pendix E.3
6.6. Ablation Study
Fig.5(b) illustrates the returns of the algorithm across two benchmarks,
and PyBullet's Ant, comparing three variations: MFSC, MFSC without bisimulation con- straints ( ('MFSC w/o Bis'), MFSC without Masking and Latent Reconstruction ( ("MFSC w/o MLR'), and MFSC with- out Self-Attentin Fusion Module ('MFSC w/o SAFM'), which means an average of the embeddings for each view.
x1.0
Push Wall
x10
Close Drawer
Episode Return
1.8
3.2
12
MFSC (red line), the complete method, achieves the high- est returns throughout the process. In contrast, removing the bisimulation constraint in MFSC (orange line) not only sig- nificantly degrades performance but also increases variance, which highlights the importance of bisimulation MFSC. MMSSC w/o MLR' (blue line) performs better than 'MFSC w/o Bis' (orange line) but still falls short of the full MFSC method, emphasizing the significance of learning cross-view information. Finally, replacing the Self- Atte-Atttntion Fusion Module (green line) with feature averaging leads to a notice- able performance drop, demonstrating that the self-attention mechanism's 's dynamic integration of cross-vive view information is more effective than simple averaging.
620
LookCieder MPSChuust
RAD
LooiCieeer PSCCuus)
Figure 6: Performance noisy view.
6.5. Robustness Against Noisy Views
To evaluate the robustness of MFSC under interference views, we introduce an additional disruptive view, com- posed of images entirely irrelevant to the task. As depicted in 6, both RAD and LookCloser suffer notable perfor- mance decline when subjected to noise, particularly in the "Closer Drawer' task. Both simple concatenation of observa-
8

---

Learning Fused State Representations for Control from Multi-View Observations
7. Discussion
References
Limitations and Future Work. One limitation of our ap- proach lies in its reliance on masking techniques to handle missing views. For certain views containing crucial infor- mation, accurately reconstructing the true environmental state remains challenging. This is due to the incomplete complementarity of information between multiple views, particularly in scenarios involving complex state transitions. To address this limitation, future research could explore the integration of state-space models (Lee et al., 2020) to bet- ter capture temporal dependencies, enabling more accurate state estimation even in the absence of certain views.
Akinola, L., Varley, J., and Kalashnikov, D. Learning precise 3d manipulation from multiple uncalibrated cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4622, 2020. doi: 10. 10.1109/ ICRA40995.2020.99977
Andrew, G., Arora, R., Bilmes, J., and Livescu, K. Deep canonical correlation analysis. In International confer- ence on machine learning, pp. 1247-1255. PMLR, 2013.
Bachman, P., Hjelm, R. D., and Buchwalter, W. Learn- ing representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Conclusion. We propose novel framework, Multi-view Fusion State for Control (MFSC), to address the challenge of representation learning in MVRL. MFSC integrates the self-attention mechanism with bisimulation metric learn- ing, guiding the fusion of task-relevant representations from multi-view observations. Additionally, we introduce auxiliary task, nultiview-based masking and latent recon- struction, to capture cross-view information and enhance robustness to missing views. Experimental results demon- strate that MFSC effectively aggregates task-relevant details and shows robustness in scenarios with missing views. Fi- nally, visualization analyses confirm the capability of MFSC to capture relevntl information and dynamically fuse multi-view observations.
Castro, S. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10069-10076, 2020.
Castro, P. S., Kastner, T., Panangaden, P., and Rowland, M. Mico: Improved representations via sampling-based state similarity for markov decision processes. Advances in Neural Information Processing Systems, 34:301 13- 30126, 2021.
Chen, B., Abbeel, P., and Pathak, D. Unsupervised learning of visual 3d keypoints for control. In International Con- ference Machine Learning, pp. 1539-1549. PMLR, 2021.
Acknowledgements
The authors would like to thank anonymous reviewers for their constructive feedback, which has significantly con- tributed to enhancing the quality and clarity of this pa- per. This work was partially supported by the NSFC under Grants 92270125 and 62276024.
Chen, X. and He, K. Exploring simple siamese representa- tion learning. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, June 2021.
Impact Statement
Coumans, E. and Bai, Y. P. a python module for physics sim- ulation for games, robotics and machine learning. 2016- 2021, 2022.
The proposed MFSC framework advances MVRL by tack- ling key challenges in learning task-relevant representations from multi-view observations. By combining bisimulation- based representation learning with a masking-based latent reconstruction auxiliary task, MFSC improves robustness and adaptability in environments characterized by redundant, distracting, or incomplete inputs. These improvements are particularly beneficial for applications such as autonomous robotics, self-driving systems and other scenarios where reliable perception is essential.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Ethical considerations include potential biases from sen- sor placement, modality imbalance, or observation dropout, which may affect policy fairness and generalizability. Future work should explore bias mitigation strategies and improve interpretability, particularly in safety-critical deployments.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. CARLA: An open urban driving simulator. In Levine, S., Vanhoucke, V., and Goldberg, K. (eds.), Pro- ceedings the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 1-16. PMLR, 13-15 Nov 2017.

---

Learning Fused State Representations for Control from Multi-View Observations
Dunion, M. and Albrecht, S. V. Multi-view disentanglement for reinforcement learning with multiple cameras. arXiv preprint arXiv:2404. Ttt1:2404.14066 14064, 2024.
Hwang, H., Seo, S., Jang, Y., Kim, S., Kim, G.-H., Hong, S., and Kim, K.-E. information-theorett state space model for multi-view reinforcement learning. 2023.
Efroni, Y., Misra, D., Krishnamurthy, A., Agarwal, A., and Langford, J. Provably filtering exogenous distractors using multistep inverse dynamics. In International Con- ference on Learning Representations, 2021.
Jangir, R., Hansen, N., Ghosal, S., Jain, M., and Wang, X. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):3046-30533 2022.
Efroni, Y., Foster, D. J., Misra, D., Krishnamurthy, A., and Langford, J. Sample-efficient reinforcement learning in the presence of exogenous information. In Conference on Learning Theory, pp. 5062-5127. PMLR, 2022.
Karpathy, A. and Fei-Fei, L. Deep seman-- align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Feng, F., Wang, X., and Li, R. Cross-modal retrieval with correspondence autoencoder. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 7-16, 2014.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.6900 2014.
Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Ferns, N., Panangaden, P., and Precup, D. Metrics for finite markov decision processes. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, UAI '04, pp. 162-169, Arlington, Virginia, USA, 2004. AUAI Press. ISBN 0974903906.
Lamb, A., Islam, R., Efroni, Y., Didolkar, A., Misra, D., Fos- ter, D.. Molu, L., Chari, R., Krishnamurthy, A., and Lang- ford, J. Guaranteed discovery of control-endogenous la- tent states with multi-step inverse models. arXiv preprint arXiv: a2Xiv:2207.08229, 2022.
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, Т. Devise: A deep visual- semantic embedding model. Advances in neural informa- tion processing systems, 26, 2013.
Laskin, M., Lee, K., Stooke, A., Pinto, L.., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data, In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 19884-19895. Curran Associates, Inc., 2020a.
Fu, X., Yang. G., Agrawal,
P., and Jaakkola, T. Learning task informed abstractions. In International Conference on Machine Learning, pp. 3480-3491. PMLR, 2021.
Geng, X., Liu, H., Lee, L.,, Schuurmans, D., Levine, S., and Abbeel, P. Multimodal masked autoencoders learn trans- ferable representations. arXiv preprint arXiv:2205.14204, 2022.
Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. International conference on machine learning, pp. 5639-5650. PMLR, 2020b.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018
Lee, A. X., Nagabandi, A.. Abbeel, P., and Levine, Stochastic latent actor-critic: Deep reinforcement learn- ing with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
Li, D., Dimitrova, N., Li, M., and Sethi, I. K. Multimedia content processing through cross-modal association. In Proceedings of the eleventh ACM international confer- ence on Multimedia, pp. 611, 2003.
Hafner, D., Lillicrap, P., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. In International Conference Learning Representations, 2021.
Li, M., Wu, L., Wang, J., and Bou Ammar, H. Multi-view reinforcement learning. Advances in neural information processing systems, 32, 2019.
Hafner, D.,. Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
Li, Y. Yang, M., and Zhang, Z. A survey of multi-view representation learning. IEEE transactions on knowledge and data engineering, 31(10))1863-188 -1833, 2018
Hansen, N., Su, H., and Wang, X. Td-mpe2: Scalable, robust world models for continuous control, 2024.
10

---

Learning Fused State Representations for Control from View Observations
Liang, D., Chen, Q. and Liu, Y. Sequential action-induced invariant representation for reinforcement learning. Neu- ral Networks, 179:00657, 2024, ISSN 0893-6080. doi: 106579.
Yarats, D., Fergus, R., Lazarie, A., and Pinto, L. Master- ing visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022.
Liu, F., Liu, H., Grover, A., and Abbeel, P. Masked autoen- coding for scalable and generalizable decision making. Advances in Neural Information Processing Systems, 35: 12608-12618, 2022.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 PMLR, 2020.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1777v.7707.06347, 2017.
Yu, T., Zhang, Z., Lan, C., Lu, Y., and Chen, Z. Mask- based latent reconstruction for reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 25117-25131. Curran Associates, Inc., 2022a.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pp. 618 618-626, 626, 2017.
Yu, T., Zhang, 7., Lan, C.,, Lu, Y., and Chen, Z. Mask- based latent reconstruction for reinforcement learning. Advances in Neural Information Processing Systems, 35: 25117-25131, 2022b.
Sco, Y., Hafner, D., Liu, H., Liu, F., James, S., Lee, K., and Abbeel, P. Masked world models for visual control. In Conference on Robot Learning, pp. 1332-1344. PMLR, 2023a.
Zang, H., Li, X., and Wang, M. Simsr: Simple distance- based state representations for deep reinforcement learn- ing. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 36(8):8997-900 Jun. 2022. doi: 10. 10.1609/aaai. 133...20883.
Seo, Y., Kim, J., James, S., Lee, K., Shin, J., and Abbeel, P. Multi-view masked world models for visual robotic manipulation. In International Conference on Machine Learning, pp. 30613-30632. PMLR, 2023b.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., al. Deepmind control suite. arXiv preprint arXiv: 18011.0000000 2018.
Zang, H., Li, X., Zhang, L.. Liu, Y., Sun, B., Islam, R. Tachet des Combes, R., and Laroche, R. Understanding and addressing the pitfalls of bisimulation- based repre- sentations in offline reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 2831 1-23400. Curran Associates, Inc., 2023.
Wan, S., Sun, H.-H., Gan, L., and Zhan, D.-C. Moser: learning sensory policy for task-specific viewpoint via view-conditional world model. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 5046-5054, 2024.
Ze, Y., Hansen, N., Chen, Y., Jain, M., and Wang, X. Visual reinforcement learning with self-supervised 3d represen- tations. IEEE Robotics and Automation Letters, 8(5); 2890-2897, 2023. doi: 10. 100//..A..023..222..
Wang, Z., Xiao, X., Xu, Z., Zhu, Y., and Stone, P. Causal dynamies learning for ioseeindeeendent state abstraction. arXiv preprint arXiv:2206. /3452, 2022.
Zhang, A., McAllister, R. T., Calandra, R., Gal. Y., and Levine, S. Learning invariant representations for rein- forcement learning without reconstruction. In Interna- tional Conference Learning Representations, 2021.
Xie, Y., Lin, B., Qu, Y., Li, C., Zhang, W., Ma, I.., Wen, Y., and Tao, D. Joint deep multi-view learning for image clustering. IEEE Transactions on Knowledge and Data Engineering,
36044,36666220 2020.
Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., Daumé III, H., and Huang, F. TACO: Temporal latent action-driven contrastive loss for visual reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 48203- 48225. Curran Associates, Inc., 2023.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented re- inforcement learning, arXiv preprint arXiv:2107.09645, 2021a.
Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021b.

---

Learning Fused State Representations for Control from Multi-View Observations
A. Proofs
Lemma 4.2. Given a latent MDP M constructed by learned aggregator & O O² x.... O² + Z that clusters multi-view observations in a E-neighborhood. The optimal value functions of the original MDP and the latent MDP are bounded as:
28
-V" (1-7)(1-0) -cc)
(11)
Proof. The proof follows straightforwardly from DBC (Zhang 2021). From Theorem 5.1 in Ferns et al. (2004) have:
(1 - (--)))
+ 1- NEO x XOL
7
max g(u, d),
(12)
where g the average distance between a multi-view observation and all other multi-view observations in its equivalence class under the bisimulation metric d. By specifying e-neighborhood for each cluster of multi-view observations, replace g:
(1 - - < 25 + 728
2
1-
[V --''0) - 1 - c
1
23 +
7
25
1 - ?
25
(1 -1] cc)
A + 0, the optimal value function of the aggregated MDP converges to the original value function. By defining a learning error for Φ, f == supa, xO2x..xX0x [l)())) ((,))1 - d((os, o,)], we can also update the bound in Lemma 4.2 to incorporate L: [V V (o(o))
26
(1-))
B. Additional Related Work Discussion
In this section, we provide a more detailed discussion on state representation learning RL. and detailed comparison between our method and other related work.
B.1. State Representation Learning in RI.
Well-constructed state representations enable agents to better understand and adapt to complex environments, thus improving task performance and rocisiio-mmaing efficiency. For example, methods such as CURL (Laskin et al., 2020b) and DrQ series (Kostrikov et al., 2020; Yarats et al., 2021a), leverage data augmentation techniques such as cropping and color jittering to enhance model generalization. However, their performance is highly dependent the specific augmentations applied, leading to variability in results. Masking-based approaches (Seo et al., 2023b; Yu et al., 2022b; Seo et al., 2023a; Liu et al., 2022), selectively obscure parts of the input to mitigate redundant information and improve training efficiency. Although these methods show promise in filtering out irrelevant data, they carry the risk of unintentionally discarding task-critical information, potentially affecting overall agent performance. Bisimulation- based methods (Zhang et al., 2021; ;; Zang et al., 2022) focus on constructing reward- sensitive state representations to ensure that states with similar values are close in the representation space, promoting sample efficiency and consistent diccisionmmaking Another line of research explores the causal relationships between state representations and control (Wang al., 2022; Lamb et al., 2022: Efroni et al., 2021; 2022; Fu et al., 2021). By analyzing the causal links between states and actions, these methods aim to improve agents' understanding and control of the environment, further optimizing RL performance.
B.2. Comparison with Additional Related Work
MV-MWM (Seo et al., 2023b). MV-MWM proposed an RL framework that trained multi-view masked autoencoder for representation learning and a world model to enhance the model-based RL algorithm's 's robustness to view variations in robotic manipulation tasks. MV-MWM adopts a dual masking strategy at both the convolutional feature and view levels, requiring an additional decoder to perform pixel-level reconstruction, which aims to recover spec-fc details from raw observations. Our method applies masking at the pixel level but conducts reconstruction in the latent space, thereby avoiding
12

---

Learning Fused State Representations for Control from Multi-Viewy Observations
the reconstruction of task-irrelevant details and focusing high-level task-related information. Furthermore, MV-MWM incorporates other robust components to enhance performance. For example, it is built on the powerful model-based RL baseline, Dreamer-v2 v2 (Hafner et al., 2021), and utilizes expert demonstrations to guide policy learning. In contrast, our method does not employ any of these components. As MV-MWM requires additional expert demonstration data for training, we did not conduct direct comparison with in our work.
MLR (Yu et al., 2022a), Although both MLR and our method leverage a mask and latent reconstruction auxiliary task to learn more compact representations, there are notable differences in their implementation details. Firstly. MLR predicts complete state representations in the latent space from observations with spatially and temporally masked pixels. It prioritizes extracting inherent information from images, often overlooking task-relevant information. In contrast, method integrates bisimulation metric learning, enabling the extraction of task-relevani information from multi-view observations. Secondly, our method employs the self-attention module as a fusion mechanism to learn fused state representation that directly contributes to downstream RL tasks. Instead, MLR uses the self-attention mechanism merely as an auxiliary loss module to encourage the CNN encoder to capture temporal dependencies within sequences.
C. Experimental Details
C.1. Hyperparameters
Table 2 and provide detailed information regarding the experimental setup and hyperparameter configurations, In the Meta-World and PyBullet environments, we utilize latent representation size of 128, following the Keypoint3D framework outlined by Chen et al. (2021). In the CARLA and DMControl environment, we utilize a representation size of 50, adhering to the SAC framework described Zhang et al. (2021). All networks in both the policy and the representation models are optimized using the Adam optimizer (Kingma, 2014), ensuring consistent performance across various environments.
Table 2: MFSC's hyperparameters, based on PPO.
Hyperparameter
Meta- Meta-World PyBullet
General
Batch size
6400
16000
Rollout buffer size
100000
100000
Frame stack
1
2
Num of views
3
3
Epochs per update
8
10
Discount
0.99
0.99
GAE lambda
0.95
0.95
Clip range
0.2
0.2
Entropy coefficient
0,0
0.0
Value function coefficient
0.5
0.5
Gradient clip
0.5
0.5
Target KL.
0.12
0.12
Learning rate
2 x 10-4
2 x 10-4
State representation dimension
128
128
Weight of loss X
1.0
1.0
MFSC
Representation learning rate
2 10
2 10
Weight of fusion loss
1.0
1.0
Weight of reconstruction loss
1.0
1.0
Number of dynamics models
5
5
Mask ratio
0.8
0.8
Cube spatial size
12 x 12
12 x 12
Cube depth
3
3
Self-attention fusion module depth
2
13

---

Learning Fused State Representations for Control from Multi- Observations
Table 3: MFSC's hyperparameters, based on SAC.
Hyperparameter
Carla
DMControl
General
Batch size
128
128
Replay buffer size
100000
100000
Frame stack
3
3
Num of views
3
4
Discount
0.99
0.99
Critic Q-function soft-update rate
0.01
0.01
Critic target update frequency
2
2
Actor update frequency Init temperature Learning rate
2
2
0.1
0.1
1 X 10-3
2 x 10
Temperature learning rate
1 1 10
1 1 10
State representation dimension
50
50
Weight of loss À
1.0
1.0
MFSC
Representation learning rate
2 10
x 10
Weight of fusion loss
1.0
1.0
Weight of reconstruction loss
1.0
0.5
Number of dynamics models
5
5
Mask ratio
0.5
0.5
Cube spatial size
12
12
12 x 12
Cube depth
3
3
Self-attention fusion module depth
2
2
C.2. Latent State Dynamics Modeling
Following approach, develop ensemble version of deterministic dynamics models (x(-)ou (x), Unlike probabilistic dynamics models, our transition models are deterministic and their outputs are consistent with the encoder's output, both of which are subjected to Iz-normalization. Instead of using a probabilistic transition, we calculate the distance using cosine similarity. Specifically, at the training step, we update the parameters of the dynamics models based on the cosine similarity loss function:
K
Pa(+low(O) (6) [Pa(.lo a) (ll
Layn
(13)
K
where E (1.2....,K) Since deterministic models share the same gradient but are initialized randomly, they may still acquire different parameters after training. This ensemble model allows us estimate the latent dynamics of the environment effectively while ensuring that the output remains consistent across the encoder and dynamics model. At the inference step. randomly sample one of K deterministic dynamics models to compute the transition to the next latent state s'.
C.3. Reward Normalization
Reward normalization crucial component our representation learning approach, it relies directly on the reward function to guide the extraction and learning of learning. In experimental tasks, the rewards used for representation learning are consistent with those used in policy learning. Following Keypoint3D (Chen et al., 2021), to ensure stable learning dynamics, we apply moving average normalization method to dynamically normalize the reward values. This method calculates the moving average of historical rewards and adjusts the rewards to have a mean and a standard deviation 1. This normalization process helps mitigate fluctuations in reward values caused by variations in task difficulty, environmental changes, or exploration strategies, enabling the model to more effectively learn meaningful representations from stable
14

---

Learning Fused State Representations for Control from Multi- View Observations
Right
Figure 8: Visualization of three view observations in the Meta- World, PyBullet' Ant and CARLA environment. Left & Middle: Third-person view observations from three different cameras in the 'Close Window' and PyBullet's Ant task. Right: First-person observations from three different views in the CARLA. It can be observed that in the Meta-World and PyBullet environments, each view exhibits a relatively high degree of coupling, encapsulating a substantial portion of the task-relevant information. However, due to factors such as occlusions (e.g., g., the robotic arm obstructing the target object or the leg joints in the PyBullet's Ant task), no single view captures the complete state information. In contrast, in the CARLA environment, the information contained in each view is mutually independent, and the three views collectively constitute the environment's state.
reward signals. Additionally, since the scale of rewards influences the bisimulation metric and the upper bound of value function errors, we adopt reward scaling to avoid feature collapse and reduce bisimulation measurement errors. Following the work of Zang et al. (2023), we apply G = 1 - and C = to compute bisimularity.
D. Environment Details
Meta-World. To evaluate whether model can accelerate policy optimization when jointly trained with the policy, perform five complex robotic arm manipulation tasks in the Meta-World environment (Yu et al., 2020). Each task involves 50 randomized configurations, such as the initial pose of the robot, object locations, and target positions. As shown in Fig.8, for each task, we use three third-person cameras from different angles to observe the robot arm and relevant objects. Since the gripper state at the end of the robotic arm may not be clearly visible from any of the three camera angles, following the settings of Chen et al. (2021) and Hwang et al. (2023), an indicator is introduced the Meta- World tasks to signify whether the gripper open or closed. This indicator is concatenated with the learned latent state and is fed into the actor network. Due to experimental variations, adopt the results reported in the F2C paper for comparison.
Py PyBullet's Ant. The PyBullet's Ant (Coumans & Bai, 2022) task is designed to simulate the motion control of quadruped robot in a highly dynamic 3D-locomotion environment. The objective of this task is to control the joints of the robot's legs, enabling to learn how to balance and move as quickly and stably as possible. The Ant robot has a highly dimensional state and action space, which includes physical quantities such as joint angles, angular velocities, and linear velocities. The robot's movement generated by controlling the torque or force applied to its joints, making fine-gres greiner understanding of the movable joints and parts essential. As locomotion environments require temporal reasoning, we use frame stack of 2. The reward function in this task is typically based on the robot's 's forward velocity while accounting for control costs (energy consumption) to incentivize efficient movement. Due to the complexity of the environment and the high-dimensional action space, the Ant task provides a significant challenge for training and testing RL algorithms.
CARLA. To evaluate the performance of our approach in more realistic multi-view tasks, we applied MFSC to the vision-based autonomous driving domain based on CARLA (Dosovitskiy et al., 2017). In this experiment, select the official 'Town04' map, featuring circular highway with an intersection, as illustrated in Fig.9. The agent's 's goal is to drive as safely as possible for limited 1. 1,000-time steps, avoiding collisions with other moving vehicles or obstacles. Following the experimental setup of DBC (Zhang al., 2021). the reward function is designed as re (s, a) v üiighwwy At- A impulse- X |steer), where v represents the velocity vector of the ego vehicle projected onto the unit vector of the highway ünighway. and then is multiplied by time discretization At to represent the effective distance. Impulse is the clash force, measured in Newton seconds, and Steer is the output of the steering amplitude. As shown in Fig.8, we use three cameras on the vehicle's roof, capturing left, front, and right views of the driver, each with a 60-degree field of vision. The agent must operate the vehicle's 's movement through first-person vision under varying conditions of wind, rain, clouds, or sunlight.
The bird's sird's-eye view of the 'Town04' map and snapshots of training period.
Figure 9:
15

---

Learning Fused State Representations for Control from Multi- View Observations
Deep Control Suite. The DeepMind Control Suite (DMControl) (Tassa al., 2018) is widely used benchmark environment for evaluating RI. algorithms, offering a collection of continuous control tasks with interpretable rewards and standardized structures, Although DMControl is not inherently designed for multi-view imputs, we adapt it by utilizing historical time-series data as proxy to simulate multi-view observations. This adaptation allows us to explore how temporal and multi-view information can enhance policy optimization. In our experiments, we select representative tasks from DMControl, including balance control, robotic manipulation, and motion planning, covering a range of complexities from simple to moderately challenging continuous control scenarios.
Multi-view observations in the Merl-World and PyBullet environments exhibit a greater degree of redundancy. This redundancy allows the attention mechanism to effectively focus shared, useful information across views. However, in CARLA, although multiple views are provided, the differences between views can be much more pronounced. As shown in the right part of Fig.8, observations from different views are highly variable (e.g., different roads, and moving vehicles), leading to visual and semantic dissimilarity between different views. The larger differences between views make it harder to fuse the information effectively, and relying solely on attention mechanisms may not capture sufficient cross- view correlations. Therefore, concatenating image observations from multiple views is a straightforward yet effective approach.
E. Additional Experimental Results
E.1. Results on Deep.ind Control Suite
To compare our method with the latest single-view RL methods, we downgrade MFSC to single-view setting on the Deep- Mind Control Suite. Specifically, as shown in Fig.10, we construct "multi-view" observations by selecting observations Ot-3, Or-2, O O₂ from four consecutive time steps.
O,,
O,,1
O2
We selected the previously high-performing model-free RL al- gorithms SAC (Haarnoja et al., 2018), CURL (Laskin et al., 2020b), DrQ-v2 (Yarats et al., 2022) and TACO (Zheng et al., 2023), along with the model-based RL algorithms Dreamer-v3 (Hafner et al., 2023) and TD-MPC2 (Hansen et al., 2024). The results of DrQ-v2, Dreamer-y3 and TD-MPC2 are obtained from Yarats al. (2022), Hafner al. (2023), Hansen et (2024). The reproduced results for TACO are implemented by the official code The results of SAC and CURL are downloaded from this link 2.
Figure 10: "Multi-view" observation in DMControl. Each observation consists of a stack of three consecutive image frames.
The results, summarized in Table 4, indicate that MFSC, when degraded to single-view setting, performs competitively with established single-view RL algorithms. Although there is slight performance gap compared to the SOTA algorithm TACO, MFSC consistently achieves results on par with first-tier methods such as Dreamer-v3 v3 and TD-MPC2 across the evaluated tasks, and it surpasses that of SAC, CURL, and DrQ-v2 across most tasks. Furthermore, these results demonstrate the efficacy of MFSC's representation learning framework, even when reduced to a sequential single-view scenario. Despite the constrained single-view conditions, MFSC's ability to remain competitive reaffirms the robustness of its design and its suitability for wide range of RL tasks.
Table 4: Comparison results on DeepMind Control Suite. The results of MFSC and TACO are presented as the mean and standard deviation across four random seed runs. The best score is marked with colorbox.
Model-free RL.
Model-based RI.
Task (1M Steps)
MFSC(ours)
SAC
CURL
DrQ-v2
TACO
Dreamer-v3
TD-MPC2
Cartpole Swingup
832::29
717+1144
758+155
865+15
864+22
86249
Cheetah Run
6794:33
19+14
502-:120
710+:52
915:16
841:4877
537-660
Finger Spin
97747
109+104
873+145
860+104
978+10
194
98512
Walker walk
909-L40
41::17
818::26
7701:371
968
959122
94018
16

---

Learning Fused State Representations for Control from Multi- View Observations
E.2. Results on CARLA
The results of CARLA are shown in Table 5. MFSC achieves the highest performance on three of six driving metrics. Our method outperforms SAC on both crucial comparisons of episodic reward and episodic distance. Furthermore, in terms of metrics reflecting vehicular stability during operation, such average steering amplitude variation, average braking intensity, and collision severity, all algorithms exhibited relatively modest performance. This is attributed to the reward function being primarily crafted to validate representation learning rather than specifically tailored to address safety considerations. Overall, in the CARLA task, our algorithm does not exhibit sigmificant improvement over the baseline algorithm SAC. This further supports our previous viewpoint that, in multi-viee view tasks like CARLA, where there is almost no shared information between views, concatenating image observations from multiple views, as in SAC, is a straightforward yet effective approach.
Table 5: Comparison results of different driving metrics in CARLA. Results of DrQ (Yarats al., 2021b) and CURL (Laskin al., 2020b) are obtained from Liang al. (2024). The best score is marked with colorbox. + corresponds to a standard deviation of three random seeds, after 400k training steps. The direction of the arrow indicates whether the desired metric is large or small.
Metrics
SAC
DrQ
CURL
MESC(ours)
Distance (m) 1
179.60 + 76.29
158.10 + 1.54
172.50 + 7.60
183.30 + 67.39
Successes episodes t
21%
21%
14%
13%
Average steer 1
10.24%
12.02%
9.76%
8.94 %
Avergae brake +
0.94% %
1.66%
1.75%
1.02%
Crash intensity (N/s) $
3750
3313
4311
4211
Episode return f
152.73 4.54
122.80 + 2.58
149.10 + 14.10
168.01 + 9.62
E.3. Visualization and Analysis Noisy View
To evaluate the robustness of our method under interference views, we introduced noisy-views setting. Specifically, as illus- trated in Fig.11, 1, in addition to the clean views O1, O2, O3, that are normally acquired, we incorporate an additional viewpoint O4 that is entirely irrelevant to the task. The noisy view O4 con- sists of distracting images, all sourced from the video hard datasets DMControl Generalization Benchmark (DMControl- GB) 3. We employed the same Grad-CAM technique as in Fig.4
O, Figure 11: Multi-view observations in noisy view set- ting.
O,
O2
to visualize the features of the four views. As shown in the visualization results of Fig. our method remains capable of capturing relevaelevant details even under the noisy interference setting.
Push w/ Wall
Close Drawer
Figure 12: Visualization of learned representation on noisy view. Pixels with the highest gradient values (top 2.5%) across all views are marked white while others are marked black.
hhttps:// com / ni ck
17

---

Learning Fused State Representations for Control from Multi-View Observations
To investigate whether bisimulation assist MFSC in countering the interference of noise, conducted ablation experiments under the setting of interference views. The results of Fig.13(aa) confirm that bisimulation effectively aids MFSC in mitigating the impact of noise, allowing it to maintain its performance even in the presence of disruptive noise. This demonstrates the robustness of MFSC when enhanced with bisimulation, highlighting its ability to focus on relevant features while disregarding irrelevant noise.
x10
Push w/ Wall
x10
Open Window
3.0
MultiView(ours)
View1
2.4
Episode Return
Episode Return
View2
View3
1.2
0.6
MFSC_ Robust view
MFSC Bisimulation
0.0
0.0
0.8
1.6
2.4
3.2
4.0
0.0
0.8
1.6
2.4
3.2
4.0
Environment Steps
1e6
Environment Steps
1e6
(a)
(b)
Figure 13: (a) Ablation study on robust view. (b) Ablation study on varying view counts.
E.4. Ablation Study on Varying View Counts
We have added ablation study that varies the number of views, and the results presented in Fig13(b). It shows that using any single view out of the three performs worse than the full multi-view setup. In particular, View 3 - top-down perspective - suffers from severe occlusion, resulting in the lowest performance among the three. This highlights the limitations of relying on individual views and underscores the effectiveness of our approach in extracting and fusing complementary information across views. Overall, these results validate the necessity and benefit of multi-view learning in achieving more robust and informative state representations.
E.5. Quantitative Analysis on Bisimulation Metric
To provide more quantitative analysis of the bisimulation metric, we include the training curve of the bisimulation loss and additionally measure the mutual information 7 (=);;) between the final fused embeddings =f, and the ground-truth states s using the MINE" method. As illustrated in Fig. 14, with the progression of training and the convergence bisimulation loss, observe consistent increase in mutual information. This indicates that the model not only optimizes the bisimulation criterion but also gradually constructs task-relevant representation space aligned with the true environment states.
Open Window
Open Window
x10
x10
2.0
Episode Return
4.4
Bisimulation Loss
1.6
Mutual Information
Episode Return
4.2
Mutu al Information
1.2
4.0
3.8
0.4
3.6
0.0
3.2
Environment Steps
le6
Environment Steps
1e6
Figure 14: Quantitative analysis bisimulation metric.
18

---

Learning Fused State Representations for Control from Multi-View Observations
Input
Input
Moving Average
Momentum
CNN Encoder
CNN Encoder
CNN Encoder
CNN Encoder
Moving Average
Transformer
Transformer Encoder
Transfomerr Encoder
Momentum Trans. Encoder
Encoder
Stop Gradient
Stop Gradient
Similarity
Pred. Hcad
Similarity
Figure 16: Comparison on network architectures. Left: SimSiam architecture. Right: Momentum encoder architecture.
'//' represents a stop-grade gop-gradient operation.
E.6. Hyperparameter Sensitivity Analysis
In the overall loss function of MFSC in Eq. 10, the weighting coefficient À is set to 1, signifying that both loss components contribute equally to the total loss. We also conducted a hyper- parameter sensitivity analysis evaluate the impact of different fusion loss and reconstruction loss contributions on the algo- rithm's performance. Specifically, compared three configu- rations: À = 1 (original setting), indicating equal contributions from both losses, À = 2, signifying a greater contribution from the fusion loss, and À = 0.5, denoting a greater influence from the reconstruction loss. The results are visualized in Fig. 15 for two tasks: 'Close Box' and 'Ant No Color'.
X30
Close
X30
Anii
Edttore Retum
12
0.4
Results of loss weight tuning experiment. The vertical lines on the barchart represent the standard deviation of the results from the three seeds.
Figure 15:
The experimental results indicate that the model performs rela- tively better when the weights of the two losses equal (i.e., == 1). Increasing decreasing the weight of either loss may result in performance variations, likely due to the model overemphasizing one loss component while neglecting the importance of the other. This finding underscores the critical importance of balancing the fusion loss and the reconstruction loss to optimize both the overall performance and stability of the model.
E.7. Results on Different Network Architectures
To verify the robustness of our method in different network architectures, we implemented two distinct MFSC network designs and performed evaluations in the Meta- World environment. As illustrated in the left part of Fig. the network employed the paper adopts a structure which similar to the SimSiam (Chen & He, 2021) architecture. The encoder includes layers that can be shared between both branches. To avoid collapsing, we applied stop-gradient operation to one branch of the network. Furthermore, we also implemented MFSC with a network architecture based on a momentum encoder. As shown in the right half of Fig. in addition the stop-grade gradient operation, the momentum encoder is updated using the exponential moving average method.
In Fig.17, 17, it is evident that although the performance of the MFSC with two distinct network architectures differs on certain tasks, such as 'Open Window' and 'Push w/Wall', overall, the algorithms exhibit comparable performance across both architectures. This outcome suggests that our method demonstrates a certain degree of robustness with respect to the design of the network architecture.
19

---

Learning Fused State Representations for Control from Multi View Observations
x127
Close
X.2
Drawer
+10
Open
Pog Unplin
-100
Vall
Edoode Retur
00
146
Dncironment
165
145
Figure 17: Performance comparison of two different network architectures on Meta-World.
E.8. Results of the Comparison with Additional MVRL Algorithm
We compare method with the model-based MVRL algorithm MOSER (Wan al., 2024). MOSER seeks the optimal viewpoint for learning task representations under multiple views. Its implementation code is available at this link.. The corresponding experimental results are illustrated in Fig. 18. Our experimental results show that our approach outperforms the MOSER algorithm both tasks, demonstrating the effectiveness of our method.
Cheetah Run
Walker Walk
x10
x10²
1.0
1.0
MFSC[ours)
0.8
MOSER
Episode Return
Episode Return
0.6
0.4
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
1e6
Environment Steps
1e6
Figure 18: Performance comparison of an MB-MVRL algorithm MOSER.
E.9. Comparison of Training Resources
We compared the training time, inference time, and resource requirements of MFSC with the following baselines: single- view methods RAD and Vanilla-PPO, as well as multi-view methods Keypoint3D and LookCloser. The experiments were conducted on a server equipped with two NVIDIA A800 GPUs (Ampere architecture) with 80GB of VRAM each, running on CUDA 12.3 and NVIDIA driver version 545.23 23.66. The server is powered by dual Intel Xcon Platinum 8370C CPUs, each featuring 32 cores and 64 threads, with a base clock speed of 2. 2.80GHz and maximum clock speed of 3.5GHz. The system has total of 128 logical processors and NUMA configuration with two nodes. The hardware setup ensures sufficient computational resources train and evaluate the proposed method.
Table 6: Comparison of training resources for different algorithms. The last column represents the improvement in the average score of five tasks in Meta- World compared to the baseline algorithm RAD. Optimal results in the MVRL algorithms are marked as bold and underlined.
Category
Method
Training time
Inference time
GPU memory
Performance improvement
View Concatenating
Vanilla PPO
3.8h
2.67ms
850MB
RAD Keypoint3D
3.8h
2.69ms
758MB
12.2h
6.01ms
1406MB
21.17%
View Fusion
LookCloser
13.9h
14.35ms
3530MB
45.66%
MFSC (ours)
14.9h
4.26ms
4098MB
97.93%
20

---

end for Learning Fused State Representations for Control from Multi-View Observations
As shown in Table due the complexities of information fusion, model architecture, and cross-view feature extraction, multi-view fusion algorithms typically require resources than simpler multi-view image concatenating methods. Compared to other multi-view fusion algorithms, our method demands greater training time and GPU memory. However, fortunately, our algorithm significantly outperforms others in terms of inference time and performance improvement. This indicates that while our method may require more resources during the training phase, it offers much more efficient performance during the inference phase, which is a crucial advantage practical applications.
F. Algorithm
Our pseudocode in Algorithm 1 and Algorithm 2, where Algorithm is based Proximal Policy Optimization (PPO) and Algorithm& 2 is based on Soft Actor-Critie (SAC). Furthermore, we illustrate the representation learning component of MFSC in Algorithm 3 using PyTorch-like style for enhanced clarity and comprchension.
Algorithm Pseudocode MFSC, based on PPO
Input: iterations repeat entire processes NRepeat batch size rollout length T..
2: for iter 1 to MRRppaa do
3: Initialize Brollout
4: for b to B do
5: Run policy The to collect (,,
6: Brotlout + Brotlout U (,,,,,,'))ur
7: end for
8: Estimate advantage values ÀITTN on Brottout
9: for to 7 do
10:
Sample (ō, a. (.rr,") 0") Brollout
11:
Patch masking the multi-view observation Ø
12:
Calculate Lpoticy b PPO algorithm
13:
Calculate Lins according to Eq.8
14:
Calculate Free according to Eq.9
Calculate Ldyn according Eq.
Optimize Cpoliey Cree + Lius + Ldyn throughout Brollout
end for
18:
Told
19:
Algorithm 2 Pseudocode of MFSC, based SAC
1: Input: maximum time steps T, batch size B, replay buffer size [D], target network update frequency т.
2: for Time
do
3: Encode state: 24 ((ō)
4: Execute action:
n (Zt)
5: Record data: D + U fō,, a,, rz, 0
6: Sample batch: Bg ~ D
7:
Update value network: Es, [(V)
> SAC algorithm
8: Update policy network: ER, [(T)] SAC algorithm
9: Update fusion network Φ: Es, [[))] according to E0.
10: Update the dynamics: Eg, (p( where j {1...., K} according to Eq.
11: end for
21

---

Learning Fused State Representations for Control from Multi- View Observations
Algorithm/ 3 Pseudocode for the representation learning component of MFSC, Py Porcch-like
encoder: cnn + mlp self-attention fusion network, the output L2-normalized transition soaiiiionnnnelll mlps, ensemble of transition modals, the output iss 12-norma ixed global mip, prediction head network Imb: waighting coefficient
def cosine dis (featuze_a, festare_b) Feature_o_no-mm
.aa.fantttaanoorrre......... ize (feature_a, dimml] deature_b_nuste cas.fuccional hoo.aal..oooiieee.eeeee.. (Ieature_b) diaml) temp - forch.matmul. (feature_a norm, feature_ b. T dis forch.diag(temg zeturn dis
spr_loss (masked_z_ā, cargst_masked_... =_a):
self, selff.glabs__prdictto (masked_z_a)
with torch. corcch.no_gra(11=
blobsl_targan parget_masked_x. x_8
lass porm_nse__oss (global_laatennn Elebel_targeeee mean-False) meean
return loss
action, zeward, next_oba):
cbs wwt....eebbatchhhllleee nelf.num_cameras, self. frame_stack, solf.imaae_ccannels,, self. self.imago_sire self.
image_size)
next_obs next_obs view (batch_siiee size, self selffauu_cameras, self. frame_stack self. colf.image_channell. self. imaga_size, sclf, image_size!
reconstruction loss
masked self. engoodr www.ddrrmmssskke.. mask-_nssdeeeeeeeeeoooeoo (obs, mask-True)
with terch.no_gradiy
target_masked____ self encoder wwwpooderr.aaaaeeeee mask_encode (obs, mask-False)
masked_z_a a.a.ed__aa..latteee.. target_masked_i
1)
res_loss spr_ (nasked target_masked_
fasion 2_a = endceeec with torch.no (1:
pred_a
(Iz_e, action), dim-l))
pext_diff compute_dis (pred_a, pred_a)
r_diff = norch.abse (roward.f reward)
z_diff comoute_dia z_a]
bisimilerity self.e r_diff (1 self.c) next dife lus_loss torch.na. () (2_diff, desimilarsty.detaee
mfsc_losa- - fua_losa self, Imb * res_
transition less
www..lllenoeoooeee self.encoderlobel (obs]
next_h sal f.encoder (next_obs) wwsqueeze (0)
next__e - self, self..rrnnsttiorrrddde (torch.cat cat [[,, action), dimml))
cos_sim P.cosine similarity (next no. detachh pext_next_h, dimm-1)
losmsition__oss cos_sim.mean
return mfsc_losse transition_lass
22


Reviews:
[{"review_id": "w9SSLDrheS", "reviewer": "Reviewer_JyV5", "summary": "This paper addresses key challenges in multi-view reinforcement learning (MVRL), specifically redundancy in observations, distracting or irrelevant information, and robustness to missing views. To overcome these issues, the authors propose a framework, Multi-view Fusion State for Control (MFSC), integrating task-relevant representation learning inspired by the bisimulation metric and a self-attention-based fusion mechanism. Additionally, the method leverages a novel masking-based latent reconstruction auxiliary task to improve robustness against incomplete or missing observations. Experimental validation demonstrates that MFSC consistently achieves superior performance and robustness compared to existing state-of-the-art methods across several robotic manipulation and locomotion benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, and we will address your concerns in the following.\n\n------\n\n**W1:**\n\nWe noticed that the citation in Definition 3.1 was incorrect; the correct reference should be the $\\pi$-bisimulation metric proposed by Castro et al.(2020)[1], rather than Castro et al.(2021)[2]. We apologize for the confusion caused. To clarify, we provide a brief clarification on the connection between these works:\n\nConventional bisimulation metric needs to compute the Wasserstein distance over the transition distributions across all actions, which is computationally expensive. Instead, Castro et al.(2020)[1] developed $\\pi$-bisimulation which removes the requirement of considering all actions and only needs to consider the actions induced by a policy $\\pi$. Castro et al.(2021)[2] introduces independent coupling and proposes a sampling-based metric that does not rely on the Wasserstein distance; however, this comes at the cost of violating the “zero self-distance”,  potentially leading to representational collapse. In contrast, our use of the cosine distance is both efficient and guarantees this key property by design, as also demonstrated in SimSR [3]. This theoretically safeguards against collapse in learned representations. We will elaborate on this point more comprehensively in the next revision.\n\n[1] Castro, et al.(2020). Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes. \n\n[2] Castro, et al.(2021). MICo: Improved representations via sampling-based state similarity for Markov decision processes. \n\n[3] Zang, H, et al.(2022). SimSR: Simple Distance-Based State Representations for Deep Reinforcement Learning.\n\n------\n\n**W2:**\n\nWe fully agree that both bisimulation metrics and self-attention mechanisms are well-established techniques, yet each has limitations when applied in isolation to MVRL. Specifically, self-attention effectively aggregates information from multiple views but typically lacks an explicit mechanism to focus on task-relevant features, potentially introducing irrelevant or redundant information into the final representations. On the other hand, bisimulation alone does not explicitly utilize the correlations across multiple views. To the best of our knowledge, such an integration has not been explored in the context of MVRL. Following your suggestions, we will carefully revise the manuscript to more clearly emphasize the distinct advantages of this integration.\n\n------\n\n**The paper title and method name**\n\nThanks for your point out. We agree that emphasizing core elements such as bisimulation and self-attention would better highlight the novelty of our method. We will consider revising the paper title and method name to more clearly reflect these components by incorporating terms like “Bisimulation-Constrained Attentive”.\n\n------\n\n**Q1: necessity of the reconstruction loss**\n\nThe reconstruction objective is designed to exploit the inherent cross-view dependencies in multi-view observations, thereby enhancing the model’s representation capacity. The effectiveness of this self-supervised objective has been demonstrated in our ablation study (Fig.7b). \n\nWe would like to offer some clarifications regarding your concerns. Crucially, the reconstruction objective is applied in the **latent space**, distinct from pixel-level reconstruction. This design encourages the model to reconstruct information that is shared across multiple views, rather than relying on superficial pixel-level redundancy. Moreover, in our downstream RL tasks, we exclusively utilize $z_{L}^{0}$, which is optimized via the bisimulation objective. The theoretical foundation of bisimulation guarantees that $z_{L}^{0}$ preserves task-relevant information aggregated from multi-view observations. For this representation, the reconstruction objective serves to recover task-relevant information. We therefore contend that the representations used for downstream control primarily capture task-relevant features rather than redundant information, as demonstrated by the visualizations in Fig.4\\&6. We hope this explanation clarifies our motivations and design choices.\n\n------\n\n**Q2: results of F2C**\t\n\nWe conducted a careful review of recent MVRL baselines and considered that F2C demonstrates competitive performance on the MetaWorld benchmark according to its performance report. However, the official implementation of F2C for MetaWorld has not been released, and we were unable to reproduce its reported results, despite reaching out to the authors multiple times without receiving any response yet. To enable a comprehensive comparison and study in MVRL, we report the convergence performance of F2C as provided in the original paper. We appreciate your suggestion and will provide a detailed explanation of this decision in the revised version of our paper.\n\n------\n\nPlease do not hesitate to let us know if you have any additional comments.", "claims_and_evidence": "The manuscript provides the following valid claims and their supporting evidences:\n\n**Claim 1:**  MFSC effectively learns compact and task-relevant multi-view representations.\n\n**Evidence 1:** Empirical evaluation across multiple robotic manipulation tasks (Meta-World, PyBullet Ant) shows MFSC significantly outperforms baselines. In addition, visualization (Figure 4) qualitatively analyses using Grad-CAM show the method consistently focuses on task-critical features, indicating successful extraction of task-relevant information.\n\n**Claim 2:** Proposed masking-based latent reconstruction improves robustness against missing views.\n\n**Evidence 2:** Experiments demonstrate MFSC maintains stable and superior performance under scenarios with missing or noisy views, outperforming state-of-the-art methods such as LookCloser and F2C. The authors also provide an ablation test on Figure 7.(b).", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate and well-justified.", "theoretical_claims": "The bisimulation metric-motivated representation learning in MFSC ensures that the difference between optimal value functions of the original and latent MDPs is bounded, thus providing a theoretical justification for learning compact yet task-relevant multi-view representations. \n\nHowever, a theoretical gap exists between the Wasserstein distance between transition distributions—originally used in the MICo update operator—and the expected cosine distance between successor observations employed in this work. Although the authors attribute this gap primarily to computational complexity, a more rigorous theoretical discussion is necessary. Specifically, clarifications on the relationship between two metrics such as how the two metrics differ theoretically, and under which conditions they become equivalent, would significantly strengthen the paper.", "experimental_designs_or_analyses": "The experimental design is comprehensive, and analyses demonstrate method effectiveness and robustness.", "relation_to_broader_scientific_literature": "This work has potential implications for real-world applications requiring control under diverse observational settings, such as autonomous driving. In addition, one particularly relevant application area is multi-modal learning, such as vision-language models (VLM).", "essential_references_not_discussed": "The manuscript adequately covers relevant literature. To my knowledge, no essential references appear to be missing.", "other_strengths_and_weaknesses": "### Strengths:\n\n**1.\tPractical Contribution:**\n\nThe paper effectively integrates bisimulation metric learning into multi-view RL settings, particularly utilizing self-attention fusion modules, demonstrating clear practical advantages.\n\n**2.\tSound Experimental Validation:**\n\nThe research questions are well-motivated, and the authors provide extensive experimental evidence supporting their approach.\n\n\n### Weaknesses\n**1. Weak Connection between Bisimulation Metrics:**\n\nThe theoretical connection between the proposed expected cosine distance and the traditional bisimulation metric (originally employing the Wasserstein distance) is relatively weak and lacks sufficient analysis or justification.\n\n**2. Novelty:**\n\nBoth bisimulation metric and self-attention mechanisms themselves are individually well-established techniques, which somewhat limits the standalone novelty of the method. The paper does not clearly emphasize the theoretical advantages or technical challenges arising specifically from integrating these two components. Clarifying how this integration uniquely contributes to MVRL would strengthen the novelty claim.", "other_comments_or_suggestions": "In the manuscript, the paper title and method name are not intuitively clear about the proposed approach. Emphasizing key components such as bisimulation and self-attention fusion could help better highlight the novelty and differentiating aspects of the research.", "questions_for_authors": "**Question 1.**\n\nWhy did the authors consider the additional reconstruction loss necessary? I am slightly concerned that the proposed auxiliary objective (reconstruction loss) may inadvertently encourage the learned representations to rely on redundant information from views that could be missing during inference.\n\n**Question 2.**\n\nWhy did the authors choose to compare MFSC against the convergence performance of the specific baseline method (F2C)? Clarification on the motivation behind this comparison method would be helpful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sgKKDQ8rtc", "reviewer": "Reviewer_RYsq", "summary": "The paper proposes a novel framework for multi-view reinforcement learning to effectively learn task-relevant representations of the state from multi-view observations. The new framework not only incorporates the bisimulation metric which aligned representation with the task’s objectives but also add latent reconstruction as an auxiliary task to retain crucial details that are specific to each individual view. The paper also provide strong experimental results in realistic scenarios which demonstrate the effectiveness of the unified fusion state representation from multi-view setting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank all the reviewers for their thoughtful evaluation, constructive feedback, and valuable suggestions, which have greatly contributed to improving the clarity, depth, and overall quality of our work. We will address your concerns in the following.\n\n------\n\n\n**Q: Could you share the differences between MFSC and DBC when using the bisimulation metric?**\n\nWe sincerely appreciate the reviewer’s insightful question and are pleased to provide a more detailed clarification of the differences between MFSC and DBC in their use of the bisimulation metric. Specifically, MFSC differs from DBC in two key aspects: \n\n1. **Avoiding Wasserstein distance for behavioral similarity.** The bisimulation metric generally requires calculating the Wasserstein distance between distributions, which can be computationally expensive. To mitigate this complexity, DBC models latent dynamics transitions as Gaussian distributions, leveraging Euclidean distances to compute a closed-form Wasserstein distance, optimizing an $\\ell_1$ distance between representations. However, this approach assumes Gaussian dynamics and introduces an inconsistency between $\\ell_1$ and $\\ell_2$ (Euclidean) distances, potentially leading to inaccurate approximations. MFSC adopts sampling-based computation approach via independent coupling strategy, encoding current observations into latent representations and explicitly learning latent dynamics to generate representations of subsequent states.\n2. **Cosine distance as the base metric for latent state representation.**  Inspired by SimSR,  MFSC adopts cosine distance as the base metric for bisimulation computation. Please refer to our response W1 to Reviewer JyV5 for more details regarding the benefit of using cosine distance as the base metric.  Besides, both $\\ell_1$ distance (used for reward differences) and cosine distance (used for state differences) are conveniently scalable to matrix operations. Consequently, MFSC can efficiently compute a correlation matrix of state representations within a batch and optimize it to accelerate representation learning. This strategy significantly enhances efficiency compared to DBC’s pairwise comparisons via permutation, resulting in faster and more effective representation updates.\n\nWe sincerely thank the reviewer for the valuable comment and will provide a detailed clarification in the appendix of the revised manuscript.\n\n------\n\nPlease do not hesitate to let us know if you have any additional comments.", "claims_and_evidence": "The paper provides clear statement and strong evidence for most of the claims. I only have a very minor holdback on the claim of \"MFSC is the first to incorporate bisimulation into the MVRL framework\". The claim is true but the main concern is that the method used to incorporate bisimulation into the MVRL framework are very similar to DBC [1] except the introduction of state aggregator to fuse multi-view observations. I would suggest authors to better discuss the proposed method and DBC in related work or Section 4.1. I think MFSC have many advantages compared to DBC, especially on retaining details with the latent reconstruction task. More discussion on the bisimulation metric part will make the contribution even stronger.\n\n\n[1] Zhang, A., McAllister, R., Calandra, R., Gal, Y., and Levine, S. Learning invariant representations for reinforcement learning without reconstruction. International Conference on Learning Representation, 2021.", "methods_and_evaluation_criteria": "Yes, the proposed methods are well evaluated.", "theoretical_claims": "Yes, they all seem correct.", "experimental_designs_or_analyses": "Yes, the experimental results are sound and valid at my best knowledge.", "relation_to_broader_scientific_literature": "I believe the proposed method pushes further in both the field of multi-view reinforcement learning and the field of robust representation learning for RL.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Could you share the differences between MFSC and DBC when using the bisimulation metric?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9VBlhOWJJR", "reviewer": "Reviewer_uqci", "summary": "The paper proposes a novel framework for Multi-View Reinforcement Learning (MVRL). The key contributions include:\n\n- Integrating bisimulation metric learning into MVRL to extract task-relevant representations from multi-view observations.\n\n- Introducing a multiview-based masking and latent reconstruction auxiliary task to enhance robustness against missing or noisy views.\n\n- Demonstrating superior performance over existing methods in robotic manipulation (Meta-World, PyBullet) and autonomous driving (CARLA) tasks, especially in scenarios with interference or missing views.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for thoughtful and constructive feedback. We are pleased that our proposed framework for Multi-View Reinforcement Learning (MVRL), including the integration of bisimulation metric learning and masking-based latent reconstruction, was well-received. We appreciate the recognition of our theoretical formulation, experimental design, and the robustness of our method across diverse benchmarks. We truly appreciate your insightful feedback and constructive suggestions, and we will address your concerns in the following.\n\n------\n**W1: experimental setup of Deep Mind Control Suite**\n\nWe would like to clarify that our primary goal is to explore how multi-view information can be leveraged to enhance decision-making efficiency. While we focus on spatial multi-view in the paper, as demonstrated in our experiments with environments like Metaworld, PyBullet, and CARLA, we also recognize that \"multi-view'' can manifest in both temporal and even multi-modal forms. Hence we further explore the temporal information fusion in DeepMind Control environments. To address the potential confusion, we will clarify the distinction between these two experimental setups in the next version of the paper.\n\n------\n\n**Other Comments Or Suggestions 1: ablation study on varying view counts**\n\nWe appreciate the reviewer’s insightful suggestion. In response, we have added an ablation study that varies the number of views, and the results are presented in Fig. 1 at the following link **[link](https://anonymous.4open.science/r/MFSC-1488/rebuttal.pdf)**. It shows that using any single view out of the three performs worse than the full multi-view setup. In particular, View 3 — a top-down perspective — suffers from severe occlusion, resulting in the lowest performance among the three. This highlights the limitations of relying on individual views and underscores the effectiveness of our approach in extracting and fusing complementary information across views. Overall, these results validate the necessity and benefit of multi-view learning in achieving more robust and informative state representations.\n\n------\n**Other Comments Or Suggestions 2: In Fig.1, the symbol was mistakenly written.**\n\nThanks for your careful review and for pointing out the mistake. We appreciate your feedback and will correct the symbol in the next version of the paper.\n\n------\n\n**Q: model design details**\n\nThanks for your comment, and we apologize for any confusion caused. We clarify this design as follows: we introduce an additional learnable ``[state]`` embedding that interacts with the view-specific embeddings through a self-attention mechanism. The output of this embedding, denoted as $z_L^{0}$, is updated through the bisimulation loss. The theory of bisimulation guarantees that the learned representations retain all task-relevant information from multiple views, enabling $z_L^{0}$ to effectively aggregate this information into a unified representation. Importantly, MFSC optimizes the bisimulation objective exclusively with respect to $z_L^{0}$, not the view-specific embeddings $z_L^{1-3}$. This design choice follows a similar paradigm to BERT and ViT, where supervision is applied exclusively to a designated token (e.g., the ``[class]`` token). We also recognize that the current depiction of gradient flow in Fig.1 may be misleading, and we will revise it in future versions to accurately reflect the optimization process.\n\n------\n\nPlease do not hesitate to let us know if you have any additional comments.", "claims_and_evidence": "Yes, the claims are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes. The methods are largely appropriate for addressing the challenges of multi-view reinforcement learning.", "theoretical_claims": "Yes. The theoretical claims are sound.", "experimental_designs_or_analyses": "The experimental designs and analyses are largely sound but exhibit minor validity concerns in specific benchmarks and theoretical assumptions. For example, on DeepMind Control Suite environment, temporal stacking conflates temporal and spatial multi-view learning, undermining conclusions about MFSC’s multi-view fusion capabilities.", "relation_to_broader_scientific_literature": "The paper advances MVRL by integrating bisimulation metrics and masking-driven reconstruction into a cohesive framework. MV-MWM (Seo et al., 2023a) relies on pixel-level reconstruction and expert demonstrations, which may retain redundant details. In this paper, the latent reconstruction loss avoids reconstructing irrelevant details, improving efficiency.", "essential_references_not_discussed": "No, the essential references are discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n- This paper is clearly written and easy to follow.\n\n- It introduces a novel framework by integrating bisimulation metrics with multi-view fusion and masking-based reconstruction, addressing critical gaps in task-relevance and robustness for MVRL.\n\n- The work is supported by extensive experiments across diverse benchmarks (Meta-World, PyBullet, CARLA), including ablation studies and robustness tests under noisy/missing views.\n\nWeaknesses：\n\n- Using sequential frames as different views in the DeepMind Control environment is an invalid design choice. A more reasonable approach might involve training with three identical images as inputs to isolate multi-view fusion setting.", "other_comments_or_suggestions": "- The paper fixes the number of views to 3. How does the number of views impact the experimental results? An ablation study on varying view counts would strengthen the analysis. If performance with 1 view is comparable to multi-view setups, it raises questions about the necessity of multi-view learning or whether the method fails to extract useful cross-view information.\n\n- In Fig.1, the symbol $\\tilde{z}^n_L$ was mistakenly written as $\\tilde{x}^n_L$.", "questions_for_authors": "In Fig.1, I would like to understand the model’s design details: How does the architecture ensure that $z_L^0$ the fused state embedding) integrates features from all views? Is this achieved by blocking gradient propagation from $z_L^{1-3}$, and is there theoretical justification for this design?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BghosAd9r4", "reviewer": "Reviewer_NPK4", "summary": "The paper proposes a novel method called Multi-view Fusion State for Control (MFSC) to improve Multi-View Reinforcement Learning (MVRL). MFSC's main contribution is its integration of bisimulation metric learning into the MVRL framework, allowing for the extraction of task-relevant representations from multi-view observations. Additionally, the authors introduce a multiview-based masking and latent reconstruction auxiliary task to enhance the robustness of representation learning, notably when views are missing or noisy. The MFSC framework utilizes a Self-Attention Fusion Module that fuses multi-view representations using a learnable state fusion embedding. The bisimulation metric guides the learning of task-relevant features, while the auxiliary masking task helps the model deal with missing views by learning shared representations across different views. Key results show that MFSC outperforms existing MVRL methods in several tasks, including robotic manipulation and locomotion tasks, demonstrating superior performance in environments with missing or noisy views. The method is remarkably robust to interference and can handle incomplete observations effectively. The paper also includes visualizations and ablation studies that validate the effectiveness of each component, such as the bisimulation constraint and the self-attention mechanism, in improving task performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments, and we will address your concerns in the following.\n\n------\n\n**Potential Issues: hyperparameter tuning**\n\nThank you for your valuable feedback. For all baseline methods with publicly available implementations, we directly used their officially released code and default configurations to ensure they achieve the performance reported in the original papers.  We will include a clarification of this in the experimental details section.\n\n------\n\n**Potential Issues: limited metrics in CARLA**\n\nThank you for your suggestion. Our choice of evaluation metrics in CARLA is consistent with previous works. In addition, we would like to clarify that our evaluation metrics not only focus on driving performance (e.g., distance traveled, success rate of reaching 100 meters), but also take driving safety into account, including steering amplitude, braking intensity, and collision severity.\n\n------\n\n**Potential Issues: reward normalization**\n\nWe acknowledge the reviewer’s point that reward normalization may mask certain nuances in the reward signal. In our experiments, whether to apply reward normalization under different benchmarks depends on the implementation of the methods we compare against. We also conducted ablation studies on MetaWorld tasks, where reward normalization is required, to analyze the impact of reward normalization. Detailed results can be found at this **[link](https://anonymous.4open.science/r/MFSC-1488/rebuttal.pdf)** (Fig.2). The results show that without reward normalization, the performance of our method degrades significantly. This is because the large scale of raw rewards causes large gradients during Q-value updates, leading to oscillations in Q-value loss and overall instability in training. Additionally, reward normalization constrains the reward signal to a feasible range, which helps stabilize the optimization of the bisimulation objective, further improving the quality of the learned representations.\n\n------\n\n**Essential References Not Discussed**\n\nWe appreciate the additional references provided and are pleased to include Multi-view Dreaming and MOSER in our manuscript, as they significantly enhance the completeness and relevance of our work.\n\n------\n\n**W1 and Q1: the lack of comparison with model-based MVRL**\n\nWe appreciate the reviewer’s suggestion to compare our method with MB-MVRL algorithms. Since MV-MWM relies on additional expert demonstrations and Multiview Dreaming does not release its implementation code, we chose to compare our method with the MB-MVRL algorithm MOSER. MOSER seeks the optimal viewpoint for learning task representations under multiple views. Its implementation code is available at https://github.com/yixiaoshenghua/MOSER, and the corresponding experimental results can be found at this **[link](https://anonymous.4open.science/r/MFSC-1488/rebuttal.pdf)** (Fig.3). Our experimental results show that our approach outperforms the MOSER algorithm in both tasks, demonstrating the effectiveness of our method. We believe these additional comparisons help validate the superiority of our method.\n\n------\n\n**W2: quantitative analysis on bisimulation metric**\n\n Thanks for your insightful comment. To provide a more quantitative analysis of the bisimulation metric, we include the training curve of the bisimulation loss and additionally measure the mutual $I\\left( z_{L}^{0};s \\right)$ information between the final fused embeddings $z_{L}^{0}$ and the ground-truth states $s$ using the MINE [1] method. The results can be found at this **[link](https://anonymous.4open.science/r/MFSC-1488/rebuttal.pdf)** (Fig.4). As training progresses and the bisimulation loss converges, we observe a consistent increase in mutual information. This indicates that the model not only optimizes the bisimulation criterion but also gradually constructs a task-relevant representation space aligned with the true environment states.\n\n[1]Belghazi, et al. Mutual information neural estimation.\n\n------\n\n**Q2: Will conducting reinforcement learning from multiple perspectives lead to increased time and storage costs, and how can this issue be mitigated?**\n\nWhile multi-view observations provide a more comprehensive understanding of the environment, they may lead to increased computational time, storage cost. To mitigate this, we can leverage pretraining techniques to reduce training costs. Additionally, inspired by MOSER, we can train a view selector to dynamically choose the most relevant views for decision-making, thus may potentially minimize unnecessary computational and storage demands.\n\n------\n\nPlease do not hesitate to let us know if you have any additional comments.", "claims_and_evidence": "Yes, the paper's claims are generally supported by clear and convincing evidence. However, one potential area for further clarification is the handling of complex scenarios with significant occlusions or highly incomplete views. While the method shows robustness in many cases, some views may still contain critical information that is difficult to reconstruct, especially in tasks involving complex state transitions. This limitation could affect the performance in very challenging real-world environments, and further research into addressing these edge cases would strengthen the overall claims.", "methods_and_evaluation_criteria": "The comparison methods are all model-free multiview reinforcement learning approaches, lacking a comparison with model-based methods, which makes it difficult to fully convince us of the performance superiority of this method.", "theoretical_claims": "Yes, the proof of Value Function Bound is convincing.", "experimental_designs_or_analyses": "**Baselines and Comparisons**\n\n**Strengths:**\n\nThe authors compared MFSC with several baseline methods, including Keypoint3D, LookCloser, F2C, MVD, RAD, and Vanilla PPO. These baselines cover a range of approaches to MVRL, providing a robust comparison. The results show that MFSC consistently outperforms these baselines in most tasks, demonstrating the effectiveness of the proposed method.\n\n**Potential Issues:**\n\nMissing Baselines: As mentioned earlier, the comparisons do not include some recent MBMVRL methods. Including these methods would strengthen the validity of the results.\n\nHyperparameter Tuning: The paper does not provide detailed information on the hyperparameter tuning for the baselines. If the baselines were not tuned to their optimal settings, the comparisons might be unfair.\n\n**Performance Metrics and Evaluation**\n\n**Strengths:**\n\nThe authors evaluated the performance of their method using various performance metrics, including episode return, environment steps, and success rates. Using multiple seeds and confidence intervals in the results helps assess the statistical significance of the findings.\n\n**Potential Issues:**\n\nLimited Metrics in CARLA: In the CARLA environment, the evaluation metrics primarily focus on driving performance (e.g., distance traveled, success rate, steering amplitude, braking intensity, and collision severity). Including additional metrics, such as the number of collisions or the time taken to complete the task, could provide a more comprehensive evaluation.\n\nReward Normalization: The paper uses reward normalization to stabilize the learning process. While this is a common practice, it could mask some of the nuances in the reward signal. The authors should discuss the impact of reward normalization on the results.", "relation_to_broader_scientific_literature": "Bisimulation Metric Learning: Extends the concept of bisimulation metrics to the multi-view setting, providing a novel approach to learning task-relevant representations.\n\nApplication to Real-World Scenarios: Evaluates the method in a real-world autonomous driving environment, highlighting its potential for practical applications.", "essential_references_not_discussed": "Multiview Dreaming[1] extends the Dreaming algorithm to achieve integrated recognition and control from multiple perspective observations. This method utilizes contrastive learning to train a shared latent space between different views and integrates the latent state distributions from multiple perspectives through an expert mixture approach, thereby addressing the limitations of single-view observations in traditional reinforcement learning methods. MOSER[2] is a model-based approach that actively seeks the optimal perspective for learning task representations under multiple views to enhance performance.\n\n**References**\n[1] Kinose, A., Okada, M., Okumura, R., & Taniguchi, T. (2023). Multi-view dreaming: Multi-view world model with contrastive learning. Advanced Robotics, 37(19), 1212-1220.\n[2] Wan, S., Sun, H. H., Gan, L., & Zhan, D. C. (2024, August). MOSER: learning sensory policy for task-specific viewpoint via view-conditional world model. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (pp. 5046-5054).", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper presents a novel approach integrating bisimulation metric learning with a multi-view-based masking and latent reconstruction auxiliary task.\n- The paper is well-structured and clearly presents the methodology, experiments, and results.\n- The authors conducted extensive experiments to validate the effectiveness of MFSC.\n\n**Weaknesses**\n\n- The lack of comparison with model-based methods makes it impossible to fully validate the superiority of this approach.\n- There is a lack of quantitative analysis and presentation regarding the Bisimulation metric, not just ablation experiments.", "other_comments_or_suggestions": "Please see the Questions.", "questions_for_authors": "1. Could you provide more comparisons with model-based methods, such as MV-MWM, Multiview Dreaming, and MOSER?\n2. Will conducting reinforcement learning from multiple perspectives lead to increased time and storage costs, and how can this issue be mitigated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel method for multi-view reinforcement learning (MVRL) that combines bisimulation metric learning with a reconstruction loss. The paper is well-written, includes both theoretical and empirical evaluations, and is in general a solid contribution to the community.

The reviewers unanimously agree on its clarity, quality, and decision to accept.
Decision: Accept (poster)