Title: Hierarchical Adaptive Value Estimation for Multi-modal Visual Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: vision-based reinforcement learning

multi-modal

event camera
Abstract: Integrating RGB frames with alternative modality inputs is gaining increasing traction in many vision-based reinforcement learning (RL) applications. Existing multi-modal vision-based RL methods usually follow a Global Value Estimation (GVE) pipeline, which uses a fused modality feature to obtain a unified global environmental description. However, such a feature-level fusion paradigm with a single critic may fall short in policy learning as it tends to overlook the distinct values of each modality. To remedy this, this paper proposes a Local modality-customized Value Estimation (LVE) paradigm, which dynamically estimates the contribution and adjusts the importance weight of each modality from a value-level perspective. Furthermore, a task-contextual re-fusion process is developed to achieve a task-level re-balance of estimations from both feature and value levels. To this end, a Hierarchical Adaptive Value Estimation (HAVE) framework is formed, which adaptively coordinates the contributions of individual modalities as well as their collective efficacy. Agents trained by HAVE are able to exploit the unique characteristics of various modalities while capturing their intricate interactions, achieving substantially improved performance. We specifically highlight the potency of our approach within the challenging landscape of autonomous driving, utilizing the CARLA benchmark with neuromorphic event and depth data to demonstrate HAVE's capability and the effectiveness of its distinct components.

Hierarchical Adaptive Value Estimation for Multi-modal Visual Reinforcement Learning

Abstract

Integrating RGB frames with alternative modality inputs is gaining increasing traction in many vision-based reinforcement learning (RL) applications. Existing multi-modal vision-based RL methods usually follow a Global Value Estimation (GVE) pipeline, which uses a fused modality feature to obtain a unified global environmental description. However, such a feature-level fusion paradigm with a single critic may fall short in policy learning as it tends to overlook the distinct values of each modality. To remedy this, this paper proposes a Local modalitycustomized Value Estimation (LVE) paradigm, which dynamically estimates the contribution and adjusts the importance weight of each modality from a valuelevel perspective. Furthermore, a task-contextual re-fusion process is developed to achieve a task-level re-balance of estimations from both feature and value levels. To this end, a Hierarchical Adaptive Value Estimation (HAVE) framework is formed, which adaptively coordinates the contributions of individual modalities as well as their collective efficacy. Agents trained by HA VE are able to exploit the unique characteristics of various modalities while capturing their intricate interactions, achieving substantially improved performance. We specifically highlight the potency of our approach within the challenging landscape of autonomous driving, utilizing the CARLA benchmark with neuromorphic event and depth data to demonstrate HAVE's capability and the effectiveness of its distinct components. The code of our paper can be found at https://github.com/Yara-HYR/HAVE.

1 Introduction

Recent years have witnessed a renewed interest in multi-modal perception in the computer vision research community [10, 36, 20, 26, 37]. For many visual tasks such as semantic segmentation and object detection, the inclusion of multi-modal data ( e.g. , depth, infrared) is proven to be indisputably beneficial [41, 29, 49, 25]. This trend equally applies to the intelligent agents in visionbased Reinforcement Learning (RL), in which multi-modal inputs can also promote decision robustness [39, 19, 1, 11, 31]. For instance, an autonomous-driving agent taking RGB frames solely as input may frequently suffer from extreme light conditions, as shown in Fig. 1(a). Nevertheless, combining additional sensory inputs such as event signals coming from a neuromorphic event camera [27] can effectively alleviate these problems, enabling a more comprehensive realization of traffic status [5, 32, 12].

RGB

Critic

𝑞 𝑀1

(d) Local modality-customized Value Estimation (LVE)

𝑞𝑡

𝑙

𝑞 𝑀2

𝑤 𝑀1

𝑤 𝑀2

𝑟 𝑡

𝑞 𝑡

𝑔

Critic

(c) Global Value Estimation (GVE)

Feature-level

Fusion

𝑎𝑡

𝑞𝑡

𝑙

𝑟 𝑡

𝑎𝑡

𝑞 𝑡

𝑔

Environment

Actor

RGB latent features

Events latent features

Feedback

Forward

Contribution

assignment

(b) Clear RGB frames vs. Noisy event signals

(a) Under-exposed RGB frames vs. HDR event signals

Value-level Fusion

Fusion

RGB

Events

Contribution

RGB Events

RGB

Events

RGB Events

Contribution

Events

Critic

RGB

Events

RGB

Events

Global modality

$

$

$

Despite the abundance of fruitful studies in the field of multi-modal visual perception, a majority of them focus on traditional static learning tasks. In contrast, the dynamic and unlabeled nature of RL renders the development of multi-modal agents exceedingly difficult [6]. To reconcile multisensory data, existing methods [18, 36, 6, 22, 40] train the agents with fused modality features and form a mixed global modality, as shown in Fig.1(c). The policy is then learned via Global Value Estimation (GVE), in which the global modality feature is responsible for environment description and then paired with action to compute a single Q-value. Although being an effective and widely adopted paradigm, utilizing GVE alone may overlook the distinct task-related contributions of different modalities. Specifically, due to the diverse attributes of sensors, single modalities may not contribute equally under multiple environmental conditions ( e.g. , the cases given in Fig. 1 (a) and (b)). The feature-level fusion process in GVE, no matter how sophisticated, inevitably disregards the unique value of each sensory data since only a single critic is shared over multiple modalities. As a result, negative interference between modalities may occur and compromise learning performance.

Based on the above analysis, we propose Local modality-customized Value Estimation (LVE) , as illustrated in Fig. 1 (d). Different from feature-level fusion, LVE learns a single policy with distinct, per-modality value calculation, forming a value-level fusion paradigm through a tailor-designed Qvalue weighting process. As a result, the modality-specific contributions can be explicitly estimated, which promotes policy flexibility. In addition, while LVE is intrinsically a better alternative than GVE, the two paradigms are not competitive but complementary. Considering this, we further develop a task-contextual re-fusion process, which utilizes an efficient fusing network guided directly by the task reward to reach a task-level balance between LVE and GVE. The collaboration of the above components eventually forms a Hierarchical Adaptive Value Estimation (HAVE) framework for multi-modal vision-based RL, yielding a potent policy that can flexibly leverage the unique strengths of each modality while profiting from the comprehensive information from all modalities. We specifically highlight the potency of our approach to the challenging autonomous driving task. In particular, we explicitly consider the neuromorphic event camera [27] signals that capture the motion information of the environment, which well-suites our task but has not yet been explored by existing multi-modal RL algorithms.

In summary, the contributions of our work are three-fold: 1) We design a novel hierarchical adaptive value estimation (HAVE) framework for multi-modal vision-based RL. HAVE distinctively features a local modality-customized value estimation (LVE) paradigm to enable optimized reward allocation based on modality importance. 2) We develop a task-contextual re-fusion process to merge the profi-

ciencies of LVE and GVE, allowing HAVE to benefit from both particularized and unified modality values. 3) Our approach achieves state-of-the-art performance on challenging autonomous driving tasks. As the first multi-modal RL algorithm considering event camera signals as one of the input modalities, our approach exhibits superior performance under various environmental conditions, demonstrating its potential as an effective multi-modal vision-based RL solution.

2 Related Work

Vision-based RL aims to train agents that receive raw image-based observations from environments for decision-making. Compared to state-based RL, vision-based RL has found significant use in many practical tasks, from video game playing [34] to robotic manipulation [21]. However, learning policy directly from such high-dimensional input is challenging. To tackle the problem, a considerable amount of works have been developed, including 1) applying data augmentation to increase the diversity of samples [50, 24, 30], 2) introducing auxiliary tasks such as contrastive loss [23, 54, 2], 3) pretraining an encoder to improve the representational ability [51, 28, 42], and 4) modeling environment dynamics in the latent space [17, 16, 38]. Although most vision-based RL methods adopt RGB camera frames as inputs, some recent works have also started to explore new sensors and data formats for RL, such as event cameras [46, 47] which captures fast and asynchronous light changes with high temporal precision.

Multi-modal Visual Learning has been extensively studied in the field of computer vision [3]. With the development of sensor technologies, it becomes effortless to acquire sufficient data from complemented visual modalities ( e.g. , RGB, infrared, depth, and event signals). As a result, many multi-modal learning methods are proposed for traditional visual tasks, such as object detection [29] and segmentation [10, 25]. For multi-modal RL, the agent's observation space is modified to include all modalities. Recent works have started to focus on multi-modal vision-based RL due to its improved effectiveness compared with using only single-modality data. Chen et al. propose a multi-modal state-space model trained with mutual information lower-bound to promote the consistency between the latent codes of each modality [6]. A fusion network is proposed by Khalil et al. to produce accurate joint multi-modal perception and motion prediction for autonomous driving [22]. Ma et al. propose a multi-modal RL approach that focuses on modality alignment and importance enhancement [31]. There are also multi-modal RL methods designed for other tasks such as robot control [4] and dialog system [33, 53]. Despite their distinct technical details, most of the existing methods perform a feature-level fusion of modalities and ignore the properties of RL itself. In fact, policy learning depends on the value estimation of the critic. Consequently, we view the multimodal visual RL problem from a novel adaptive value estimation perspective. Instead of concentrating solely on robust global modality features, our approach synergistically balances individual modalities' contributions, leading to a more equitable and efficient value allocation. Leveraging the task-contextual re-fusion mechanism, our method further capitalizes on both feature-based and value-based fusion paradigms, resulting in a more robust policy.

3 Methodology

3.1 Preliminaries

Problem Definition We formulate the task of multi-modal vision-based RL as a Markov Decision Process (MDP) [44] with multiple observations, which is defined by a tuple ( S , A , P , R , γ ) , where S = ∏ d i =1 O M i is the joint observation space of d modalities and O M i is the observation space of modality i . A is all possible actions the agent can take. P : S × A × S → [0 , 1] is the transition probability function, P ( o t +1 | o t , a t ) denotes the probability of transitioning from joint observation o t = ( o M 1 t , o M 2 t , . . . , o M d t ) at time step t to the next joint observation o t +1 = ( o M 1 t +1 , o M 2 t +1 , . . . , o M d t +1 ) after taking action a t from a policy function π .

Soft Actor-Critic (SAC) Our approach is built on SAC [14, 15], which goal is to learn a policy that maximizes the expected cumulative reward while maintaining exploration by encouraging diverse actions. In SAC, the objective function is given by introducing a policy entropy term:

∓

B

F

u

o

n

s

i

⊆

¬

c

C

r

i

t

∓

B

G

V

E

H

A

ΓςΓUΓLΓWΓLΓF

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]↖

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]ˇ

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

∓Sglyph[circledivide]glyph[negationslash]

∓Sglyph[circledivide]glyph[triangleinv]

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

contexualglyph[suppress]

∓⊤ glyph[circledivide]glyph[negationslash]

∓⊤ glyph[circledivide]glyph[triangleinv]

(a)

(b)glyph[suppress]

(c)glyph[suppress]

glyph[circledot]⊃ glyph[circledot]ˇ

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]ˇ

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]↖

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]ˇ

Featurglyph[suppress]

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<223>

glyph[circledot]⊃glyph[circledot]↖

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]↖

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

±GLYPH<132>

ΓςΓUΓLΓWΓLΓF

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<223>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<223>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<223>

⊕GLYPH<226>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

·Æ

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<223>

)

glyph[circledivide]GLYPH<222>

glyph[circledivide]GLYPH<228>

glyph[circledivide]Ø

⊕GLYPH<226>

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

·Æ

∓M glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<223>

)

glyph[circledivide]˘

glyph[circledot]GLYPH<212>

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘

glyph[circledot]GLYPH<212>

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<220>

glyph[circledivide]GLYPH<215>

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘

glyph[circledot]GLYPH<212>

glyph[circledivide]GLYPH<220>

glyph[circledivide]GLYPH<215>

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<218>

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<231>

glyph[circledivide]˘ glyph[circledot]ˇ

glyph[Ifractur]∓ ·Æ

glyph[circledivide]GLYPH<231>

glyph[circledivide]GLYPH<220>

glyph[circledivide]GLYPH<215>

L

E

V

V

E

where Q is the value function, and α is a temperature parameter that controls the trade-off between exploration and exploitation. The value function is trained using the Bellman equation and a soft Q-function update:

where o t and a t are sampled from the replay buffer D and γ is the discount factor. V ( o t +1 ) is the soft state value function, defined as:

where ¯ Q denotes an exponential moving average of the critic network Q and ˜ a t +1 comes from the current policy.

Event Cameras and Representations Despite RGB frames, another major modality that we use to evaluate our approach is the event signals generated by a neuromorphic event camera [27]. Each pixel in the event camera outputs a positive/negative event signal whenever the log light intensity of that pixel has increased/decreased by a constant threshold. The event signals have an extremely high dynamic range (up to 120 dB) and can reach a high temporal resolution in the order of µ s. Therefore, they are able to capture the missing motion clues that are missed in the RGB frames for many visual tasks such as autonomous driving and robot navigation. To utilize event signals, we use the stacking based on time (SBT) [48] representation that splits the event sequence into fixed temporal bins, forming an event frame representation with multiple channels similar to RGB frames.

3.2 Global Value Estimation

In Global Value Estimation (GVE), a total of d observation encoders are used to extract observation features f M i t for each modality observation o M i t at time step t . A global modality feature f g t is then calculated by fusing all modality features. For simplicity, we directly concatenate f M i t ( i = 1 , 2 , . . . , d ) along the channel dimension. Given f g t , GVE first computes the value estimation q g t = Q θ g ( f g t , a t ) using a single global critic network Q θ g , as shown in Fig. 2(a). Following the SAC algorithm, the target value y g t is then calculated as:

where R ( o t , a t ) is the reward returned by the environment. The soft state value function V ( f g t +1 ) is denoted as:

However, since using a single value function Q θ g shared over all modalities, it is hard to dynamically balance the relative contribution of each f M i t under complex environmental conditions.

⊆

∅

∓

B

∓

glyph[owner]

glyph[circledot]

•

⊆

¬

∓

B

⊆

¬

∓

B

C

o

n

t

r

i

b

u

g

n

e

A

s

i

m

⊆

∅

⊆

∅

∓

glyph[owner]

glyph[circledot]

•

⊆

¬

∓

B

t

⊆

¬

∓

B

⊆

∅

⊆

∅

⊆

glyph[Rfractur]

∓

glyph[owner]

∓

B

∓

∃

⊆

glyph[Ifractur]

glyph[circledivide]

ℵ

glyph[circledivide]

glyph[triangleinv]

∓

∃

a

x

S

o

m

f

t

M

a

t

u

l

∓

B

e

S

c

a

l

⊆

glyph[Rfractur]

∓

∃

∓

B

⊆

glyph[Ifractur]

glyph[circledivide]

ℵ

glyph[circledivide]

glyph[triangleinv]

⊆

¬

k

a

s

T

∓

B

E

V

G

R

e

-

⊆

∅

-

⊆

¬

o

n

f

u

s

i

∓

B

L

E

V

glyph[circledivide]

ℵ

glyph[circledivide]

glyph[triangleinv]

⊆

¬

o

k

s

H

y

p

e

r

n

t

w

∓

B

3.3 Local Modality Customized Value Estimation

The objective of Local modality-customized Value Estimation (LVE) is to effectively differentiate and quantify the unique contribution of each modality, thereby facilitating a more granular and modality-aware decision-making process. Specifically, in LVE, d individual value functions Q θ M 1 , Q θ M 2 , ..., Q θ Md are setup instead of global-only functions in GVE. These value functions share the same network architecture but are learned separately. To facilitate value decomposition, an assignment network is designed to assimilate all estimated values and re-calibrate them according to the collective environmental condition portrayed by modality features.

Value Inference: We assume that the locally-aggregated total value q l t can be approximately decomposed into a linear combination of the separate value functions across different modalities under a shared policy:

where w M i denotes the contribution weight of modality M i to the total value. The value function Q θ Mi for each modality processes the current individual modal observation feature f M i t along with the current action a t , generating the estimated action values at each time step. Each Q θ Mi is learned by backpropagating gradients from the Q-learning rule. Similarly, the total target value y l t can be defined as:

The soft state value function V ( f M 1 t +1 , ..., f M d t +1 ) is then defined as:

where v M i t is the contribution weight of modality M i to the the estimated Q-value from ¯ Q θ Mi at the next time. ˜ a t +1 ∼ π ( ·| f g t +1 ) comes from the current policy according to the global modality feature f g t +1 since there is only one policy network for the multi-modal decision task.

Contribution Assignment: Given value decomposition in Eq. 6 and Eq. 7, a critical next step is to calculate accurate modality weights w M i t and v M i t to ensure precise total value estimation. To achieve this, the contribution assignment process involves modality interactions through an assignment network based on the attention mechanism. Taking the calculation of w M i t as an example, the global modality feature f g t is used as a bridge to ensure information flow between all individual modalities. As demonstrated in Fig. 2(b), two fully connected (FC) layers with parameters W q and W k are used to project both f M i t and f g t into a p -dimensional common latent subspace. Then w M i t can be obtained through a softmax function as:

where √ p is used for scaling to prevent vanishing gradients [45]. The computation of v M i t +1 follows a similar derivation, which is omitted here for brevity.

By assigning the value function in a modality-customized manner, LVE adeptly manages multimodal sensory inputs, enhancing model efficiency and interpretability. Our approach readily adapts to complex environmental scenarios, accommodating varying modality importance levels.

3.4 Task-Contextual Re-fusion

Feature-based vs. Value-based Fusion From the details of GVE and LVE, we see that the main difference distinguishing them is their fusion principle (feature-based vs. value-based). Intuitively, the two paradigms are not competitive but complementary. Specifically, in the context of RL, valuebased fusion offers significant advantages over feature-based fusion since the former is more related to the actual decision. In the meantime, the feature-based fusion method can use its collective modality value to offer a value-based one with a sturdy global reference, mitigating inaccuracies caused by input noise. Therefore, we further develop task-contextual re-fusion for a subsequent re-fuse of GVE with LVE in the reward/task level. The idea is to directly bridge the two fusion mechanisms

based on environmental reward feedback, allowing the task to determine which paradigm better suits the current situation.

Details of the Re-fusion Process Given the estimated value q g t and q l t obtained from GVE and LVE, respectively, we adopt a dynamic fusion mechanism with a re-fusion network H to fuse q g t and q l t . Specifically, H consists of a series of FC layers and hypernetworks. Each hypernetwork consists of a single linear layer, which receives the global modality feature f g t as input and produces a weight matrix or a bias vector for a single FC layer in H . To fuse q g t and q l t , f g t is first sent into all hypernetworks to obtain the parameters of the FC layers in H , then q g t and q l t are sent into H to get the final total value H ( q g t , q l t ) . The final target value can be obtained using a similar way, wherein the estimated Q-value from the target Q-network at the next time is processed by another mixing network H ′ . For brevity, we denote the this target value as H ′ ( y g t , y l t ) .

3.5 Learning Framework

Coping GVE with LVE and task-contextual re-fusion, our approach forms a unified framework, which we name as Hierarchical Adaptive Value Estimation (HAVE) for multi-modal vision-based RL. In this section, we elaborate on the training details of our HA VE framework.

Policy Evaluation with HAVE For the training of policy evaluation, we minimize the temporal difference (TD) error between the predicted value H ( q g t , q l t ) , and the target value H ′ ( y g t , y l t ) following the SAC algorithm:

where D denotes the replay buffer.

Policy Improvement The policy evaluation described above can cover reasonable Q-values to help find better policies. In the policy improvement step, we leverage the values obtained from GVE, LVE, and the task-contextual re-fusion process to update the policy π . To be specific, the goal is to maximize the expected cumulative reward by selecting actions that have the highest combined value estimates:

L π = E a t ∼ π [ H ( q g t , q l t ) -α log π ( a t | f g t ) ] . (11) After identifying the optimal action under the improved policy, we can update the policy parameters by backpropagating gradients.

Auxiliary Losses Besides policy evaluation and improvement, we also learn to predict the rewards and next latent states as in DeepMDP [13] to assist representation learning, providing a latent state space that is consistent with environmental dynamics:

where P g and P M i ( i = 1 , 2 , . . . , d ) are transition networks for the global and individual modalities, respectively. R g is the reward prediction network. All these networks are formed by FC layers and a more detailed architecture description can be found in the supplementary material.

Overall Training Objective Given the above training objectives, the overall loss of HA VE is finally defined by:

By iteratively performing policy evaluation and improvement steps, HAVE can effectively exploit the strengths of both global and distinct value estimates, ultimately converging to an improved policy that can handle the challenges posed by a broad range of environmental modalities.

3.6 Further Analyses

Our hierarchical adaptive value estimation framework for multi-modal RL process offers several key benefits, which we list as follows:

Remark 1 (Prevention of Modality Dominance): The mechanism of LVE and task-contextual refusion prevents one modality from dominating the others, thereby avoiding the issue of modality collapse.

To realize this, consider that the mixing network H in the task-contextual re-fusion performs fixed non-linear transformations on both q g t and q l t given f g t , where q l t is formed by the convex combination of individual modality values (Eq. 6) as in LVE. Therefore, we have:

where sign ( · ) is the real sign function, q M i t = Q θ Mi ( f M i t , a t ) is the individual modality value and ∂q l t ∂q Mi t = w M i t > 0 are the modality weights. Eq. 14 indicates that the contributions of all modalities are either increasing or decreasing together during the learning process, which prevents opposite gradient values between modalities that enhance some of them while suppressing others. In addition, the calculation of modality weights w M i t based on softmax global-individual feature similarity (Eq. 9) further prevents modality collapse caused by persistent zero weights.

Remark 2 (Pareto Optimality of Modalities): Assuming continuous action space, the global optimal action produced by our approach achieves a Pareto optimum across individual modalities.

Suppose the optimal action given some fixed modality features is a ∗ , which implies ∂ H ( q g t ,q l t ) ∂a ∗ = 0 . Combining Eq. 6 with the chain rule, we have:

that ∑ d i =1 w M i t ∂q Mi t ∂a ∗ = 0 for all i = 1 , 2 , . . . , d , which suggests that at the point a ∗ , the positive weighted sum of the gradients of individual modalities' values is zero. This implies that there is no alternative action assignment in the vicinity of a ∗ that can increase the value for any individual modality without decreasing the value for some other modality. Such condition forms a Pareto optimum across the modalities, which indicates the resources (modality values) are allocated in the most efficient way possible [35].

Considering the non-linear structure of H , ∂ H ( q g t ,q l t ) ∂q l t is unlikely to be zero for all q l t . This implies

4 Experiments

4.1 Settings

Environments To evaluate our approach under realistic and challenging multi-modal environments, we employ the CARLA simulator [8], which is a widely used open-source platform for autonomous driving research. CARLA provides a rich and realistic urban environment to evaluate autonomous driving agents in various traffic scenarios. Three distinct modalities are adopted: 1) RGB frames, 2) event signals generated by CARLA's synthetic event-camera simulator, and 3) per-pixel depth frames. Eight different weather settings are used to provide thorough coverage of different environmental conditions. The action space consists of continuous control actions, such as steering, acceleration, and braking. Similar to [52, 9], the reward function is designed to encourage the agent to maintain a safe distance from 20 other moving vehicles and obstacles, and drive as far as possible along the highway of CARLA's Town04 map in 1000 time steps. We use the single camera view setting on the vehicle's roof with 60-degree views.

Implementation Details Our approach is implemented based on SAC [14, 15] and DeepMDP [13]. The same encoder network architecture and training hyperparameters are adopted for all comparative methods. Following common practices [50, 23], we convert each modality data into its corresponding image-based representations and stack several consecutive images to infer temporal information. The spatial resolution of the input images is 128 × 128 and the channel numbers of RGB, event and depth frames are 3, 5 and 1, respectively. All methods are trained for 120k frames across 5 random seeds to report the mean and standard deviation of the rewards. We evaluate the performance of each approach in terms of driving episode reward and distance. Other details are provided in the supplementary material.

±

±

4.2 Comparison with State of the Art

Wecompare our method with a variety of methods, including the SAC [15] baseline, DeepMDP [13], DrQ [50], TransFuser [7], and EFNet [43]. For RL methods SAC, DrQ and DeepMDP, we directly concatenate the features of the different modalities as the input for subsequent value and policy learning. For TransFuser and EFNet which are designed for traditional visual tasks, we only adopt their advanced modality fusion modules and keep DeepMDP as the RL algorithm for a fair comparison.

Results of Two Modalities We first evaluate different methods under two modality inputs (RGB frames and event signals) in Table 1. The results show that our method achieves the highest episode reward and driving distance under all eight weather conditions. DrQ and DeepMDP perform better than the SAC baseline with limited improvement, showing that the feature-fusion based GVE paradigm, together with simple feature concatenation, cannot fully extract the expressive power of each modality. The improved performance of TransFuser and EFNet over DeepMDP reflects the importance of the advanced modality feature fusion mechanism. However, they are still inferior to ours-LVE, which uses the proposed LVE paradigm to explicitly assign modality contributions weights with value-based fusion. The results indicate that the key to multi-modal RL is to consider the suitability of individual modalities. Finally, by using the task-contextual re-fusion process to integrate GVE and LVE, our full method achieve superior performance even with simple feature concatenation fusion, proving the effectiveness of our task-driven hierarchical design.

Results of Multiple Modalities We then evaluate our method on three modalities (RGB frames, event signals, and depth frames). Although TransFuser and EFNet technically possible to scale both methods to accommodate more than two modalities, such an adaptation would require significant modifications to the implementation, along with a quadratic increase in computational complexity due to cross attention mechanism. Thus we directly

compare our method with SAC, DrQ and DeepMDP. The training curves under two weather conditions are demonstrated in Fig. 3, which also show the advantage of our methods. Additional experiment results can be found in the supplementary material.

4.3 Performance Analysis

Ablation Studies To systematically evaluate the effectiveness of the proposed HAVE approach and its individual components, we conduct a series of ablation experiments on the CARLA benchmark that trains agents using: 1) single modality data with RGB frames or event signals, 2) GVE with feature concatenation, 3) our proposed LVE paradigm, and 4) our full method. The training curves and testing performance are shown in Fig. 4 and Tab. 2, respectively. First, we observe that the indi-

±

±

±

±

±

ClearNight

CloudyNight

HardRainNight

WetNight

Episodereturn

250

350

300

200

150

RGB

LVE

HAVE

Events

GVE

Episodereturn

200

150

300

250

RGB

Events

GVE

LVE

HAVE

Episode return

150

300

200

250

=RGB

Events

GVE

LVE

HAVE

Episode return

300

250

200

150

RGB

LVE

HAVE

Events

GVE

100

100

100

100

50

50

50

50

0

7

0

0

0

0.00

0.020.040.060.080.10

0.12

0.00

0.020.040.060.080.10

0.12

0.00

0.020.040.060.080.10

0.12

0.00

0.020.040.060.080.10

0.12

Number of frames (×106)

Number of frames (×106)

Number of frames (×106)

Number of frames (×106)

ClearNoon

CloudyNoon

HardRainNoon

WetNoon

300

RGB

Events

250

= RGB

Events

400

RGB

Events

350

= RGB

Events

Episodereturn

200

250

GVE

LVE

HAVE

Episode return

200

GVE

LVE

HAVE

Episodereturn

300

GVE

LVE

HAVE

Episodereturn

250

300

GVE

LVE

HAVE

150

200

150

200

150

100

100

100

50

50

100

50

0

0

0

0

0.00

0.020.040.060.080.10

0.12

0.00

0.020.040.060.080.10

0.12

0.00

0.020.040.060.080.10

0.12

0.00

0.020.040.060.080.10

0.12

Number of frames (×106)

Number of frames (×106)

Number of frames (×106)

Number of frames (×106)

±

±

±

±

±

±

±

±

A

B

145

C

A

B

C

RGB frames

Event signals

RGB frames

Event signals

A

B

C

A

B

C

Modality Weights (RGB frames)

ClearNight

(RGBframes)

ClearNoon

1.00

1.00

0.95

0.95

Modality Weights(

0.90

0.90

0.85

0.80

0.85

0.75

0

100

200

0

200

400

Time step

Time step

vidual modality is vulnerable to failure under extreme weather conditions. For example, the episode reward trained with RGB frame under ClearNight weather is pretty poor as shown in Fig. 4. Second, directly fusing the two modalities with GVE generally improves performance. However, the performance gain is limited, and in some cases even close to the results of using a single modality ( e.g. , in WetNight and ClearNoon). This might be caused by the heterogeneity of multi-modal data, which leads to difficulty in determining the target reward. Third, in most cases, LVE performs significantly higher than single modality or GVE-based agents, showing better cooperation of different modality data. Finally, our full method with task-contextual re-fusion clearly outperforms all other models thanks to the synergistic interplay of GVE and LVE at the reward/task-level.

Visualization To further obtain an intuitive understanding of our approach, we visualize the modality weights obtained by HAVE in Fig. 5 using RGB frames and event signals under two different weather conditions. The following observations can be made: 1) at the beginning, all vehicles have zero initial speed. As a result, no valid event signal is generated and the weights of RGB modality are nearly 1.0. 2) After speeding up, the weights of event signals are continuously increasing, especially when there are many moving vehicles ahead. 3) However, when the agent is facing complex background (such as road fences and clear roads without close vehicles), the event signals mainly consist of background noise, and the weights of RGB frames will raise again. The varying modality weights show that our approach can indeed adjust modality contributions under different situations, which serves as a key advantage over previous multi-modal RL methods.

5 Conclusion and Limitation

We have studied the representational capacity of value functions in multi-modal vision-based RL problems. We hypothesize that the limitation of feature-level fusion methods may come from unclear contributions for different modalities under a single value function. To mitigate this, we have presented a novel Hierarchical Adaptive Value Estimation (HAVE) framework to reconcile both feature-level and value-level fusion in a task/reward-driven manner. Our approach represents one of the first explorations of modality-specific and hierarchical value estimation for multi-modal visionbased RL tasks. Extensive experiment results demonstrate the effectiveness of our approach. However, one limitation of our work is that we mainly focus on the value estimation of multiple visual modalities, while the effectiveness of other forms of modalities ( e.g. , audio, text) is not verified. In our future work, we will consider utilizing both visual and other modalities, forming a more generalized multi-modal RL framework.

References

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In Proceedings of the International Conference on Learning Representations , 2022.

Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Mtibaa. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. Visual Computer , 2022.

Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, and Dhruv Batra. Embodied multimodal multitask learning. In Proceedings of the International Joint Conference on Artificial Intelligence , 2020.

Guang Chen, Hu Cao, Jorg Conradt, Huajin Tang, Florian Rohrbein, and Alois Knoll. Event-based neuromorphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine , 2020.

Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. In IEEE Proceedings of the International Conference on Robotics and Automation . IEEE, 2021.

Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning . PMLR, 2017.

Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Proceedings of the International Conference on Machine Learning , 2021.

Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems , 2020.

Wen Fu, Yanjie Li, Zhaohui Ye, and Qi Liu. Decision making for autonomous driving via multimodal transformer and deep reinforcement learning. In IEEE International Conference on Real-time Computing and Robotics , 2022.

Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters , 2021.

Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In Proceedings of the International Conference on Machine Learning , 2019.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning , 2018.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 , 2019.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning , 2019.

Zhiyu Huang, Chen Lv, Yang Xing, and Jingda Wu. Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding. IEEE Sensors Journal , 2020.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094 , 2022.

Yang Jiao, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Xiaolin Wei, Lin Ma, and Yu-Gang Jiang. Msmdfusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection. arXiv preprint arXiv:2209.03102 , 2022.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv:1806.10293 , 2018.

Yasser H Khalil and Hussein T Mouftah. Exploiting multi-modal fusion for urban autonomous driving using latent deep reinforcement learning. IEEE Transactions on Vehicular Technology , 2022.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In Proceedings of the International Conference on Machine Learning , 2020.

Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Proceedings of the Advances in Neural Information Processing Systems , 2020.

Jiale Li, Hang Dai, Hao Han, and Yong Ding. Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving. arXiv preprint arXiv:2303.08600 , 2023.

Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128 × 128 120 db 15 µ s latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits , 2008.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Proceedings of the Advances in Neural Information Processing Systems , 2021.

Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning. arXiv preprint arXiv:2210.04561 , 2022.

Jinming Ma, Feng Wu, Yingfeng Chen, Xianpeng Ji, and Yu Ding. Effective multimodal reinforcement learning with modality alignment and importance enhancement. arXiv preprint arXiv:2302.09318 , 2023.

Ana I. Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Eventbased vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2017.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Martin J Osborne and Ariel Rubinstein. A Course in Game Theory . The MIT Press. MIT Press, London, England, July 1994.

Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical textconditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Proceedings of the Conference on Robot Learning . PMLR, 2023.

Afagh Mehri Shervedani, Siyu Li, Natawut Monaikul, Bahareh Abbasi, Barbara Di Eugenio, and Milos Zefran. Multimodal reinforcement learning for robots collaborating with humans. arXiv preprint arXiv:2303.07265 , 2023.

Hailuo Song, Ao Li, Tong Wang, and Minghui Wang. Multimodal deep reinforcement learning with auxiliary task for obstacle avoidance of indoor mobile robot. Sensors , 2021.

Mengke Song, Wenfeng Song, Guowei Yang, and Chenglizhao Chen. Improving rgb-d salient object detection via modality-aware decoder. IEEE Transactions on Image Processing , 2022.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the International Conference on Machine Learning . PMLR, 2021.

Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention. In Proceedings of European Conference on Computer Vision, year=2022 .

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the International Conference on Neural Information Processing Systems , 2017.

Sai Vemprala, Sami Mian, and Ashish Kapoor. Representation learning for event-based visuomotor policies. In Proceedings of the Advances in Neural Information Processing Systems , 2021.

Celyn Walters and Simon Hadfield. Ceril: Continuous event-based reinforcement learning. arXiv preprint arXiv:2302.07667 , 2023.

Lin Wang, I S Mohammad Mostafavi, Yo-Sung Ho, and Kuk-Jin Yoon. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Zongwei Wu, Guillaume Allibert, Fabrice Meriaudeau, Chao Ma, and Cédric Demonceaux. Hidanet: Rgb-d salient object detection via hierarchical depth awareness. arXiv preprint arXiv:2301.07405 , 2023.

Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In Proceedings of the International Conference on Learning Representations , 2020.

Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pretrained image encoder for generalizable visual reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems , 2022.

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742 , 2020.

Jiaping Zhang, Tiancheng Zhao, and Zhou Yu. Multimodal hierarchical reinforcement learning policy for task-oriented visual dialog. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue , 2018.

Yue Zhao, Chenzhuang Du, Hang Zhao, and Tiejun Li. Intrinsically motivated self-supervised learning in reinforcement learning. In Proceedings of the International Conference on Robotics and Automation . IEEE, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new vision-based reinforcement learning algorithm for autonomous driving scenarios. In previous feature fusion methods, a single critic could amplify dominant features and obscure other features. To address this issue, the authors propose a local critic for each feature and combine them into a global critic using attention fusion. Additionally, the authors introduce a task-contextual re-fusion component to rebalance the estimator. The algorithm achieves promising performance on the CARLA driving task.\n\nSTRENGTHS:\n1. The writing in the paper is easy to follow, and the figures are clear and understandable.\n2. The proposed algorithm has a simple and effective structure. The improvements made to existing methods in the framework section are intuitive.\n3. The argumentation in the \"Further Analyses\" section of the paper is thorough and provides a good explanation for the potential sources of performance improvement brought by the algorithm.\n\nWEAKNESSES:\n1. The decomposition of the Q-network lacks novelty. Although this paper focuses on vision-based RL, the approach of using a separate critic for each modality and obtaining a global critic using self-attention is similar to some existing methods in the field of multi-agent RL[1]. The paper does not adequately highlight the differences in information processing for vision-based tasks compared to other types of input.\n\n2. The central contribution of the paper lies in the proposed framework. However, apart from the front-end modality fusion and task-specific re-balance, all the content mentioned in section 3.5 appears to be supplementary to existing research[2][3] rather than original improvements. Therefore, I believe there is a lack of substantial contribution in the paper.\n\nref\n[1] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning\n[2] Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\n[3] Deepmdp: Learning continuous latent space models for representation learning\n\nQUESTIONS:\nIn Section 4.2, I noticed that the authors mentioned, \"Since TransFuser and EFNet only support two modalities, we directly compare our method with SAC, DrQ, and DeepMDP.\" This suggests that these two baseline algorithms are not suitable for the task environment used in the current paper. Then why use them as strong baselines to compare?\n\nLIMITATIONS:\nThe experimental setup is not sufficiently fair. Although the baselines used for comparison include newer multimodal vision algorithms, they were originally designed for traditional visual tasks. In the tasks used for algorithm validation in this paper, there is a lack of a strong baseline that specifically demonstrates the superiority of the proposed algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Local modality-customized Value Estimation (LVE) paradigm, which dynamically estimates and adjusts the importance weight of each modality from the perspective of value-level to compensate for the multi-modal vision-based RL methods that usually use fused modality features for learning Global Value Estimation (GVE), which may be insufficient in policy learning. Furthermore, the authors propose a task-contextual re-fusion procedure to achieve a task-level rebalancing of estimates from feature and value levels. Experimental CARLA benchmark results show the improvement of multi-modal vision-based autonomous driving RL tasks.\n\nSTRENGTHS:\n- The originality and novelty of the paper are solid. Inspired by some ideas in algorithms in Multi-Agent RL, an ingenious weighted local value estimation method suitable for multi-modal RL is proposed. And a task contextual re-fusion process is designed to combine better global value estimation and weighted local value estimation to obtain better multi-modal value estimation.\n- The paper is written with high quality. The writing structure is clear and organized, the sentences are fluent and smooth without grammatical errors, and the language style is simple and easy to understand. Although there is no rigorous theoretical analysis, it gives some interesting insights from mathematics. In addition, the code and appendices are well documented and provide useful analysis and explanation.\n- The significance of the paper is fine. Developing better algorithms to solve multi-modal RL tasks is an important research topic.\n\nWEAKNESSES:\n- The experiments for this paper are still not adequate. The entire manuscript has always only conducted experiments on an autonomous driving environment CARLA, and it is difficult to evaluate whether HAVE is suitable for other broader multi-modal RL environments (such as simulated robot control). In addition, this work only conducts experiments on tasks of multiple visual modalities and does not consider more common modal inputs such as text and voice.\n- The paper lacks further theoretical analysis. Although two interesting remarks are in Section 3.6 of the manuscript, the theoretical insight obtained is still relatively shallow. Lack of more in-depth theoretical analyses, such as the overall convergence of the algorithm and whether the algorithm optimizes some lower bounds under the multi-modal RL setting.\n\nQUESTIONS:\n- The baselines compared in this paper are still not representative and pertinent. This paper mainly focuses on the multi-modal RL task of visual input, but there is no comparison with the classic Vision-based RL work, and there are more suitable baselines than SAC, DrQ, and DeepMDP. For example:\n  - Visual Reinforcement Learning with Imagined Goals, NeurIPS 2018\n  - Improving Sample Efficiency in Model-Free Reinforcement Learning from Images, AAAI 2021\n  - Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation, NeurIPS 2021\n- Observing Figure 3 and Figure 4, it seems that the training curves have not yet converged. If the training process is extended, would HAVE be surpassed by other comparative methods?\n- Why do TransFuser and EFNet only support two modalities, as mentioned in article 4.2?\n- Would using neuromorphic event signals lead to totally fair comparisons? The algorithm's performance may be related to the peculiarity of the modality itself. For example, TransFuser may have its advantages when using the Bird’s Eye View (BEV) modality, but it may be challenging to have a good command of neuromorphic event signals\n- Different local areas of each modality could also have different importance. Can HAVE balance the value of this finer-grained \"modality\"? And how?\n\nLIMITATIONS:\nThe authors explained the limitations of their work well. For example, this work only conducts experiments on RL tasks of multiple visual modalities and does not consider more extensive multimodal input such as text and voice. In addition, the authors explained their potential negative social impact in the appendix. I do not see an obvious negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a fusion scheme for multi-modal RL, with a particular focus on fusing the modalities in estimating the value function. The authors propose a combination of global value estimation, local value estimation and another conditioned fusion of the global and local value estimation. The approach is evaluated in particular on fusing RGB and event-based data in an autonomous driving scenario with the CARLA simulator.\n\nSTRENGTHS:\nThe paper proposes a novel scheme of \"value level fusion\" that is new to me.\n\nWEAKNESSES:\n- The authors propose a very complex scheme for fusing the q-value estimate based on the different modalities, having GVE, LVE and then another fusing of those two with a hyper network. Especially since the policy is selecting actions based on a simple concatenation of feature vectors of both modalities.\n\n- There is a very high spread on the results which makes me wonder how significant the improvement is of the HAVE approach. Also, from Fig 3 and 4, it seems that the other methods are not necessarily converged yet. It would be interesting to see how the curves evolve if training would continue.\n\n- All experiments are done on this single environment with a custom reward function. It would be more convincing if the approach was demonstrated on one extra environment, i.e. it is hard to assess how significant it is to drive for instance 42m further than another method (+/- 52m) to take the first line of Table 1 as an example.\n\nQUESTIONS:\n- Why would the policy benefit from all the fusion in the value function if it is a neural network only taking the global modality feature as input?\n\n- From the results in Fig 5 it seems bulk of the contribution of the value is from the RGB frame, as its weight never goes below 0.75?\n\n- The only difference between the proposed approach and the baselines is the value function estimation. I would wonder whether the architecture yields a better value function (i.e. closer to the ground truth value) faster, which yields the improved training of the policy. However, if in the long run, any value iteration converges to the true value function, it should yield the same policy, no? I would be interesting to inspect the MSE of the value function compared a ground truth value estimate for the different algorithms over time, to see if this is indeed the case.\n\nLIMITATIONS:\nThe authors only mention the limitation to one set of modalities in their work. I would make this broader to note that the approach is only tested on this one environment.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a Hierarchical adaptive value estimation (HAVE) framework for multi-modal (RGB, event, depth) visual reinforcement learning. Firstly, a modality-specific value function learning process is proposed, and an assignment module is proposed to weight different modality. Second, a re-fusion is proposed to futher combine global feature based value and modality-specific based value. Method is evaluated on CARLA simulation environment. Various baselines, including SAC, DrQ, DeepMDP, etc are compared and the results show the proposed HAVE perform better under different weather conditions.\n\nSTRENGTHS:\nThe decomposition of global value function into weighted combination of modality value function is novel and well-motivated by the observation that simple feature fusion might cause dominant modality behaviour. The weight assignment module is well designed and also show meaningful behaviour in Figure 5. The re-fuse further makes the method can enjoy both the value and feature fusion, achieving better results. Results are strong compared to other methods on CARLA across different weather conditions\n\nWEAKNESSES:\n1. The auxiliary loss term in equation 14 is not ablated. \n2. Ablation study on the design of re-fusion is missing. Why such dynamic fusion (i.e. predict FC weights and bias) is necessary, will other simple way of fusion work? \n3. The introduce of weight assignment and re-fusion leads to increase of #params and #FLOPS, which is not listed in the paper. And further, to show the performance improvement is not coming from increase of #param and FLOPs, ablation study can be designed and conducted to compare baselines with similar #params and FLOPS.\n\nQUESTIONS:\nCan authors address the concerns in weaknesses part\n\nLIMITATIONS:\nAs discussed by authors, currently one RGB, event, Depth modalities are considered, other modalities remains unknown. And currently the method is only evaluated on one driving simulation environment, whether it can be applied to other environment or real world is unknown.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}