Hierarchical Adaptive Value Estimation for Multi-modal Visual Reinforcement Learning
Authors: Yangru Huang
Peixi Peng
Yifan Zhao
Haoran Xu
Mengyue Geng
Yonghong Tian
Venue: NeurIPS 2023 poster (2023)
Keywords: vision-based reinforcement learning
multi-modal
event camera

Abstract: Integrating RGB frames with alternative modality inputs is gaining increasing traction in many vision-based reinforcement learning (RL) applications. Existing multi-modal vision-based RL methods usually follow a Global Value Estimation (GVE) pipeline, which uses a fused modality feature to obtain a unified global environmental description. However, such a feature-level fusion paradigm with a single critic may fall short in policy learning as it tends to overlook the distinct values of each modality. To remedy this, this paper proposes a Local modality-customized Value Estimation (LVE) paradigm, which dynamically estimates the contribution and adjusts the importance weight of each modality from a value-level perspective. Furthermore, a task-contextual re-fusion process is developed to achieve a task-level re-balance of estimations from both feature and value levels. To this end, a Hierarchical Adaptive Value Estimation (HAVE) framework is formed, which adaptively coordinates the contributions of individual modalities as well as their collective efficacy. Agents trained by HAVE are able to exploit the unique characteristics of various modalities while capturing their intricate interactions, achieving substantially improved performance. We specifically highlight the potency of our approach within the challenging landscape of autonomous driving, utilizing the CARLA benchmark with neuromorphic event and depth data to demonstrate HAVE's capability and the effectiveness of its distinct components.

Hierarchical Adaptive Value Estimation for Multi-modal Visual Reinforcement Learning
Yangru Huang', Peixi Penga.3 2.3 Yifan Zhao Haoran Xu3.4 Mengyue Geng', Yonghong Tian 1,2,3
School of Computer Science, Peking University
School of Electronic and Computer Engineering. Shenzhen Graduate School, Peking University *Peng Cheng Laboratory
"School of Intelligent Systems Engineering, Sun Yat-sen University
yrhuang@stu, pku. edu. en, (pxpeng, zhaoyf, mygeng, yhtian]Opku, edu, en, xuhr9mail2. wtw.:mmmmmm.............mm..m..m edu, cn
Abstract
Integrating RGB frames with alternative modality inputs is gaining increasing traction in many vision-based reinforcement learning (RL) applications. Existing multi-modal vision-based RL methods usually follow a Global Value Estimation (GVE) pipeline, which uses fused modality feature obtain unified global environmental description. However, such a feature-leve fusion paradigm with a single critic may fall short in policy learning as it tends overlook the distinct values of each modality. To remedy this, this paper proposes Local modality- customized Value Estimation (LVE) paradigm, which dynamically estimates the contribution and adjusts the importance weight of each modality from value- level perspective. Furthermore, task-contextual re-fusion process developed to achieve a task-level re re-balance of estimations from both feature and value lev- els. To this end, a Hierarchical Adaptive Value Estimation (HAVE) framework is formed, which adaptively coordinates the contributions of individual modalities as well as their collective efficacy. Agents trained by HAVE are able to exploit the unique characteristics of various modalities while capturing their intricate inter- actions, achieving substantially improved performance. We specifically highlight the potency of our approach within the challenging landscape of autonomous driv- ing, utilizing the CARLA benchmark with neuromorphic event and depth data to demonstrate HAVE's capability and the effectiveness of its distinct components. The code of our paper can be found at https://github.ccm com/ Yara- HYR/HAVE.
Introduction
Recent years have witnessed renewed interest in multi-modal perception in the computer vision research community [I00. H.. M21 26. [2] For many visual tasks such as semantic segmentation and object detection, the inclusion of multi-modal data (e.g., depth, infrared) is proven be indis- putably beneficial [III, 29, 29, 25). This trend equally applies to the intelligent agents in vision- based Reinforcement Learning (RL), which multi-modal inputs can also promote decision ro- bustness [39, [9. IL I. BT]] For instance, an cattoooou--drivinng agent taking RGB frames solely as input may frequently suffer from extreme light conditions, as shown in Fig. I[Ka). Neverthe- less, combining additional sensory inputs such as event signals coming from a neuromorphic event camera [27] can effectively alleviate these problems, enabling a more comprehensive realization of
traffic status R 32 [2]
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Contribation
Contribution
RGR
Events
RGB Fvents
RGB
Events
RGB Fvents
(a) Under-exposed RGB frames vs. HDR event signals
(b) Clear RGB fromes vs. Noisy event signals
Feature-
modality
Value-level Fusion
RGB
Fusion
RGB
10.-
RGB Critic
Contriburion
sssignment
Fision
Critic
Events Critic
BD
Events
Events
(c) Global Value Estimatioon (GVE)
(d) Local modality Value Estimation (LVE)
Environment
RGB
Feedback Forward
Top: Desired modality contributions of (a) under-exposed RGB frame vs. high dynamic range (HDR) event signals, and (b) clear RGB frame vs. event signals saturated by background noise. Bottom: Multi-modal fusion at (c) the feature level, where only the fused modality feature is used for value evaluation, and (d) the value level, where individual value estimation is conducted for each modality to identify which one performs better under the current eircumstance.
Figure 1:
Despite the abundance of fruitful studies in the field of multi-modal visual perception, majority of them focus on traditional static learning tasks. In contrast, the dynamic and unlabeled nature of RL renders the development of multi-modal agents exceedingly difficult [6]] To reconcile multi-
sensory data, existing methods [T88 66, G. 22 200 train the agents with fused modality features and form mixed global modality, as shown in Fig I(c). The policy is then learned via Global Value Estimation (GVE), in which the global modality feature is responsible for environment description and then paired with action compute single Q-value. Although being an effective and widely adopted paradigm, utilizing GVE alone may overlook the distinct task-related contributions of dif- ferent modalities. Specifically, due to the diverse attributes of sensors, single modalities may not contribute equally under multiple environmental conditions (e.g., the cases given in Fig. M (a) and (b)). The feature-level fusion process GVE, no matter how sophisticated, inevitably disregards the unique value of each sensory data since only single critic is shared over multiple modalities. As result, negative interference between modalities may occur and compromise learning performance.
Based on the above analysis, we propose Local modality-customizze Value Estimation (LVE), as illustrated in Fig. (d). Differrett from feature-level fusion, LVE learns single policy with distinct, per-modality value calculation, forming value-level fusion paradigm through a tailor-designed Q- value weighting process. As a result, the modality-specifee contributions can explicitly estimated, which promotes policy flexibility. In addition, while LVE is intrinsically a better alternative than GVE, the two paradigms are not competitive but complementary. Considering this, we further de- velop a lask-contextual re-fusion process, which utilizes an efficient fusing network guided directly by the task reward to reach a task-level balance between LVE and GVE. The collaboration of the above components eventually forms Hierarchical Adaptive Value Estimation (HAVE) frame- work for multi- modal vision- RL, yielding potent policy that can flexibly leverage the unique strengths of each modality while profiting from the comprehensive information from all modalities. We specifically highlight the potency of our approach to the challenging autonomous driving task. In particular, explicitly consider the neuromorphic event camera [27]] signals that capture the motion information of the environment, which well-suites our task but has not yet been explored by existing multi-modal RL algorithms.
In summary, the contributions of our work are three-fold: 1) We design novel hierarchical adaptive value estimation (HAVE) framework for multi-modal vision-based RI. HAVE distinctively features value estimation (LVE) paradigm to enable optimized reward allocation based on modality importance. 2) We develop a cosk-cont re-fusion process to merge the profi-
a local

---

ciencies of LVE and GVE. allowing HAVE to benefit from both particularized and unified modality values. 3) Our approach achieves state-of-the-ae performance on challenging autonomous driving tasks. As the first multi-modal RI. algorithm considering event camera signals as one of the in- put modalities, our approach exhibits superior performance under various environmental conditions, demonstrating its potential as an effective multi-modal vision-based RI. solution.
2 Related Work
Vision-based RL aims to train agents that receive raw image-based observations from environments for decision-making. Compared to state-based RL, vision-based RL has found sigmificant use in many practical tasks, from video game playing [ to robotic manipulation [2II]- However, learning policy directly from such high- nigh-dimensional input is challenging. To tackle the problem, a consider- able amount of works have been developed, including 1) applying data augmentation to increase the diversity of samples [50, 2al [0]. 2) introducing auxiliary tasks such as contrastive loss [2X,, 5z 2], 3) pretraining an encoder to improve the representational ability [5I 28, B2], and 4) modeling envi- ronment dynamics in the latent space [117, L6, [88]. Although most vision-based RL methods adopt RGB camera frames as inputs, some recent works have also started to explore new sensors and data formats for RL, such as event cameras [AR 27] which captures fast and asynchronous light changes with high temporal precision.
Multi-modal Visual Learning has been extensively studied in the field of computer vision [G]] With the development of sensor technologies, it becomes effortless to acquire sufficient data from complemented visual modalities (e.g.. RGB, infrared, depth, and event signals). As a result, many multi-modal learning methods are proposed for traditional visual tasks, such as object detection [29] and segmentation [ITO, 25]. For multi-modal RL, the agent's observation space is modified to in- clude all modalities. Recent works have started to focus on multi-modal vision-based RL due to its improved effectiveness compared with using only singlc-modality data. Chen et al. propose a multi-modal state-space model trained with mutual information lower-bound promote the consis- tency between the latent codes of each modality [K]] A fusion network is proposed by Khalil et al.to produce accurate joint multi-modal perception and motion prediction for autonomous driving [22]] Ma er al. propose multi-modal RL approach that focuses on modality alignment and importance enhancement [[II]] There are also multi-modal RL methods designed for other tasks such as robot control [ and dialog system [53, 53]]. Despite their distinct technical details, most of the exist- ing methods perform feature-level fusion of modalities and ignore the properties of RL itself. In fact, policy learning depends on the value estimation of the critic. Consequently, we view the multi- modal visual RL problem from a novel adaptive value estimation perspective. Instead concen- trating solely on robust global modality features, approach synergistically balances individual modalities' contributions, leading to a more equitable and efficient value allocation. Leveraging the task-contextual re-fusion mechanism, our method further capitalizes on both feature- based and value-based fusion paradigms, resulting in a more robust policy.
Methodology
3.1 Preliminaries
Problem Definition We formulate the task of multi-modal vision-based RL as a Markov Decision Process (MDP) [B_]] with multiple observations, which is defined by a tuple (S, A, y), where
S = = O.: is the joint observation space of d modalities and OM, is the observation space of modality i. A is all possible actions the agent can take. P S x A x S - [0 is the transition probability function, P(oe+1lor, at) denotes the probability of transitioning from joint observation Ma
or = -0,"a ,0 ..... o Ma) at time step to the next joint observation 0++1 = (o/41 0411
after taking action a from a policy function
Soft Actor-Critic (SAC) Our approach is built on SAC [ which goal is to learn a policy that maximizes the expected cumulative reward while maintaining exploration by encouraging diverse actions. In SAC, the objective function given by introducing policy entropy term:
Lu Ea Nir [Q(or, oz) a log T(a:lo)))
(I)
3

---

26 eb
3CG2 40
h
Critic
Contribution Assignment
Task Re-fusion
Softmax
0/
b
Scale
Feature Fusion
MatMul
Critic
Critic
GIE
LVE
Hypennetworks
W
Ge .a)
(U/" melli
(a) GVE
(b) LVE
(c) HAVE
Figure 2: HAVE framework and distinct components. (a) Global Value Estimation (GVE), (b)
Local modality-customized Value Estimation (LVE), and (c) HAVE with task-level value re-fusion.
where Q is the value function, and is temperature parameter that controls the trade-off between exploration and exploitation. The value function is trained using the Bellman equation and a soft Q-function update:
LQ - [Q(or, (.) - (R(os, az) + BBor,i (V(+++)))))
(2)
where o₂ and a are sampled from the replay buffer and y is the discount factor. V(o++1) is the soft state value function, defined as:
V(o++1) = Eayprr [Q(o++1, (+++)) - a log T ((
where Q denotes an exponential moving average of the critic network Q and ãzυ1 comes from the current policy.
Event Cameras and Representations Despite RGB frames, another major modality that we use to evaluate approach is the event signals generated by neuromorphie event camera [27]. Each pixel in the event camera outputs a positive/negative event signal whenever the log light intensity of that pixel has increased/decreased by constant threshold. The event signals have extremely high dynamic range (up to 120 dB) and can reach a high temporal resolution in the order us. Therefore, they are able capture the missing motion clues that missed in the RGB frames for many visual tasks such as autonomous driving and robot navigation. To utilize event signals, we use the stacking based time (SBT) [AS] representation that splits the event sequence into fixed temporal bins, forming an event frame representation with multiple channels similar to RGB frames.
Global Value Estimation
In Global Value Estimation (GVE), total of d observation encoders used to extract observation features A.M for each modality observation M: time step t. A global modality feature fî is
then calculated by fusing all modality features. For simplicity,
directly concatenate I J., (i
1, .......d) d) along the channel dimension. Given f2 GVE first computes the value estimation of
Qf+ at) using a single global critic network Q²s, as shown Fig. D(a). Following the SAC algorithm, the target value y is then calculated as:
v = R(o, a.) (
(4)
where R(o,, at) is the reward returned by the environment. The soft state value function V(f++) is denoted as:
v( = Ea, 1~- (JE+1: 1+41) log n (ā,ạ [2-1)]
(5)
However, since using a single value function Qo, shared over all modalities, it is hard to dynamically balance the relative contribution of each P. under complex environmental conditions.

---

3.3 Local Modality Customized Value Estimation
The objective of Local ate and quantify the unique contribution of each modality, thereby facilitating a more granular and modality-aware decision-making process. Specifically, in LVE, d individual value functions Q QPM² Q are setup instead of global-only functions in GVE. These value functions share the same network architecture but are learned separately. To facilitate value decomposition, an assignment network is designed to assimilate all estimated values and re-calibrate them according to the collective environmental condition portrayed by modality features.
Value Estimation (LVE) is effectively differenti-
Value Inference: We assume that the locally-aggregated total value of can be approximately decom- posed into linear combination of the separate value functions across different modalities under a shared policy:
d ~
wi Mi QOME (ft)),))) ,a);
(6)
where Mi denotes the contribution weight of modality M, to the total value. The value function Qaar, for each modality processes the current individual modal observation feature f. along with the current action or, generating the estimated action values at each time step. Each Qou M. is learned by backpropagating gradients from the Q-learning rule. Similarly, the total target value yf can be defined as:
vt ~ R(o,, at) + + -.. 1211-
The soft state value function V( .. l(f11 Si4 is then defined as:
V(
Mi
Ea., CM
Σ
Mr Qfa (141), a log x (ār+1ll22))))
(8)
where v/ is the contribution weight of modality Mj to the the estimated Q-value from Qaa, at the next time. ăe+1 T(-151+1) comes from the current policy according to the global modality feature l241 since there is only one policy network for the multi-modal decision task.
Contribution Assignment: Given value decomposition in Eq. and Eq. a critical next step is calculate accurate modality weights we and ug ensure precise total value estimation. To achieve this, the contribution assignment process involves modality interactions through an assign- ment network based on the attention mechanism. Taking the calculation of w; as an example, the global modality feature f is used a bridge to ensure information flow between all individual modalities. As demonstrated in Fig. [kbb, two fully connected (FC) layers with parameters W,, and Wa are used to project both J, and J into p-dimensional common latent subspace. Then w.M. can be obtained through a softmax function as:
Mi exp((W, (W..)) vD) W
DL (W) /VP))
M, where v is used for scaling to prevent vanishing gradients [55]. The computation of uttl follows a similar derivation, which is omitted here for brevity.
By assigning the value function in a modality-customizze manner, LVE adeptly manages multi- modal sensory inputs, enhancing model efficiency and interpretability. Our approach readily adapts to complex environmental scenarios, accommodating varying modality importance levels.
3.4 Task-Contextual Re-fusion
Feature-based vs. Value-based Fusion From the details of GVE and LVE, we see that the main difference distinguishing them is their fusion principle (feature-based vs. value-based). Intuitively, the two paradigms are not competitive but complementary. Specifically, in the context of RL, value- based fusion offers significant advantages over feature- based fusion since the former is more related to the actual decision. the meantime, the feature-based fusion method can use its collective modal- ity value to offer a value-based one with a sturdy global reference, mitigating inaccuracies caused by input noise. Therefore, we further develop task-contextual re-fusion for a subsequent re- re-fuse of GVE with LVE in the reward/task level. The idea is to directly bridge the two fusion mechanisms
5

---

based on environmental reward feedback, allowing the task to determine which paradigm better suits the current situation.
Details of the Re-fusion Process Given the estimated value of and gे obtained from GVE and LVE, respectively, we adopt dynamic fusion mechanism with re-fusion network H. fuse of and df. Specifically, H consists of a series of FC layers and hypernetworks. Each hypernetorkk consists of a single linear layer, which receives the global modality feature fi as input and produces a weight matrix or bias vector for a single FC layer in H. To fuse of and d/. !e is first sent into all hypernetworks obtain the parameters of the FC layers in H. then of and qi are sent into H. to get the final total value H(g (.). The final target value can be obtained using similar way, wherein the estimated Q-value from the target Q-network at the next time is processed by another mixing
network H'. For brevity, we denote the this target value as H' y!).
3.5 Learning Framework
Coping GVE with LVE and cast-contettual re-fusion, our approach forms a unified framework, which we name as Hierarchical A Adaptive Value Estimation (HAVE) for multi-modal vision-based RL. this section, we elaborate on the training details of our HAVE framework.
Policy Evaluation with HAVE For the training of policy evaluation, we minimize the temporal dif- ference (TD) error between the predicted value H(g 4f). and the target value H' (1² vi) following the SAC algorithm:
Lg = B (Hl d - H('vi.1)))
(10)
where D denotes the replay buffer.
Policy Improvement The policy evaluation described above can cover reasonable Q-values to help find better policies. In the policy improvement step, we leverage the values obtained from GVE, LVE, and the aas-cconexttual re-fusion process to update the policy æ. To be specific, the goal is to maximize the expected cumulative reward by selecting actions that have the highest combined value estimates:
C. = Eo ~ [H( , d) a log (a.l)())
(11)
After identifying the optimal action under the improved policy, we can update the policy parameters by backpropagating gradients.
Auxiliary Losses Besides policy evaluation and improvement, also learn predict the rewards and next latent states as in DeepMDP [T3]] to assist representation learning, providing a latent state space that is consistent with environmental dynamics:
Lour [IP as) 12 + Σ IIP (f. ar) - -f44 112 +
((01))
(12)
where Pa and PM: 1, d) are transition networks for the global and individual modalities, respectively. R9 is the reward prediction network. All these networks are formed by FC layers and a more detailed architecture description can be found in the supplementary material.
Overall Training Objective Given the above training objectives, the overall loss of HAVE is finally defined by:
L Lq + Lx + Laur-
(13)
By iteratively performing policy evaluation and improvement steps, HAVE can effectively exploit the strengths of both global and distinct value estimates, ultimately converging to an improved policy that can handle the challenges posed by a broad range of environmental modalities.
3.6 Further Analyses
Our hierarchical adaptive value estimation framework for multi-modal RL process offers several key benefits, which we list as follows:
Remark (Prevention of Modality Dominance): The mechanism of LVE and task-contextual re- fusion prevents one modality from dominating the others, thereby avoiding the issue of modality collapse.

---

To realize this, consider that the mixing network H. in the task-contextual re-fusion performs fixed non-lineat transformations on both of and q; given ff" where of formed by the convex combination of individual modality values (Eq. ( as in LVV. Therefore, we have:
HHig st) sign( 0y6
HH gb)
dut Ou Mi
HHih 96
sign(
sign(
Vi E [1,2, ... dj,
(14)
0g.
0gk
where sign(-) the real sign function, qi = Q0s (feM a.) is the individual modality value and ogj wn
are the modality weights. Eq. I- indicates that the contributions of all
Bq:
modalities are either increasing or decreasing together during the learning process, which prevents opposite gradient values between modalities that enhance some of them while suppressing others. In addition, the calculation of modality weights wa' based on softmax global-individun feature similarity (Eq. Hi further prevents modality collapse caused by persistent zero weights.
Remark 2 (Pareto Optimality of Modalities): Assuming continuous action space, the global opti- mal action produced by our approach achieves Pareto optimum across individual modalities.
Suppose the optimal action given some fixed modality features is a", which implies Combining Eq. K with the chain rule, we have:
0.
Ou
0Hlgi df) Ou"
ddt
OHlL d
M, Oye wi
Σ
O.
(15)
dqt
Oa
Oqk
Oa-
Considering the non-linear structure of H. ani a is unlikely be zero for all of. This implies 8q;
that Di= M. ay for all = 1.......e
.......dd d. which suggests that at the point a", the positive weighted sum of the gradients of individual modalities' values is zero This implies that there is no alternative action assignment in the vicinity of a that can increase the value for any individual modality without decreasing the value for some other modality. Such condition forms Pareto optimum across the modalities, which indicates the resources (modality values) are allocated in the most efficient way possible [35]].
Ba
Experiments
4.1 Settings
Environments To evaluate our approach under realistic and challenging multi-modal environments, we employ the CARLA simulator [8], which is a widely used open-source platform for autonomous driving research. CARLA provides a rich and realistic urban environment to evaluate autonomous driving agents in various traffic scenarios. Three distinct modalities are adopted: 1) RGB frames, 2) event signals generated by CARLA'S synthetic event-camera simulator, and 3) per-pixel depth frames. Eight different weather settings are used to provide thorough coverage of different envi- ronmental conditions. The action space consists of continuous control actions, such as steering, acceleration, and braking. Similar to [52] B] the reward function is designed to encourage the agent to maintain safe distance from 20 other moving vehicles and obstacles, and drive as far as possible along the highway of CARLA's Town04 map in 1000 time steps. We use the single camera view setting on the vehicle's 's roof with 60-degree views.
Implementation Details Our approach is implemented based on SAC [14 [-]] and DeepMDP [ The same encoder network architecture and training hyperparameters are adopted for all comparative methods. Following common practices [500, [3]] we convert each modality data into its correspond- ing image-based representations and stack several consecutive images to infer temporal information. The spatial resolution of the input images is 128 x 128 and the channel numbers of RGB, event and depth frames are 3, 5 and 1, respectively. All methods are trained for 120k frames across 5 random seeds to report the mean and standard deviation of the rewards. We evaluate the performance of each approach terms of driving episode reward and distance. Other details are provided in the supplementary material.
7

---

Neather
Neasures
SAC
D=q
DeepMDP
TransFuser
BFNet
Durs-LVE
Ours-HAVE
ER
186 65
242= 84
225::: 87
2602 95
2411
274 68
3191 71
ClearNight
D(m)
1124 39
169 41
161: 51
178 43
170:t: 62
1924 50
212:1 52
CloudyNight
ER
218: 71
248 69
265 85
280 71
289: 64
2954 67
322:1 96
1Xm)
167 36
55
195+ 43
1974
213+
217-+
ER
170= 90
261 91
255 77
287 65
275+ 72
294 85
327 87
HardRainNight
D(m)
107+ 63
163= 63
160+ 36
204+ 56
203+ 54
2094 54
229+ 51
1894
234=
97
24
274 76
290:1
289 112
304= 102
WeiNight
D(m)
1274 57
155= 53
1644 36
171: 58
2014
196:1
222:1 69
ClearNoon
ER
235+ 58
280 94
269
282+ 53
234
294 82
336 76
D(m)
153: 49
195=
1874
193 39
150 46
1862 61
2231 44
CloudyNoon
201 87
274 77
226: 24
277:
261
2934
3154
D(m)
138
170= 42
1364 16
171
1641
1864
2091
ER
1894 74
220 72
248
264 99
279 91
2874 95
316:1: 88
HardRainNoon
D(m)
104
129= 60
161+
178+ 62
208 76
207+ 69
218+ 63
209+
245- 83
226+
304+
2734
2964
341-
WetNoom
D(m)
136 64
172
58
169
213+ 51
204+ 70
215+ 54
239: 55
Table 1: Comparison with state-of-the-art methods eight different kinds of weather. ER denotes
episode return and distance meters. The best results are bolded and the second best results underlined.
4.2 Comparison with State of the Art
We compare our method with a variety of methods, including the SAC [TS]] baseline, DeepMDP [L_]] DrQ [50], TransFuser [Z]], and EFNet [E3]]. For RL methods SAC, DrQ and DeepMDP, we directly concatenate the features of the different modalities as the input for subsequent value and policy learn- ing. For TransFuser and EFNet which are designed for traditional visual tasks, we only adopt their advanced modality fusion modules and keep DeepMDP as the RL algorithm for a fair comparison.
Results of Two Modalities We first evaluate different methods under two modality inputs (RGB frames and event signals) in Table II The results show that our method achieves the highest episode reward and driving distance under all eight weather conditions. DrQ and DeepMDD perform het- ter than the SAC baseline with limited improvement, showing that the feature-fusion based GVE paradigm, together with simple feature concatenation, cannot fully extract the expressive power of each modality. The improved performance of TransFuser and EFNet over DeepMDP reflects the importance of the advanced modality feature fusion mechanism. However, they are still inferior to OVEs which uses the proposed LVE paradigm to explicitly assign modality contributions weights with value- based fusion. The results indicate that the key to multi-modal RL is consider the suitability of individual modalities. Finally, by using the task-contextual re-fusion process to integrate GVE and LVE, our full method achieve superior performance even with simple feature concatenation fusion, proving the effectiveness of our taskedriven hierarchical design.
Results of Multiple Modalities We then eval- uate our method on three modalities (RGB frames, event signals, and depth frames). Al- though TransFuser and EFNet technically pos- sible to scale both methods to accommodate more than two modalities, such an adapta- tion would require significant modifications to the implementation, along with quadratic increase in computational complexity due to cross attention mechanism. Thus we directly
Clennhiigt
Clearlicon
350
800
150
0.33
009
6.22
0.04
606
B12
Number of frames (x10")
Number of frames (x10%)
Figure 3: Performance with three modalities.
compare our method with SAC, DrQ and DeepMDP. The training curves under two weather con- ditions are demonstrated in Fig. C. which also show the advantage of our methods. Additional experiment results can be found the supplementary material.
4.3 Performance Analysis
Ablation Studies To systematically evaluate the effectiveness of the proposed HAVE approach and its individual components, we conduct a series of ablation experiments on the CARLA benchmark that trains agents using: 1) single modality data with RGB frames or event signals, 2) GVE with feature concatenation, 3) our proposed LVE paradigm, and 4) our full method. The training curves and testing performance shown in Fig. A and Tab. E, respectively. First, we observe that the indi-
8

---

Cherlligh
HardPawnight
Weslligh
350
303
250
300
300
100
100
2044 0.9 Number 1x10%)
0.99 Number [x10")
000
3.2
Number frames 1x100)
Number frames [x300)
Clearhaon
Claudlyhoon
Weetfiion
400
220
200
200
300
tte
Number framms (x10P)
Number tramne (x330))
Number frames (x30P)
Number trames (x
Figure 4: Training curves of individual modalities and different multi-modal training paradigms.
Methods
Measures
Clear
Cloudy
Night
HardRain
Wet
Clear
Clondy
Noon
HardRain
Wet
RGB
ER
75 1
194
36
189
88 23
214 1 55
201 37
223 67
232 74
DXm)
46 + 21
121
24
136 +
53 +
26
125 + 42
121
25
128
+ 51
141 54
ER
158 1. 29
187 1.32
164 28
175 1 29
145 1 37
151
1 41
154
1 34
147 40
Events
DXm)
102 + 23
129
+-26 26
105
+ 21
120
+ 18
86 27
92
+ 32
104
21
86
+ 36
GVE
ER
225
1 87
265 8.85
255
1 77
241
1 47
269
1
226
24
248
59
226
1.52
DXm)
161 51
183 4-55
160
+
164 + 36
187 + 36
136
16
161
+ 43
169
+ 38
LVE
ER
274
68
295
67
294
289 112
294 1 82
293
76
287
1 95
296
84
DXm)
192
+ 50
213
+ 44
209 +
196 + 61
186 + 61
186
44
207
+ 69
215
54
ER
319
71
322
96
327
1 87
304 1 102
336
1 76
315
82
316
1 88
341
1. 78
HAVE
D(m)
212
52
217
+ 53
229 51
222 + 69
223 + 44
209
68
218 + 63
239 55
Table 2: Test performance of different models in Fig. EL Table notations are the same as in Table
Modality Weights (MGB frames)
ClearNight
Modalit Weight (RGB frames)
ClearNoan
1.00
1.00
Event SEnAS R5B fraites
3.05
0.05
0.90
0.50
0.85
0.80
0.83
100
250
8
A
step
Figure 5: Visualization of modality weights during testing. Note that only weights of RGB frames drawn and the weights of event signals are one minus RGB frame weights any time step.
vidual modality is vulnerable to failure under extreme weather conditions. For examplc. the episode reward trained with RGB frame under ClearNight weather is pretty poor as shown in Fig. A. Second, directly fusing the two modalities with GVE generally improves performance. However, the perfor- mance gain is limited, and in some cases even close to the results of using single modality (e.g., in WetNight and ClearNoon). This might be caused by the heterogeneity of multi- data, which leads to difficulty in determining the target reward. Third, in most cases, LVE performs significantly higher than single modality or GVE-based agents, showing better cooperation of different modality data. Finally, our full method with cask-contextual re-fusion clearly outperforms all other models thanks to the synergistic interplay of GVE and LVE at the reward/task-leva
Visualization To further obtain an intuitive understanding of our approach, we visualize the modal- ity weights obtained by HAVE in Fig. El using RGB frames and event signals under two different weather conditions. The following observations can be made: at the beginning, all vehicles have zero initial speed. As a result, no valid event signal is generated and the weights of RGB modality are nearly 1.0. 2) After speeding up. the weights of event signals are continuously increasing, espe- cially when there are many moving vehicles ahead. 3) However, when the agent is facing complex background (such as road fences and clear roads without close vehicles), the event signals mainly consist of background noise, and the weights of RGB frames will raise again. The varying modality weights show that our approach can indeed adjust modality contributions under different situations, which serves as a key advantage over previous multi-modal RI. methods.

---

Conclusion and Limitation
We have studied the representational capacity of value functions in multi-modal vision-based RL problems. We hypothesize that the limitation of feature-level fusion methods may come from un- clear contributions for different modalities under a single value function. To mitigate this, we have presented novel Hierarchical Adaptive Value Estimation (HAVE) framework to reconcile both feature-level and value-level fusion task/reward-driven manner. Our approach represents one of the first explorations of coddaliy-sseccfiii and hierarchical value estimation for based RL tasks. Extensive experiment results demonstrate the effectiveness our approach. How- one limitation of our work is that mainly focus the value estimation of multiple visual modalities, while the effectiveness In future work, will consider utilizing both visual and other modalities, forming more generalized multi-modal RL framework.
vision-
other forms of modalities (e.g., audio, text) is not verified.
6 Acknowledgement
The study was funded by the Key-Area Research and Development Program of Guangdong Province with contract No. 20218010140000,, the National Natural Science Foundation of China under con- tracts No. 62027804, No. 61825 101, No. 62088102 and No. 62202010, and the major key project of the Peng Cheng Laboratory (PCI.2021A13), Computing support was provided by Pengcheng Cloudbrain.
References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, al. Do as can, as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0165 2022.
[2] Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In Proceedings of the International Con- ference Learning Representations, 2022.
[3] Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Múbaa. survey deep multimodal learning for computer vision: advances, trends, applications, and datasets. Visual Computer, 2022.
[4] Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, and Dhruv Batra. Embodied mul- timodal multitask learning. In Proceedings of the International Joint Conference on Artificial Intelligence, 2020.
[5] Guang Chen, Hu Cao, Jorg Conradt, Huajin Tang, Florian Rohrbein, and Alois Knoll. Event- based neuro- morphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine, 2020.
[6] Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. In IEEE Proceedings of the International Conference on Robotics and Automation. IEFF, 2021.
[7] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Trans- fuser: Imitation with transformer- based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[8] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning. PMLR, 2017.
[9] Linxi Fan, Guanzhi Wang, De-An Huang. Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandku- mar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Proceedings of the International Conference on Machine Learning, 2021.
[10] Feng, Christian Scnsit, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser. Fabian Timm, Werner Wiesbeok, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transporta- tion Systems, 2020.
10

---

[11] Wen Fu, Yanjie Li, Zhaohui Ye, and Qi Liu. Decision making for autonomous driving via multimodal transformer and deep reinforcement learning. In IEEE International Conference Real-time Computing and Roboticx, 2022.
[12] Mathias Gehrig. Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: stereo event camera dataset for driving scenarios. IEEF Robotics and Automation Letters, 2021.
[13] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofrr Nachum, and Mare G Bellemare. Deepmdd: Learn ing continuous latent space models for representation learning. In Proceedings of the International Con- ference on Machine Learning. 2019.
[14] Tuomas Haarmjja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference on Machine Learning, 2018.
[15] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta. Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint Xrv: hafp:/88200555 2018.
[16] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint
2019.
[17] Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lec, and James David- son. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning, 2019.
[18] Zhiyu Huang. Chen Lv, Yang Xing, and Jingda Wu. Multi-modal sensor fusion- deep neural network for end-to-end autonomous driving with scene understanding. IEEE Sensors Journal, 2020.
[19] Yunfan Jiang. Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: 22 10.03094, 2022.
[20] Yang Jiao, Zequn Jic, Shaoxiang Chen, Jingjing Chen, Xiaolin Wei, Lin Ma, and Yu- Jiang. Msmd- fusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection. arXiv preprint arXiv:2209.0 03 102, 2022.
[21] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz. Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learn ing for rision-based robotic manipulation. arXiv preprint 186. 10293, 2018
[22] Yasser H Khalil and Hussein T Mouftah. Exploiting multi-modal fusion for urban autonomous driving using latent deep reinforcement learning. IEEE Transactions on Vehicular Technology. 2022.
[23] Michael Laskin, Aravind Srinivas, and Pieter Abbecl. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the International Conference on Machine Learning, 2020.
[24] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Picter Abbeel, and Aravind Srinivas. Reinforce- ment learning with augmented data. In Proceedings the Advances in Neural Information Prucessing Systems, 2020.
[25] Jiale Li, Hang Dai, Hao Han, and Yong Ding. Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving. arXiv preprint arXiv:2303. 2023.
[26] Yingwei Li. Adams Wei Yu, Tianjian Meng. Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu. Denny Zhou, Quoe Le, al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[27] Patrick Lichtstciner, Christoph Posch, and Tobi Delbruck. A 128x 128 120 db 15 jss latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits, 2008.
[28] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Proceedings of the Advances in Neural Information Processing Systems, 2021.
[29] Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and a multi-scenario modti-my benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
11

---

[30] Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A compre- hensive survey of data augmentation in visual reinforcement learning. arXiv preprint urXiv:2210.04561. 2022.
[31] Jinming Ma, Feng Wu, Yingfeng Chen, Xianpeng Ji, and Yu Ding. Effective multimodal reinforcement learning with modality alignment and importance enhancement. arXiv preprint arXiv:2302. 09318, 2023,
[32] Ana I. Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza. Event- based vision meets deep learning on steering prediction for self-driving cars. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2018.
[33] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to ac- tions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2017.
[34] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 132.5560, 2013.
[35] Martin J Osborne and Ariel Rubinstein. A Course in Game Theory. The MIT Press, MIT Press, London, England, July 1994.
[36] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2021.
[37] Aditya Ramesh. Prafulla Dhariwal, Alex Nichol, Casey Chu. and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06125, 2022.
[38] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Proceedings of the Conference on Robot Learning. PMLR, 2023.
[39] Afagh Mehri Shervedani, Siyu Li. Natawut Monaikul, Bahareh Abbasi, Barbara Di Eugenio, and Mi- los Zefran. Multimodal reinforcement learning for robots collaborating with humans. arXiv preprint arXiv:2303.07265, 2023.
[40] Hailuo Song, Ao Li, Tong Wang, and Minghui Wang. Multimodal deep reinforcement learning with auxiliary task for obstacle avoidance of indoor mobile robot. Sensors, 2021.
[41] Mengke Song, Wenfeng Song, Guowei Yang, and Chenglizhao Chen. Improving rgb-d salient object detection via modality-aware decoder. IEEE Transactions on Image Processing, 2022.
[42] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 2021.
[43] Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool. Event-based fusion for motion deblurring with cross-modal attention. In Proceedings of European Conference on Computer Vision, year=2022.
[44] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhia. Attention is all you need. In Proceedings of the International Conference on Neural Information Processing Systems, 2017.
[46] Sai Vemprala, Sami Mian, and Ashish Kapoor. Representation learning for event-based visuomotor poli- cies. In Proceedings of the Advances in Neural Information Processing Systems, 2021.
[47] Celyn Walters and Simon Hadfield, Ceril: Continuous event-based reinforcement learning. arXiv preprint arXiv:2302.077 076677 2023.
[48] Lin Wang. IS Mohammad Mostafavi, Yo-Sung Ho, and Kuk-Jin Yoon. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
[49] Zongwei Wu, Guillaume Allibert, Fabrice Meriaudeau, Chao Ma, and Cédric Demonceaux. Hidanet: Rgh-d salient object detection via hierarchical depth awareness. arXiv preprint arXiv:2301.07400 2023.
12

---

[50] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all need: Regularizing deep reinforcement learning from pixels. In Proceedings the International Conference Learning Repre- sentations, 2020.
[51] Zhecheng Yuan. Zhengrong Xue, Bo Yuan, Xueqian Wang. Yi Wu, Yang Gao, and Huazhe Xu. Pre- trained image encoder for generalizable visual reinforcement learning. Proceedings of the Advances Newral Information Processing Systems, 2022.
[52] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant rep- resentations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[53] Jiaping Zhang. Tiancheng Zhao, and Zhou Multimodal hierarchical reinforcement learning policy for task-oriented visual dialog. In Proceedingy of the Annual SIGdial Meeting on Discourse and Dialogue, 2018.
[54] Yue Zhao, Chenzhuang Du, Hang Zhao, and Tiejun Li. Intrinsically motivated self supe-suue learning in reinforcement learning. In Proceedings of the International Conference on Robotics and Automation. TEFE, 2022.
13


Reviews:
[{"review_id": "5wgYrKYXII", "reviewer": "Reviewer_VLp8", "summary": "The paper proposes a Hierarchical adaptive value estimation (HAVE) framework for multi-modal (RGB, event, depth) visual reinforcement learning. Firstly, a modality-specific value function learning process is proposed, and an assignment module is proposed to weight different modality. Second, a re-fusion is proposed to futher combine global feature based value and modality-specific based value. Method is evaluated on CARLA simulation environment. Various baselines, including SAC, DrQ, DeepMDP, etc are compared and the results show the proposed HAVE perform better under different weather conditions. ", "questions": "Can authors address the concerns in weaknesses part", "limitations": "As discussed by authors, currently one RGB, event, Depth modalities are considered, other modalities remains unknown. And currently the method is only evaluated on one driving simulation environment, whether it can be applied to other environment or real world is unknown. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The decomposition of global value function into weighted combination of modality value function is novel and well-motivated by the observation that simple feature fusion might cause dominant modality behaviour. The weight assignment module is well designed and also show meaningful behaviour in Figure 5. The re-fuse further makes the method can enjoy both the value and feature fusion, achieving better results. Results are strong compared to other methods on CARLA across different weather conditions", "weaknesses": "1. The auxiliary loss term in equation 14 is not ablated. \n2. Ablation study on the design of re-fusion is missing. Why such dynamic fusion (i.e. predict FC weights and bias) is necessary, will other simple way of fusion work? \n3. The introduce of weight assignment and re-fusion leads to increase of #params and #FLOPS, which is not listed in the paper. And further, to show the performance improvement is not coming from increase of #param and FLOPs, ablation study can be designed and conducted to compare baselines with similar #params and FLOPS.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we would like to express our gratitude for your positive comments. We also deeply appreciate the practical experimental suggestions given in your constructive feedback, which offers valuable perspectives that will strengthen our work. Below are our responses to your concerns:\n\n>1. The auxiliary loss term in equation 14 is not ablated.\n\nWe agree that conducting an ablation study of auxiliary losses would elucidate their contributions. We have carried out the corresponding experiments in the ClearNight environment of CARLA. The results are illustrated in Fig.2(d) of the rebuttal PDF. Specifically, three terms in Eq.13 comprise $L_{aux}$: the global transition loss, the sum of individual transition losses, and the reward prediction loss. When compared to the full results of HAVE in Fig.4 (top left) of the main paper, Fig.2(d) reveals that omitting any of these terms negatively impacts model performance. Among them, the reward prediction loss is the most crucial, leading to the most significant performance drop. When all three terms are excluded (i.e., the 'w/o aux' curve in Fig.2(d)), there is a notable decline in performance.\n\n\n>2. Ablation study on the design of re-fusion is missing. Why dynamic fusion is necessary, will other simple way of fusion work?\n\nThanks for your good question. This question can be separated into two parts. \n\nOn why we need dynamic fusion over static ones (such as taking the mean of GVE and LVE values), since LVE does not always outperform when GVE provides accurate estimates, and conversely, GVE does not always excel when LVE estimates are precise. Their effectiveness is contingent upon the attributes of the current global modality, which can vary drastically depending on the environmental context. To address this variability, we employ a hypernetwork for a dynamic combination of both methods. While we have experimented with directly merging GVE and LVE, Table 1 in the rebuttal PDF indicates that, although direct fusion is feasible, the dynamic fusion through the hypernetwork proves more efficacious.\n\nOn why we need a hypernetwork to perform dynamic fusion, the rationale behind employing hypernetworks is to avoid directly putting the global modality feature $f_t^g$ into the mixing network. This approach is taken because ultimately, we want the aggregated value benefits from the additional global modality information in $f_t^g$ but not overly influenced by it. If we directly put $f_t^g$ into the mixing network, the output value will arithmetically depend on it. This could impose unnecessary fusion difficulty since $f_t^g$ may vary significantly across different environment situations. Meanwhile, the re-fusion process is not strictly a task-level value fusion anymore because the feature $f_t^g$ has actively engaged in the fusion process. Instead, the utilization of hypernetworks offers the ability to condition the mixing network's weights on $f_t^g$ in a versatile manner while not overly constraining the fusion process. \n\nHypernetworks generate weights for primary networks, allowing for dynamic adaptation, parameter efficiency, and conditional computations, making them versatile tools for various tasks[1,2,3].  \n\n[1] A Brief Review of Hypernetworks in Deep Learning. arXiv preprint arXiv:2306.06955 (2023).\n\n[2] Principled weight initialization for hypernetworks. ICLR. 2019.\n\n[3] Continual learning with hypernetworks. ICLR. 2020.\n\n\n>3. Increased #params and #FLOPS are not listed in the paper. Ablation studies should be designed to compare baselines with similar #params and FLOPS.\n\n\nThanks for your suggestion. A comparison of baselines with a similar model complexity is indeed instructive. To address this, in the rebuttal PDF's Table 2, we outline the number of parameters (#params) and FLOPS for HAVE, TransFuser, and EFNet. A few key observations emerge:\n\n1. All three methods share the same observation encoder architecture.\n2. As a value estimation method, HAVE directly concatenates modality features and doesn't incorporate advanced feature fusion modules. Conversely, TransFuser and EFNet introduce significant #params and FLOPS with their attention-based modality feature fusion modules.\n3. The critic network in HAVE includes weight assignment and re-fusion networks, adding to its complexity. Nonetheless, this added complexity is modest when juxtaposed with the feature fusion modules of TransFuser and EFNet. Overall, HAVE still achieves the best result with the least #params and FLOPS. It is also worth noting that the critic is utilized only during the training phase and not used in testing. Thus, there is no actual increase in either the parameter counts or FLOPS for HAVE at the test stage.\n\n\n\n"}, {"review_id": "eDKSIdENo4", "reviewer": "Reviewer_9NFJ", "summary": "This paper proposes a fusion scheme for multi-modal RL, with a particular focus on fusing the modalities in estimating the value function. The authors propose a combination of global value estimation, local value estimation and another conditioned fusion of the global and local value estimation. The approach is evaluated in particular on fusing RGB and event-based data in an autonomous driving scenario with the CARLA simulator.", "questions": "- Why would the policy benefit from all the fusion in the value function if it is a neural network only taking the global modality feature as input?\n\n- From the results in Fig 5 it seems bulk of the contribution of the value is from the RGB frame, as its weight never goes below 0.75?\n\n- The only difference between the proposed approach and the baselines is the value function estimation. I would wonder whether the architecture yields a better value function (i.e. closer to the ground truth value) faster, which yields the improved training of the policy. However, if in the long run, any value iteration converges to the true value function, it should yield the same policy, no? I would be interesting to inspect the MSE of the value function compared a ground truth value estimate for the different algorithms over time, to see if this is indeed the case.", "limitations": "The authors only mention the limitation to one set of modalities in their work. I would make this broader to note that the approach is only tested on this one environment.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a novel scheme of \"value level fusion\" that is new to me. ", "weaknesses": "- The authors propose a very complex scheme for fusing the q-value estimate based on the different modalities, having GVE, LVE and then another fusing of those two with a hyper network. Especially since the policy is selecting actions based on a simple concatenation of feature vectors of both modalities.\n\n- There is a very high spread on the results which makes me wonder how significant the improvement is of the HAVE approach. Also, from Fig 3 and 4, it seems that the other methods are not necessarily converged yet. It would be interesting to see how the curves evolve if training would continue.\n\n- All experiments are done on this single environment with a custom reward function. It would be more convincing if the approach was demonstrated on one extra environment, i.e. it is hard to assess how significant it is to drive for instance 42m further than another method (+/- 52m) to take the first line of Table 1 as an example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly thankful for the thoughtful remarks and the experimental recommendations you have provided. These suggestions shed light on what we can improve, and we believe they will be instrumental in refining our work further. We address your main concerns as follows:\n\n> ...fusing the q-values is complex. Why would the policy benefit from this if it only takes global modality feature as input?\n\nThanks for the great question. First, we wish to emphasize: it is the value function that provides supervision signals for the policy network. Therefore, the factors that influence the policy network are two-fold: 1) accurate value estimation, and 2) robust input features. Our paper primarily addresses the first, using multi-modal strategies for better Q-value estimations. When the Q-value function is accurate, it enriches the feedback to the policy by reliably estimating future rewards for potential actions. This is the reason behind our hierarchical design, which balances individual modality contributions for optimal Q-values.\n\n\nAs for the input features, our work shows that the robust Q-values estimated by HAVE can already drive the policy network to select predictive features from only the global modality feature. While we could introduce a complex modality fusion module, its success still hinges on precise Q-values, which drive system learning. Thus, we prioritized Q-value estimation. However, your question raises a novel and natural extension of our work: to develop proper feature selection mechanisms for the policy network. We sincerely appreciate this and will definitely work on it in the future.\n\n\n> ...very high spread on the results. How significant is the improvement?  ... seems the other methods are not necessarily converged yet.\n\nThanks for highlighting this. Result variance is due to initialization randomness, a known RL challenge (cf. experiments in [1],[2]). We averaged results from five runs with different seeds for Fig. 3 and Fig. 4 following common practice [1][2]. The results reveals HAVE's superior performance by 120K steps. Some curves might not seem converged at 120K. With limited computation resources during rebuttal, we re-run experiments on the most questionable non-converged figures (i.e. Fig. 3 (left) and Fig. 4 (bottom left) ), each with the two most suspicious non-converged methods. The results are given in Fig.2 (b) and Fig.2 (c) in the rebuttal PDF, respectively. These results show convergence around 150K steps, with HAVE still leading. In future revisions, we'll extend all experiments to 150K steps.\n\n> It would be more convincing if the approach was demonstrated on one extra environment.\n\nThanks for your valuable suggestion.  We have conducted more experiments on an additional environment DeepMind Control Suite for robot control. Please refer to our general response posted above for detailed results. Experiment results show that HAVE also works positively in the new environment and can improve over other methods.\n\n> ...the contribution of the value from the RGB frame...never goes below 0.75?\n\nThanks for this keen observation. A higher weight to RGB signals over event signals can be attributed to two factors: \n\n\n1) Content richness. Compared to events, RGB images contain substantially more information, especially static ones like colors and textures. Such information is important for decision-making. Event signals, on the other hand, mostly reflect edges that are moving. In our demonstration, the RGB frames are clearly discernible by the agent. As a result, they dominate the modality contribution with abundant content. In Fig.2(a) of the rebuttal PDF, we provide the modality weight under extremely low-light conditions where the RGB frames are barely observable, we see the event signals play a more important role in such situations.\n\n\n2) Event signals are derived from discrete intensity shifts, which can introduce temporal noise and affect data reliability. Meanwhile, RGB signals offer a smoother, more consistent environment depiction, reducing noise concerns.\n\n> Whether the architecture yields a better value function faster, I would be interesting to inspect the MSE of the value function compared a ground truth value estimate for the different algorithms over time.\n\nWe provide the MSE curve in the ClearNight weather of CARLA, please see Fig.2(e) in the rebuttal PDF. The ''ground truth'' depicted in this figure is computed using the Monte Carlo method, which calculates the cumulative reward from the current state to the end of the test episode. From Fig.2(e) we see that although all methods tend to reduce their value error. Their MSE is closer along training, yet HAVE still outperforms all methods at the end of training. It is notable that such a ground truth value can not be obtained beforehand during training, so this is just for reference and the rigorous analysis needs to further explore. \n\nRL always uses Temporal Difference (TD) learning to update network, where the learning goal of value functions is updated by collective rewards and current value estimators (i.e., critic network). That is, there is no real ground truths for Q value during training.  We cannot train a good value estimator just by a long-time training because due to accumulate errors. Hence, it is very important to design a better value estimator, which is our main goal.\n\n**Reference**\n\n\n[1] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, ICML 2018.\n\n\n[2] Image augmentation is all you need: Regularizing deep reinforcement learning from pixels, ICLR 2020.\n\n"}, {"review_id": "loTWoH6z9p", "reviewer": "Reviewer_P2RM", "summary": "The authors propose a Local modality-customized Value Estimation (LVE) paradigm, which dynamically estimates and adjusts the importance weight of each modality from the perspective of value-level to compensate for the multi-modal vision-based RL methods that usually use fused modality features for learning Global Value Estimation (GVE), which may be insufficient in policy learning. Furthermore, the authors propose a task-contextual re-fusion procedure to achieve a task-level rebalancing of estimates from feature and value levels. Experimental CARLA benchmark results show the improvement of multi-modal vision-based autonomous driving RL tasks.", "questions": "- The baselines compared in this paper are still not representative and pertinent. This paper mainly focuses on the multi-modal RL task of visual input, but there is no comparison with the classic Vision-based RL work, and there are more suitable baselines than SAC, DrQ, and DeepMDP. For example:\n  - Visual Reinforcement Learning with Imagined Goals, NeurIPS 2018\n  - Improving Sample Efficiency in Model-Free Reinforcement Learning from Images, AAAI 2021\n  - Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation, NeurIPS 2021\n- Observing Figure 3 and Figure 4, it seems that the training curves have not yet converged. If the training process is extended, would HAVE be surpassed by other comparative methods?\n- Why do TransFuser and EFNet only support two modalities, as mentioned in article 4.2?\n- Would using neuromorphic event signals lead to totally fair comparisons? The algorithm's performance may be related to the peculiarity of the modality itself. For example, TransFuser may have its advantages when using the Bird’s Eye View (BEV) modality, but it may be challenging to have a good command of neuromorphic event signals\n- Different local areas of each modality could also have different importance. Can HAVE balance the value of this finer-grained \"modality\"? And how?", "limitations": "The authors explained the limitations of their work well. For example, this work only conducts experiments on RL tasks of multiple visual modalities and does not consider more extensive multimodal input such as text and voice. In addition, the authors explained their potential negative social impact in the appendix. I do not see an obvious negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The originality and novelty of the paper are solid. Inspired by some ideas in algorithms in Multi-Agent RL, an ingenious weighted local value estimation method suitable for multi-modal RL is proposed. And a task contextual re-fusion process is designed to combine better global value estimation and weighted local value estimation to obtain better multi-modal value estimation.\n- The paper is written with high quality. The writing structure is clear and organized, the sentences are fluent and smooth without grammatical errors, and the language style is simple and easy to understand. Although there is no rigorous theoretical analysis, it gives some interesting insights from mathematics. In addition, the code and appendices are well documented and provide useful analysis and explanation.\n- The significance of the paper is fine. Developing better algorithms to solve multi-modal RL tasks is an important research topic.", "weaknesses": "- The experiments for this paper are still not adequate. The entire manuscript has always only conducted experiments on an autonomous driving environment CARLA, and it is difficult to evaluate whether HAVE is suitable for other broader multi-modal RL environments (such as simulated robot control). In addition, this work only conducts experiments on tasks of multiple visual modalities and does not consider more common modal inputs such as text and voice.\n- The paper lacks further theoretical analysis. Although two interesting remarks are in Section 3.6 of the manuscript, the theoretical insight obtained is still relatively shallow. Lack of more in-depth theoretical analyses, such as the overall convergence of the algorithm and whether the algorithm optimizes some lower bounds under the multi-modal RL setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for your time and effort invested in reviewing our work. Your positive feedback has been both encouraging and instrumental for the further improvement of our paper.  Herein, we address the points raised in your comments:\n\n>1. The experiments for this paper are still not adequate due to only with CARLA.  \n\nThanks for highlighting this. Following your advice, we have conducted experiments on an additional environment (DeepMind Control Suite for robot control). Please refer to our general response posted above for detailed results. Experiment results show that HAVE also works effectively in the new environment.\n\n>2. The paper lacks further theoretical analysis.  \n\nThank you for pointing out the need for a deeper theoretical analysis. We recognize the importance of a comprehensive theoretical foundation. The remarks in Section 3.6 were our initial efforts to provide theoretical insights into our approach. However, despite our earnest efforts to provide more theoretical proof, we encountered challenges that prevented us from establishing definitive results at this stage given the constraints of our current rebuttal timeline.\n\nWe believe the empirical results presented do offer a validation of our method's effectiveness in the multi-modal RL setting. We sincerely value your feedback and, in future iterations of this work, aim to delve deeper into the theoretical aspects you've highlighted. Incorporating such a comprehensive analysis would undoubtedly strengthen our work, and we're committed to pursuing this in subsequent research.\n\n>3. The baselines compared in this paper are still not representative and pertinent. \n \nThanks for the valuable suggestion. In Table 3 of our uploaded rebuttal PDF, we give additional results of three baselines. Two of which are based on your suggestion (SAC+AE [1] and SVEA [2]) and the rest (MuMMI[3]) is a multi-modal vision-based RL method. Compare with Table 1 in the main paper, we see that HAVE still possesses advantages over these baselines.\n\n>4. Observing Figure 3 and Figure 4, it seems that the training curves have not yet converged.  \n\nThanks for this astute observation. We conducted further experiments focusing on the figures that seemed to have the most serious convergence issues, namely Fig. 3 (left) and Fig. 4 (bottom left). For each figure, we evaluated two methods that appeared to be the least converged. The findings, presented in Fig.2 (b) and Fig.2 (c) of the rebuttal PDF, indicate convergence by 150K steps, with HAVE consistently outperforming the others. In subsequent revisions, we plan to present results up to 150K steps for all experiments.\n\n>5. Why do TransFuser and EFNet only support two modalities?  \n\nWe apologize for the confusion caused by our previous statement in Sec. 4.2. Our statement was primarily grounded in concerns about computational complexity and practical implementation. For clarity, the cross-modality attention mechanisms embedded within EFNet are designed for paired modalities. In this design, one modality calculates the query tensor, while the other calculates the key and value tensors. We also observed that only two modalities are utilized in both the Transfuser paper and its official implementation. Although it's technically possible to scale both methods to accommodate more than two modalities, such an adaptation would require significant modifications to the implementation, along with a quadratic increase in computational complexity. For instance, with EFNet, a cross-modal attention operation must be performed for every modality pair. In comparison, the complexity added by extra modalities only scales linearly with HAVE. We will revise our statement to articulate this more precisely and prevent further misunderstandings.  \n\n>6. Would using neuromorphic event signals lead to totally fair comparisons?  \n\nWe fully understand your concern. Indeed, it can generally be difficult to disentangle the contribution of an architecture’s modality compatibility from the effectiveness of its multi-modal learning mechanism. To maintain as fair a comparison as possible, for TransFuser and EFNet, we utilized the same observation encoder as with HAVE. With EFNet, we can safely assume there is no compatibility issue, as it is specifically designed for event signals and has proven to be highly effective. TransFuser was originally designed for LiDAR BEV and RGB. However, the architectures of TransFuser and EFNet share considerable similarities (both are transformers with cross-modality attention), and LiDAR BEV has characteristics similar to event frames (e.g., sparsity). Based on these observations, we conjecture that the influence of compatibility might be negligible. In our experiments, TransFuser outperforms EFNet in certain environments, which supports our conjecture. For a more comprehensive comparison of the modality generalization ability of HAVE and TransFuser, we also tested them in DeepMind Control Suite under robot control environments, where the inputs are RGB and depth. Experimental results show that HAVE maintains its advantage over TransFuser. \n\n>7.  Can HAVE balance the value of this finer-grained \"modality\"?  \n\nYes, it can. Specifically, the LVE in HAVE works by maintaining separate critic networks for each modality, allowing it to estimate individual state-values. Each critic in HAVE can therefore assign importance to local areas within its modality by activating only significant regions in the feature maps. As an illustration, we visualize the heatmap of different critic networks in Fig.2(f) in the rebuttal PDF, which aligns with our analysis.\n\n\n**Reference**\n\n[1] Improving Sample Efficiency in Model-Free Reinforcement Learning from Images, AAAI 2021\n\n[2] Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation, NeurIPS 2021\n\n[3] Multi-modal mutual information (MuMMI) training for robust self-supervised deep reinforcement learning.\" ICRA. 2021"}, {"review_id": "RQWzXT6NA3", "reviewer": "Reviewer_HRjH", "summary": "This paper presents a new vision-based reinforcement learning algorithm for autonomous driving scenarios. In previous feature fusion methods, a single critic could amplify dominant features and obscure other features. To address this issue, the authors propose a local critic for each feature and combine them into a global critic using attention fusion. Additionally, the authors introduce a task-contextual re-fusion component to rebalance the estimator. The algorithm achieves promising performance on the CARLA driving task.", "questions": "In Section 4.2, I noticed that the authors mentioned, \"Since TransFuser and EFNet only support two modalities, we directly compare our method with SAC, DrQ, and DeepMDP.\" This suggests that these two baseline algorithms are not suitable for the task environment used in the current paper. Then why use them as strong baselines to compare?", "limitations": "The experimental setup is not sufficiently fair. Although the baselines used for comparison include newer multimodal vision algorithms, they were originally designed for traditional visual tasks. In the tasks used for algorithm validation in this paper, there is a lack of a strong baseline that specifically demonstrates the superiority of the proposed algorithm.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing in the paper is easy to follow, and the figures are clear and understandable.\n2. The proposed algorithm has a simple and effective structure. The improvements made to existing methods in the framework section are intuitive.\n3. The argumentation in the \"Further Analyses\" section of the paper is thorough and provides a good explanation for the potential sources of performance improvement brought by the algorithm.", "weaknesses": "1. The decomposition of the Q-network lacks novelty. Although this paper focuses on vision-based RL, the approach of using a separate critic for each modality and obtaining a global critic using self-attention is similar to some existing methods in the field of multi-agent RL[1]. The paper does not adequately highlight the differences in information processing for vision-based tasks compared to other types of input.\n\n2. The central contribution of the paper lies in the proposed framework. However, apart from the front-end modality fusion and task-specific re-balance, all the content mentioned in section 3.5 appears to be supplementary to existing research[2][3] rather than original improvements. Therefore, I believe there is a lack of substantial contribution in the paper.\n\nref\n[1] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning\n[2] Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\n[3] Deepmdp: Learning continuous latent space models for representation learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your thoughtful review of our paper. Your positive remarks on our writing and analysis are truly appreciated, and we feel encouraged by your feedback. Regarding your concerns about the novelty of our work, we sincerely apologize if our presentation did not sufficiently highlight our unique contributions. We value your feedback and hope the following responses can clarify and address any misunderstandings that might have arisen.\n\n> 1. lacks novelty compared with QMIX.  \n\nThank you for your comments. Your observation regarding the similarity between HAVE and multi-agent is incisive. We kindly ask for your patience and attention as we delve into a more detailed explanation below.  \n\nSpecifically, HAVE intends to answer: “What is the appropriate paradigm to determine state-action values considering the modality-specific contribution?” Through harnessing resources of the RL field itself, we observed a similarity between the roles of “agent'' in multi-agent RL and “modality'' in multi-modal RL: both exhibit distinctive contributions to the final reward and require effective cooperation to optimize the global decision. This resemblance highlights a shared principle across different domains within RL, which by itself is a new kind of discovery not mentioned by existing works. \n\nHowever, the devil is in the details. The subtle yet critical aspects, when examined closely, differentiate our method from the multi-agent ones. Particularly,  our method aims to solve totally different problem compared with QMIX :\n\n\n1. QMIX is proposed for multi-agent cooperative learning tasks where multiple agents share a global or collective reward. QMIX aims to decompose the global Q value for each agent's action, which is further used for training each agent's policy.  \n\n2. The proposed HAVE aims to solve “How to predict Q value of a unified action from a single agent considering the modality-specific contribution under inconstant environment dynamics?” Different with the general multi-modality supervised learning method, the Q value doesn't have true groundtruth during learning, and the current learning goal of Q is related to the current critic. Hence, the feature-level fusion is inadequate, and it is necessary to study how to estimate more accurate Q value by fusing multi-modality estimators. Therefore we have hierarchical and dynamic value fusion. In contrast, QMIX does not consider either the choice of fusion paradigm or bi-level fusion of values.\n\n\n\n> 2. ...all the content mentioned in section 3.5 appears to be supplementary to existing research (SAC[2] and DeepMDP [3]) rather than original improvements.   \n\nOur method focuses on multi-modality reinforcement learning on visual control tasks. SAC and DeepMDP are two well-known and classical baselines on single modality reinforcement learning (RL) and visual presentation learning of RL. It is reasonable and general to study our problem based on them. In fact, there are many approaches that take them as the baseline, such as:\n\n[1] Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. ICLR. 2020.  \n[2] Curl: Contrastive unsupervised representations for reinforcement learning. ICML.2020.  \n[3] Stabilizing deep q-learning with convents and vision transformers under data augmentation. NeurIPS. 2021.  \n[4] Learning invariant representations for reinforcement learning without reconstruction. ICLR. 2021.  \n[5] Masked contrastive representation learning for reinforcement learning. TPAMI 2022.  \n[6] Information Optimization and Transferable State Abstractions in Deep Reinforcement Learning. TPAMI 2022.  \n[7] Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. ICRA. 2021.  \n[8] Isolating and Leveraging Controllable and Noncontrollable Visual Dynamics in World Models. NeurIPS. 2022.  \n[9] Denoised mdps: Learning world models better than the world itself. ICML. 2022.  \n[10] Reinforcement learning with action-free pre-training from videos. ICML, 2022.  \n[11] Masked world models for visual control. CoRL, 2023.\n\nSAC is used as the baseline in [1,2,3,4,5,6], and DeepMDP is used in [7,8,9,10,11]. \n\n\n> 3.  Why use TransFuser and EFNet as strong baselines to compare?  \n\nWe apologize for the confusion caused by our previous statement in Sec. 4.2. Our statement was primarily grounded in concerns about computational complexity and practical implementation. Although it's technically possible to scale both methods to accommodate more than two modalities, such an adaptation would require significant modifications to the implementation, along with a quadratic increase in computational complexity due to cross attention mechanism. \n\nIn addition, most of the experiments in our paper adopted two modalities. Since EFNet and TransFuser are proven very robust with paired modalities, we use them as strong baselines for our comparison following [1-3] and only omit them in our three-modality tests, which is only a small portion of the total experiments.\n\n[1] Safety-enhanced autonomous driving using interpretable sensor fusion transformer. PMLR 2023\n\n[2] Plant: Explainable planning transformers via object-level representations. CoRL 2022\n\n[3] Rgb-event fusion for moving object detection in autonomous driving. ICRA 2023\n\n>4. The experimental setup is not sufficiently fair.  \n\nThanks for your suggestion. We introduced a new baseline named MuMMI [1], which is also designed to manage multiple sensors in reinforcement learning scenarios. The results are detailed in Table 3 of the rebuttal PDF. Relative to other methods, HAVE has a pronounced advantage in performance.    \n\n[1] Multi-modal mutual information (MuMMI) training for robust self-supervised deep reinforcement learning. ICRA. 2021."}]

Author rebuttal:
**To All Reviewers (Additional Experiments on New Environment)**

We thank all the reviewers for their valuable time and insightful feedback. In this general response, we would like to address the concerns about the effectiveness of our HAVE in environments other than autonomous driving. Specifically, we conduct additional experiments using the Mujoco-powered **DeepMind Control Suite** [1] with RGB and depth as input modalities on two challenging tasks (Cheetah, run and Walker, walk). The training and testing curves of HAVE and other comparable methods are given in Fig.1 in the uploaded one-page rebuttal PDF. From the results, it is clear that HAVE also outperforms other methods in these robot control tasks.

**Reference**:

[1] dm_control: Software and tasks for continuous control, Software Impacts, 2020


Meta-review: The paper offers a novel approach to multi-modal reinforcement learning and addresses a significant research area with practical implications. The quality of writing, presentation, and language in the paper is commendable. The authors' commitment to addressing reviewers' feedback through additional experiments and plans for future research demonstrates their dedication to improving the paper.

The AC thinks indeed there are similarities between the attention based combination of value functions in multi-agent RL (Q-Mix) and the idea presented in the paper. But the authors adequately explained this concern in their rebuttal: "a similarity between the roles of “agent'' in multi-agent RL and “modality'' in multi-modal RL: both exhibit distinctive contributions to the final reward and require effective cooperation to optimize the globaldecision. This resemblance highlights a shared principle across diff erent domains within RL, which by itself isa new kind of discovery not mentioned by existing works." The AC thinks the observation that a similar idea of cooperation between different multi-modal features and multi-agent features is novel and interesting.

Regarding the lack of string baselines, the authors also added another baseline in rebuttal.

Given the overall response from reviewers and the authors' proactive approach to addressing concerns, I recommend accepting this paper.
Decision: Accept (poster)