SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our hierarchical adaptive value estimation framework for multi-modal RL process offers several key benefits, which we list as follows:
Remark 1 (Prevention of Modality Dominance): The mechanism of LVE and task-contextual refusion prevents one modality from dominating the others, thereby avoiding the issue of modality collapse.
To realize this, consider that the mixing network  $\mathcal{H}$  in the task-contextual re-fusion performs fixed non-linear transformations on both  $q_{t}^{g}$  and  $q_{t}^{l}$  given  $f_{t}^{g}$ , where  $q_{t}^{l}$  is formed by the convex combination of individual modality values (Eq. 6) as in LVE. Therefore, we have:
$$
\operatorname {s i g n} \left(\frac {\partial \mathcal {H} \left(q _ {t} ^ {g} , q _ {t} ^ {l}\right)}{\partial q _ {t} ^ {M _ {i}}}\right) = \operatorname {s i g n} \left(\frac {\partial \mathcal {H} \left(q _ {t} ^ {g} , q _ {t} ^ {l}\right)}{\partial q _ {t} ^ {l}} \frac {\partial q _ {t} ^ {l}}{\partial q _ {t} ^ {M _ {i}}}\right) = \operatorname {s i g n} \left(\frac {\partial \mathcal {H} \left(q _ {t} ^ {g} , q _ {t} ^ {l}\right)}{\partial q _ {t} ^ {l}}\right), \forall i \in [ 1, 2, \dots , d ], \tag {14}
$$
where  $\mathrm{sign}(\cdot)$  is the real sign function,  $q_{t}^{M_{i}} = Q^{\theta_{M_{i}}} (f_{t}^{M_{i}}, a_{t})$  is the individual modality value and  $\frac{\partial q_{t}^{l}}{\partial q_{t}^{M_{i}}} = w_{t}^{M_{i}} > 0$  are the modality weights. Eq. 14 indicates that the contributions of all modalities are either increasing or decreasing together during the learning process, which prevents opposite gradient values between modalities that enhance some of them while suppressing others. In addition, the calculation of modality weights  $w_{t}^{M_{i}}$  based on softmax global-individual feature similarity (Eq. 9) further prevents modality collapse caused by persistent zero weights.
Remark 2 (Pareto Optimality of Modalities): Assuming continuous action space, the global optimal action produced by our approach achieves a Pareto optimum across individual modalities.
Suppose the optimal action given some fixed modality features is  $a^*$ , which implies  $\frac{\partial \mathcal{H}(q_t^g, q_t^l)}{\partial a^*} = 0$ . Combining Eq. 6 with the chain rule, we have:
$$
\frac {\partial \mathcal {H} \left(q _ {t} ^ {g} , q _ {t} ^ {l}\right)}{\partial a ^ {*}} = \frac {\partial \mathcal {H} \left(q _ {t} ^ {g} , q _ {t} ^ {l}\right)}{\partial q _ {t} ^ {l}} \frac {\partial q _ {t} ^ {l}}{\partial a ^ {*}} = \frac {\partial \mathcal {H} \left(q _ {t} ^ {g} , q _ {t} ^ {l}\right)}{\partial q _ {t} ^ {l}} \sum_ {i = 1} ^ {d} w _ {t} ^ {M _ {i}} \frac {\partial q _ {t} ^ {M _ {i}}}{\partial a ^ {*}} = 0. \tag {15}
$$
Considering the non-linear structure of  $\mathcal{H}$ ,  $\frac{\partial\mathcal{H}(q_t^g,q_t^l)}{\partial q_t^l}$  is unlikely to be zero for all  $q_{t}^{l}$ . This implies that  $\sum_{i = 1}^{d}w_{t}^{M_{i}}\frac{\partial q_{t}^{M_{i}}}{\partial a^{*}} = 0$  for all  $i = 1,2,\ldots ,d$ , which suggests that at the point  $a^*$ , the positive weighted sum of the gradients of individual modalities' values is zero. This implies that there is no alternative action assignment in the vicinity of  $a^*$  that can increase the value for any individual modality without decreasing the value for some other modality. Such condition forms a Pareto optimum across the modalities, which indicates the resources (modality values) are allocated in the most efficient way possible [35].
**Environments** To evaluate our approach under realistic and challenging multi-modal environments, we employ the CARLA simulator [8], which is a widely used open-source platform for autonomous driving research. CARLA provides a rich and realistic urban environment to evaluate autonomous driving agents in various traffic scenarios. Three distinct modalities are adopted: 1) RGB frames, 2) event signals generated by CARLA's synthetic event-camera simulator, and 3) per-pixel depth frames. Eight different weather settings are used to provide thorough coverage of different environmental conditions. The action space consists of continuous control actions, such as steering, acceleration, and braking. Similar to [52, 9], the reward function is designed to encourage the agent to maintain a safe distance from 20 other moving vehicles and obstacles, and drive as far as possible along the highway of CARLA's Town04 map in 1000 time steps. We use the single camera view setting on the vehicle's roof with 60-degree views.
Implementation Details Our approach is implemented based on SAC [14, 15] and DeepMDP [13]. The same encoder network architecture and training hyperparameters are adopted for all comparative methods. Following common practices [50, 23], we convert each modality data into its corresponding image-based representations and stack several consecutive images to infer temporal information. The spatial resolution of the input images is  $128 \times 128$  and the channel numbers of RGB, event and depth frames are 3, 5 and 1, respectively. All methods are trained for 120k frames across 5 random seeds to report the mean and standard deviation of the rewards. We evaluate the performance of each approach in terms of driving episode reward and distance. Other details are provided in the supplementary material.
Weather	Measures	SAC	DrQ	DeepMDP	TransFuser	EFNet	Ours-LVE	Ours-HAVE
ClearNight	ER	186± 65	242± 84	225± 87	260± 95	241± 89	274± 68	319± 71
D(m)	112± 39	169± 41	161± 51	178± 43	170± 62	192± 50	212± 52
CloudyNight	ER	218± 71	248± 69	265± 85	280± 71	289± 64	295± 67	322± 96
D(m)	132± 64	167± 36	183± 55	195± 43	197± 41	213± 44	217± 53
HardRainNight	ER	170± 90	261± 91	255± 77	287± 65	275± 72	294± 85	327± 87
D(m)	107± 63	163± 63	160± 36	204± 56	203± 54	209± 54	229± 51
WetNight	ER	189± 79	234± 97	241± 47	274± 76	290± 93	289± 112	304± 102
D(m)	127± 57	155± 53	164± 36	171± 58	201± 56	196± 61	222± 69
ClearNoon	ER	235± 58	280± 94	269± 64	282± 53	234± 79	294± 82	336± 76
D(m)	153± 49	195± 40	187± 36	193± 39	150± 46	186± 61	223± 44
CloudyNoon	ER	201± 87	274± 77	226± 24	277± 67	261± 78	293± 76	315± 82
D(m)	138± 68	170± 42	136± 16	171± 42	164± 58	186± 44	209± 68
HardRainNoon	ER	189± 74	220± 72	248± 59	264± 99	279± 91	287± 95	316± 88
D(m)	104± 56	129± 60	161± 43	178± 62	208± 76	207± 69	218± 63
WetNoon	ER	209± 81	245± 83	226± 52	304± 81	273± 82	296± 84	341± 78
D(m)	136± 64	172± 58	169± 38	213± 51	204± 70	215± 54	239± 55
Table 1: Comparison with state-of-the-art methods on eight different kinds of weather. ER denotes episode return and D is distance in meters. The best results are bolded and the second best results are underlined.
4.2 Comparison with State of the Art
We compare our method with a variety of methods, including the SAC [15] baseline, DeepMDP [13], DrQ [50], TransFuser [7], and EFNet [43]. For RL methods SAC, DrQ and DeepMDP, we directly concatenate the features of the different modalities as the input for subsequent value and policy learning. For TransFuser and EFNet which are designed for traditional visual tasks, we only adopt their advanced modality fusion modules and keep DeepMDP as the RL algorithm for a fair comparison.
Results of Two Modalities We first evaluate different methods under two modality inputs (RGB frames and event signals) in Table 1. The results show that our method achieves the highest episode reward and driving distance under all eight weather conditions. DrQ and DeepMDP perform better than the SAC baseline with limited improvement, showing that the feature-fusion based GVE paradigm, together with simple feature concatenation, cannot fully extract the expressive power of each modality. The improved performance of TransFuser and EFNet over DeepMDP reflects the importance of the advanced modality feature fusion mechanism. However, they are still inferior to ours-LVE, which uses the proposed LVE paradigm to explicitly assign modality contributions weights with value-based fusion. The results indicate that the key to multi-modal RL is to consider the suitability of individual modalities. Finally, by using the task-contextual re-fusion process to integrate GVE and LVE, our full method achieve superior performance even with simple feature concatenation fusion, proving the effectiveness of our task-driven hierarchical design.
Results of Multiple Modalities We then evaluate our method on three modalities (RGB frames, event signals, and depth frames). Although TransFuser and EFNet technically possible to scale both methods to accommodate more than two modalities, such an adaptation would require significant modifications to the implementation, along with a quadratic increase in computational complexity due to cross attention mechanism. Thus we directly
Figure 3: Performance with three modalities.
compare our method with SAC, DrQ and DeepMDP. The training curves under two weather conditions are demonstrated in Fig. 3, which also show the advantage of our methods. Additional experiment results can be found in the supplementary material.
4.3 Performance Analysis
Ablation Studies To systematically evaluate the effectiveness of the proposed HAVE approach and its individual components, we conduct a series of ablation experiments on the CARLA benchmark that trains agents using: 1) single modality data with RGB frames or event signals, 2) GVE with feature concatenation, 3) our proposed LVE paradigm, and 4) our full method. The training curves and testing performance are shown in Fig. 4 and Tab. 2, respectively. First, we observe that the indi
Figure 4: Training curves of individual modalities and different multi-modal training paradigms.
Methods	Measures	Night	Noon
Clear	Cloudy	HardRain	Wet	Clear	Cloudy	HardRain	Wet
RGB	ER	75 ± 5	194 ± 36	189 ± 17	88 ± 23	214 ± 55	201 ± 37	223 ± 67	232 ± 74
D(m)	46 ± 21	121 ± 24	136 ± 15	53 ± 26	125 ± 42	121 ± 25	128 ± 51	141 ± 54
Events	ER	158 ± 29	187 ± 32	164 ± 28	175 ± 29	145 ± 37	151 ± 41	154 ± 34	147 ± 40
D(m)	102 ± 23	129 ± 26	105 ± 21	120 ± 18	86 ± 27	92 ± 32	104 ± 21	86 ± 36
GVE	ER	225 ± 87	265 ± 85	255 ± 77	241 ± 47	269 ± 64	226 ± 24	248 ± 59	226 ± 52
D(m)	161 ± 51	183 ± 55	160 ± 36	164 ± 36	187 ± 36	136 ± 16	161 ± 43	169 ± 38
LVE	ER	274 ± 68	295 ± 67	294 ± 85	289 ± 112	294 ± 82	293 ± 76	287 ± 95	296 ± 84
D(m)	192 ± 50	213 ± 44	209 ± 54	196 ± 61	186 ± 61	186 ± 44	207 ± 69	215 ± 54
HAVE	ER	319 ± 71	322 ± 96	327 ± 87	304 ± 102	336 ± 76	315 ± 82	316 ± 88	341 ± 78
D(m)	212 ± 52	217 ± 53	229 ± 51	222 ± 69	223 ± 44	209 ± 68	218 ± 63	239 ± 55
Table 2: Test performance of different models in Fig. 4. Table notations are the same as in Table 1.
Figure 5: Visualization of modality weights during testing. Note that only weights of RGB frames are drawn and the weights of event signals are one minus RGB frame weights at any time step.
vidual modality is vulnerable to failure under extreme weather conditions. For example, the episode reward trained with RGB frame under ClearNight weather is pretty poor as shown in Fig. 4. Second, directly fusing the two modalities with GVE generally improves performance. However, the performance gain is limited, and in some cases even close to the results of using a single modality (e.g., in WetNight and ClearNoon). This might be caused by the heterogeneity of multi-modal data, which leads to difficulty in determining the target reward. Third, in most cases, LVE performs significantly higher than single modality or GVE-based agents, showing better cooperation of different modality data. Finally, our full method with task-contextual re-fusion clearly outperforms all other models thanks to the synergistic interplay of GVE and LVE at the reward/task-level.
Visualization To further obtain an intuitive understanding of our approach, we visualize the modality weights obtained by HAVE in Fig. 5 using RGB frames and event signals under two different weather conditions. The following observations can be made: 1) at the beginning, all vehicles have zero initial speed. As a result, no valid event signal is generated and the weights of RGB modality are nearly 1.0. 2) After speeding up, the weights of event signals are continuously increasing, especially when there are many moving vehicles ahead. 3) However, when the agent is facing complex background (such as road fences and clear roads without close vehicles), the event signals mainly consist of background noise, and the weights of RGB frames will raise again. The varying modality weights show that our approach can indeed adjust modality contributions under different situations, which serves as a key advantage over previous multi-modal RL methods.
We have studied the representational capacity of value functions in multi-modal vision-based RL problems. We hypothesize that the limitation of feature-level fusion methods may come from unclear contributions for different modalities under a single value function. To mitigate this, we have presented a novel Hierarchical Adaptive Value Estimation (HAVE) framework to reconcile both feature-level and value-level fusion in a task/reward-driven manner. Our approach represents one of the first explorations of modality-specific and hierarchical value estimation for multi-modal vision-based RL tasks. Extensive experiment results demonstrate the effectiveness of our approach. However, one limitation of our work is that we mainly focus on the value estimation of multiple visual modalities, while the effectiveness of other forms of modalities (e.g., audio, text) is not verified. In our future work, we will consider utilizing both visual and other modalities, forming a more generalized multi-modal RL framework.
The study was funded by the Key-Area Research and Development Program of Guangdong Province with contract No. 2021B0101400002, the National Natural Science Foundation of China under contracts No. 62027804, No. 61825101, No. 62088102 and No. 62202010, and the major key project of the Peng Cheng Laboratory (PCL2021A13). Computing support was provided by Pengcheng Cloudbrain.
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[2] Andrea Banino, Adrià Puidomenech Badia, Jacob Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. Coberl: Contrastive bert for reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2022.
[3] Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Mtbaa. A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. Visual Computer, 2022.
[4] Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, and Dhruv Batra. Embodied multimodal multitask learning. In Proceedings of the International Joint Conference on Artificial Intelligence, 2020.
[5] Guang Chen, Hu Cao, Jorg Conradt, Huajin Tang, Florian Rohrbein, and Alois Knoll. Event-based neuro-morphic vision for autonomous driving: A paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Processing Magazine, 2020.
[6] Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. In IEEE Proceedings of the International Conference on Robotics and Automation. IEEE, 2021.
[7] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[8] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning. PMLR, 2017.
[9] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In Proceedings of the International Conference on Machine Learning, 2021.
[10] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems, 2020.
[50] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In Proceedings of the International Conference on Learning Representations, 2020.
[51] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, 2022.
[52] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020.
[53] Jiaping Zhang, Tiancheng Zhao, and Zhou Yu. Multimodal hierarchical reinforcement learning policy for task-oriented visual dialog. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue, 2018.
[54] Yue Zhao, Chenzhuang Du, Hang Zhao, and Tiejun Li. Intrinsically motivated self-supervised learning in reinforcement learning. In Proceedings of the International Conference on Robotics and Automation. IEEE, 2022.