SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yangru Huang $^{1}$ , Peixi Peng $^{2,3*}$ , Yifan Zhao $^{1}$ , Haoran Xu $^{3,4}$ , Mengyue Geng $^{1}$ , Yonghong Tian $^{1,2,3*}$
$^{2}$ School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University  $^{3}$ Peng Cheng Laboratory
$^{4}$ School of Intelligent Systems Engineering, Sun Yat-sen University, yrhuang@stu.pku.edu.cn,
{pxpeng, zhaoyf, mygeng, yhtian}@pktu.edu.cn,xuhr9@mail2.sysu.edu.cn
Integrating RGB frames with alternative modality inputs is gaining increasing traction in many vision-based reinforcement learning (RL) applications. Existing multi-modal vision-based RL methods usually follow a Global Value Estimation (GVE) pipeline, which uses a fused modality feature to obtain a unified global environmental description. However, such a feature-level fusion paradigm with a single critic may fall short in policy learning as it tends to overlook the distinct values of each modality. To remedy this, this paper proposes a Local modality-customized Value Estimation (LVE) paradigm, which dynamically estimates the contribution and adjusts the importance weight of each modality from a value-level perspective. Furthermore, a task-contextual re-fusion process is developed to achieve a task-level re-balance of estimations from both feature and value levels. To this end, a Hierarchical Adaptive Value Estimation (HAVE) framework is formed, which adaptively coordinates the contributions of individual modalities as well as their collective efficacy. Agents trained by HAVE are able to exploit the unique characteristics of various modalities while capturing their intricate interactions, achieving substantially improved performance. We specifically highlight the potency of our approach within the challenging landscape of autonomous driving, utilizing the CARLA benchmark with neuromorphic event and depth data to demonstrate HAVE's capability and the effectiveness of its distinct components. The code of our paper can be found at https://github.com/Yara-HYR/HAVE.
Recent years have witnessed a renewed interest in multi-modal perception in the computer vision research community [10, 36, 20, 26, 37]. For many visual tasks such as semantic segmentation and object detection, the inclusion of multi-modal data (e.g., depth, infrared) is proven to be indisputably beneficial [41, 29, 49, 25]. This trend equally applies to the intelligent agents in vision-based Reinforcement Learning (RL), in which multi-modal inputs can also promote decision robustness [39, 19, 1, 11, 31]. For instance, an autonomous-driving agent taking RGB frames solely as input may frequently suffer from extreme light conditions, as shown in Fig. 1(a). Nevertheless, combining additional sensory inputs such as event signals coming from a neuromorphic event camera [27] can effectively alleviate these problems, enabling a more comprehensive realization of traffic status [5, 32, 12].
RGB Events
(a) Under-exposed RGB frames vs. HDR event signals
(c) Global Value Estimation (GVE)
Environment
Figure 1: Top: Desired modality contributions of (a) under-exposed RGB frame vs. high dynamic range (HDR) event signals, and (b) clear RGB frame vs. event signals saturated by background noise. Bottom: Multi-modal fusion at (c) the feature level, where only the fused modality feature is used for value evaluation, and (d) the value level, where individual value estimation is conducted for each modality to identify which one performs better under the current circumstance.
(b) Clear RGB frames vs. Noisy event signals
(d) Local modality-customized Value Estimation (LVE)
RGB latent features
Events latent features  $\rightarrow$  Feedback  $\rightarrow$  Forward
Despite the abundance of fruitful studies in the field of multi-modal visual perception, a majority of them focus on traditional static learning tasks. In contrast, the dynamic and unlabeled nature of RL renders the development of multi-modal agents exceedingly difficult [6]. To reconcile multisensory data, existing methods [18, 36, 6, 22, 40] train the agents with fused modality features and form a mixed global modality, as shown in Fig.1(c). The policy is then learned via Global Value Estimation (GVE), in which the global modality feature is responsible for environment description and then paired with action to compute a single Q-value. Although being an effective and widely adopted paradigm, utilizing GVE alone may overlook the distinct task-related contributions of different modalities. Specifically, due to the diverse attributes of sensors, single modalities may not contribute equally under multiple environmental conditions (e.g., the cases given in Fig. 1 (a) and (b)). The feature-level fusion process in GVE, no matter how sophisticated, inevitably disregards the unique value of each sensory data since only a single critic is shared over multiple modalities. As a result, negative interference between modalities may occur and compromise learning performance.
Based on the above analysis, we propose Local modality-customized Value Estimation (LVE), as illustrated in Fig. 1 (d). Different from feature-level fusion, LVE learns a single policy with distinct, per-modality value calculation, forming a value-level fusion paradigm through a tailor-designed Q-value weighting process. As a result, the modality-specific contributions can be explicitly estimated, which promotes policy flexibility. In addition, while LVE is intrinsically a better alternative than GVE, the two paradigms are not competitive but complementary. Considering this, we further develop a task-contextual re-fusion process, which utilizes an efficient fusing network guided directly by the task reward to reach a task-level balance between LVE and GVE. The collaboration of the above components eventually forms a Hierarchical Adaptive Value Estimation (HAVE) framework for multi-modal vision-based RL, yielding a potent policy that can flexibly leverage the unique strengths of each modality while profiting from the comprehensive information from all modalities. We specifically highlight the potency of our approach to the challenging autonomous driving task. In particular, we explicitly consider the neuromorphic event camera [27] signals that capture the motion information of the environment, which well-suites our task but has not yet been explored by existing multi-modal RL algorithms.
In summary, the contributions of our work are three-fold: 1) We design a novel hierarchical adaptive value estimation (HAVE) framework for multi-modal vision-based RL. HAVE distinctively features a local modality-customized value estimation (LVE) paradigm to enable optimized reward allocation based on modality importance. 2) We develop a task-contextual re-fusion process to merge the profi
ciencies of LVE and GVE, allowing HAVE to benefit from both particularized and unified modality values. 3) Our approach achieves state-of-the-art performance on challenging autonomous driving tasks. As the first multi-modal RL algorithm considering event camera signals as one of the input modalities, our approach exhibits superior performance under various environmental conditions, demonstrating its potential as an effective multi-modal vision-based RL solution.
Vision-based RL aims to train agents that receive raw image-based observations from environments for decision-making. Compared to state-based RL, vision-based RL has found significant use in many practical tasks, from video game playing [34] to robotic manipulation [21]. However, learning policy directly from such high-dimensional input is challenging. To tackle the problem, a considerable amount of works have been developed, including 1) applying data augmentation to increase the diversity of samples [50, 24, 30], 2) introducing auxiliary tasks such as contrastive loss [23, 54, 2], 3) pretraining an encoder to improve the representational ability [51, 28, 42], and 4) modeling environment dynamics in the latent space [17, 16, 38]. Although most vision-based RL methods adopt RGB camera frames as inputs, some recent works have also started to explore new sensors and data formats for RL, such as event cameras [46, 47] which captures fast and asynchronous light changes with high temporal precision.
Multi-modal Visual Learning has been extensively studied in the field of computer vision [3]. With the development of sensor technologies, it becomes effortless to acquire sufficient data from complemented visual modalities (e.g., RGB, infrared, depth, and event signals). As a result, many multi-modal learning methods are proposed for traditional visual tasks, such as object detection [29] and segmentation [10, 25]. For multi-modal RL, the agent's observation space is modified to include all modalities. Recent works have started to focus on multi-modal vision-based RL due to its improved effectiveness compared with using only single-modality data. Chen et al. propose a multi-modal state-space model trained with mutual information lower-bound to promote the consistency between the latent codes of each modality [6]. A fusion network is proposed by Khalil et al. to produce accurate joint multi-modal perception and motion prediction for autonomous driving [22]. Ma et al. propose a multi-modal RL approach that focuses on modality alignment and importance enhancement [31]. There are also multi-modal RL methods designed for other tasks such as robot control [4] and dialog system [33, 53]. Despite their distinct technical details, most of the existing methods perform a feature-level fusion of modalities and ignore the properties of RL itself. In fact, policy learning depends on the value estimation of the critic. Consequently, we view the multi-modal visual RL problem from a novel adaptive value estimation perspective. Instead of concentrating solely on robust global modality features, our approach synergistically balances individual modalities' contributions, leading to a more equitable and efficient value allocation. Leveraging the task-contextual re-fusion mechanism, our method further capitalizes on both feature-based and value-based fusion paradigms, resulting in a more robust policy.
Problem Definition We formulate the task of multi-modal vision-based RL as a Markov Decision Process (MDP) [44] with multiple observations, which is defined by a tuple  $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$ , where  $\mathcal{S} = \prod_{i=1}^{d}\mathcal{O}^{M_i}$  is the joint observation space of  $d$  modalities and  $\mathcal{O}^{M_i}$  is the observation space of modality  $i$ .  $\mathcal{A}$  is all possible actions the agent can take.  $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0,1]$  is the transition probability function,  $\mathcal{P}(o_{t+1}|o_t, a_t)$  denotes the probability of transitioning from joint observation  $o_t = (o_t^{M_1}, o_t^{M_2}, \ldots, o_t^{M_d})$  at time step  $t$  to the next joint observation  $o_{t+1} = (o_{t+1}^{M_1}, o_{t+1}^{M_2}, \ldots, o_{t+1}^{M_d})$  after taking action  $a_t$  from a policy function  $\pi$ .
Soft Actor-Critic (SAC) Our approach is built on SAC [14, 15], which goal is to learn a policy that maximizes the expected cumulative reward while maintaining exploration by encouraging diverse actions. In SAC, the objective function is given by introducing a policy entropy term:
$$
\mathcal {L} _ {\pi} = \mathbb {E} _ {a _ {t} \sim \pi} [ Q (o _ {t}, a _ {t}) - \alpha \log \pi (a _ {t} | o _ {t}) ], \tag {1}
$$
(a) GVE
(b) LVE
(c) HAVE
Figure 2: HAVE framework and its distinct components. (a) Global Value Estimation (GVE), (b) Local modality-customized Value Estimation (LVE), and (c) HAVE with task-level value re-fusion.
where  $Q$  is the value function, and  $\alpha$  is a temperature parameter that controls the trade-off between exploration and exploitation. The value function is trained using the Bellman equation and a soft Q-function update:
$$
\mathcal {L} _ {Q} = \mathbb {E} _ {\left(o _ {t}, a _ {t}\right) \sim \mathcal {D}} \left[ \left(Q \left(o _ {t}, a _ {t}\right) - \left(\mathcal {R} \left(o _ {t}, a _ {t}\right) + \gamma \mathbb {E} _ {o _ {t + 1} \sim p} [ V \left(o _ {t + 1}\right) ]\right)\right) ^ {2} \right], \tag {2}
$$
where  $o_t$  and  $a_t$  are sampled from the replay buffer  $\mathcal{D}$  and  $\gamma$  is the discount factor.  $V(o_{t + 1})$  is the soft state value function, defined as:
$$
V \left(o _ {t + 1}\right) = \mathbb {E} _ {\tilde {a} _ {t + 1} \sim \pi} \left[ \bar {Q} \left(o _ {t + 1}, \tilde {a} _ {t + 1}\right) - \alpha \log \pi \left(\tilde {a} _ {t + 1} \mid o _ {t + 1}\right) \right], \tag {3}
$$
where  $\bar{Q}$  denotes an exponential moving average of the critic network  $Q$  and  $\tilde{a}_{t + 1}$  comes from the current policy.
Event Cameras and Representations Despite RGB frames, another major modality that we use to evaluate our approach is the event signals generated by a neuromorphic event camera [27]. Each pixel in the event camera outputs a positive/negative event signal whenever the log light intensity of that pixel has increased/decreased by a constant threshold. The event signals have an extremely high dynamic range (up to  $120\mathrm{dB}$ ) and can reach a high temporal resolution in the order of  $\mu s$ . Therefore, they are able to capture the missing motion clues that are missed in the RGB frames for many visual tasks such as autonomous driving and robot navigation. To utilize event signals, we use the stacking based on time (SBT) [48] representation that splits the event sequence into fixed temporal bins, forming an event frame representation with multiple channels similar to RGB frames.
3.2 Global Value Estimation
In Global Value Estimation (GVE), a total of  $d$  observation encoders are used to extract observation features  $f_{t}^{M_{i}}$  for each modality observation  $o_{t}^{M_{i}}$  at time step  $t$ . A global modality feature  $f_{t}^{g}$  is then calculated by fusing all modality features. For simplicity, we directly concatenate  $f_{t}^{M_{i}}(i = 1,2,\ldots ,d)$  along the channel dimension. Given  $f_{t}^{g}$ , GVE first computes the value estimation  $q_{t}^{g} = Q^{\theta_{g}}(f_{t}^{g},a_{t})$  using a single global critic network  $Q^{\theta_g}$ , as shown in Fig. 2(a). Following the SAC algorithm, the target value  $y_{t}^{g}$  is then calculated as:
$$
y _ {t} ^ {g} = \mathcal {R} \left(o _ {t}, a _ {t}\right) + \gamma V \left(f _ {t + 1} ^ {g}\right), \tag {4}
$$
where  $\mathcal{R}(o_t, a_t)$  is the reward returned by the environment. The soft state value function  $V(f_{t+1}^g)$  is denoted as:
$$
V \left(f _ {t + 1} ^ {g}\right) = \mathbb {E} _ {\tilde {a} _ {t + 1} \sim \pi} \left[ \bar {Q} ^ {\theta_ {g}} \left(f _ {t + 1} ^ {g}, \tilde {a} _ {t + 1}\right) - \alpha \log \pi \left(\tilde {a} _ {t + 1} \mid f _ {t + 1} ^ {g}\right) \right]. \tag {5}
$$
However, since using a single value function  $Q^{\theta_g}$  shared over all modalities, it is hard to dynamically balance the relative contribution of each  $f_t^{M_i}$  under complex environmental conditions.
3.3 Local Modality Customized Value Estimation
The objective of Local modality-customized Value Estimation (LVE) is to effectively differentiate and quantify the unique contribution of each modality, thereby facilitating a more granular and modality-aware decision-making process. Specifically, in LVE,  $d$  individual value functions  $Q^{\theta_{M_1}}, Q^{\theta_{M_2}}, \ldots, Q^{\theta_{M_d}}$  are setup instead of global-only functions in GVE. These value functions share the same network architecture but are learned separately. To facilitate value decomposition, an assignment network is designed to assimilate all estimated values and re-calibrate them according to the collective environmental condition portrayed by modality features.
Value Inference: We assume that the locally-aggregated total value  $q_{t}^{l}$  can be approximately decomposed into a linear combination of the separate value functions across different modalities under a shared policy:
$$
q _ {t} ^ {l} \approx \sum_ {i = 1} ^ {d} w _ {t} ^ {M _ {i}} Q ^ {\theta_ {M _ {i}}} \left(f _ {t} ^ {M _ {i}}, a _ {t}\right), \tag {6}
$$
where  $w^{M_i}$  denotes the contribution weight of modality  $M_{i}$  to the total value. The value function  $Q^{\theta_{M_i}}$  for each modality processes the current individual modal observation feature  $f_{t}^{M_{i}}$  along with the current action  $a_{t}$ , generating the estimated action values at each time step. Each  $Q^{\theta_{M_i}}$  is learned by backpropagating gradients from the Q-learning rule. Similarly, the total target value  $y_{t}^{l}$  can be defined as:
$$
y _ {t} ^ {l} \approx \mathcal {R} \left(o _ {t}, a _ {t}\right) + \gamma V \left(f _ {t + 1} ^ {M _ {1}}, \dots , f _ {t + 1} ^ {M _ {d}}\right). \tag {7}
$$
The soft state value function  $V(f_{t + 1}^{M_1},\dots,f_{t + 1}^{M_d})$  is then defined as:
$$
V \left(f _ {t + 1} ^ {M _ {1}}, \dots , f _ {t + 1} ^ {M _ {d}}\right) = \mathbb {E} _ {\tilde {a} _ {t + 1} \sim \pi} \left[ \sum_ {i = 1} ^ {d} v _ {t + 1} ^ {M _ {i}} \bar {Q} ^ {\theta_ {M _ {i}}} \left(f _ {t + 1} ^ {M _ {i}}, \tilde {a} _ {t + 1}\right) - \alpha \log \pi \left(\tilde {a} _ {t + 1} \mid f _ {t + 1} ^ {g}\right) \right], \tag {8}
$$
where  $v_{t}^{M_{i}}$  is the contribution weight of modality  $M_{i}$  to the estimated Q-value from  $\bar{Q}^{\theta_{M_i}}$  at the next time.  $\tilde{a}_{t + 1}\sim \pi (\cdot |f_{t + 1}^{g})$  comes from the current policy according to the global modality feature  $f_{t + 1}^{g}$  since there is only one policy network for the multi-modal decision task.
Contribution Assignment: Given value decomposition in Eq. 6 and Eq. 7, a critical next step is to calculate accurate modality weights  $w_{t}^{M_{i}}$  and  $v_{t}^{M_{i}}$  to ensure precise total value estimation. To achieve this, the contribution assignment process involves modality interactions through an assignment network based on the attention mechanism. Taking the calculation of  $w_{t}^{M_{i}}$  as an example, the global modality feature  $f_{t}^{g}$  is used as a bridge to ensure information flow between all individual modalities. As demonstrated in Fig. 2(b), two fully connected (FC) layers with parameters  $W_{q}$  and  $W_{k}$  are used to project both  $f_{t}^{M_{i}}$  and  $f_{t}^{g}$  into a  $p$ -dimensional common latent subspace. Then  $w_{t}^{M_{i}}$  can be obtained through a softmax function as:
$$
w _ {t} ^ {M _ {i}} = \frac {\exp \left(\left(W _ {q} f _ {t} ^ {g}\right) \left(W _ {k} f _ {t} ^ {M _ {i}}\right) ^ {\top} / \sqrt {p}\right)}{\sum_ {i = 1} ^ {d} \exp \left(\left(W _ {q} f _ {t} ^ {g}\right) \left(W _ {k} f _ {t} ^ {M _ {i}}\right) ^ {\top} / \sqrt {p}\right)}, \tag {9}
$$
where  $\sqrt{p}$  is used for scaling to prevent vanishing gradients [45]. The computation of  $v_{t + 1}^{M_i}$  follows a similar derivation, which is omitted here for brevity.
By assigning the value function in a modality-customized manner, LVE adeptly manages multimodal sensory inputs, enhancing model efficiency and interpretability. Our approach readily adapts to complex environmental scenarios, accommodating varying modality importance levels.
3.4 Task-Contextual Re-fusion
Feature-based vs. Value-based Fusion From the details of GVE and LVE, we see that the main difference distinguishing them is their fusion principle (feature-based vs. value-based). Intuitively, the two paradigms are not competitive but complementary. Specifically, in the context of RL, value-based fusion offers significant advantages over feature-based fusion since the former is more related to the actual decision. In the meantime, the feature-based fusion method can use its collective modality value to offer a value-based one with a sturdy global reference, mitigating inaccuracies caused by input noise. Therefore, we further develop task-contextual re-fusion for a subsequent re-fuse of GVE with LVE in the reward/task level. The idea is to directly bridge the two fusion mechanisms
based on environmental reward feedback, allowing the task to determine which paradigm better suits the current situation.
Details of the Re-fusion Process Given the estimated value  $q_{t}^{g}$  and  $q_{t}^{l}$  obtained from GVE and LVE, respectively, we adopt a dynamic fusion mechanism with a re-fusion network  $\mathcal{H}$  to fuse  $q_{t}^{g}$  and  $q_{t}^{l}$ . Specifically,  $\mathcal{H}$  consists of a series of FC layers and hypernetworks. Each hypernetwork consists of a single linear layer, which receives the global modality feature  $f_{t}^{g}$  as input and produces a weight matrix or a bias vector for a single FC layer in  $\mathcal{H}$ . To fuse  $q_{t}^{g}$  and  $q_{t}^{l}$ ,  $f_{t}^{g}$  is first sent into all hypernetworks to obtain the parameters of the FC layers in  $\mathcal{H}$ , then  $q_{t}^{g}$  and  $q_{t}^{l}$  are sent into  $\mathcal{H}$  to get the final total value  $\mathcal{H}(q_{t}^{g}, q_{t}^{l})$ . The final target value can be obtained using a similar way, wherein the estimated Q-value from the target Q-network at the next time is processed by another mixing network  $\mathcal{H}'$ . For brevity, we denote the this target value as  $\mathcal{H}'(y_{t}^{g}, y_{t}^{l})$ .
3.5 Learning Framework
Coping GVE with LVE and task-contextual re-fusion, our approach forms a unified framework, which we name as Hierarchical Adaptive Value Estimation (HAVE) for multi-modal vision-based RL. In this section, we elaborate on the training details of our HAVE framework.
Policy Evaluation with HAVE For the training of policy evaluation, we minimize the temporal difference (TD) error between the predicted value  $\mathcal{H}(q_t^g,q_t^l)$ , and the target value  $\mathcal{H}'(y_t^g,y_t^l)$  following the SAC algorithm:
$$
\mathcal {L} _ {Q} = \mathbb {E} _ {\left(o _ {t}, a _ {t}\right) \sim \mathcal {D}} \left[ \left(\mathcal {H} \left(q _ {t} ^ {g}, q _ {t} ^ {l}\right) - \mathcal {H} ^ {\prime} \left(y _ {t} ^ {g}, y _ {t} ^ {l}\right)\right) ^ {2} \right], \tag {10}
$$
where  $\mathcal{D}$  denotes the replay buffer.
Policy Improvement The policy evaluation described above can cover reasonable Q-values to help find better policies. In the policy improvement step, we leverage the values obtained from GVE, LVE, and the task-contextual re-fusion process to update the policy  $\pi$ . To be specific, the goal is to maximize the expected cumulative reward by selecting actions that have the highest combined value estimates:
$$
\mathcal {L} _ {\pi} = \mathbb {E} _ {a _ {t} \sim \pi} \left[ \mathcal {H} \left(q _ {t} ^ {g}, q _ {t} ^ {l}\right) - \alpha \log \pi \left(a _ {t} \mid f _ {t} ^ {g}\right) \right]. \tag {11}
$$
After identifying the optimal action under the improved policy, we can update the policy parameters by backpropagating gradients.
Auxiliary Losses Besides policy evaluation and improvement, we also learn to predict the rewards and next latent states as in DeepMDP [13] to assist representation learning, providing a latent state space that is consistent with environmental dynamics:
$$
\mathcal {L} _ {a u x} = \left\| \mathcal {P} ^ {g} \left(f _ {t} ^ {g}, a _ {t}\right) - f _ {t + 1} ^ {g} \right\| ^ {2} + \sum_ {i = 1} ^ {d} \left\| \mathcal {P} ^ {M _ {i}} \left(f _ {t} ^ {M _ {i}}, a _ {t}\right) - f _ {t + 1} ^ {M _ {i}} \right\| ^ {2} + \left[ \mathcal {R} ^ {g} \left(f _ {t} ^ {g}, a _ {t}\right) - \mathcal {R} \left(o _ {t + 1}, a _ {t + 1}\right) \right] ^ {2}, \tag {12}
$$
where  $\mathcal{P}^g$  and  $\mathcal{P}^{M_i}(i = 1,2,\dots ,d)$  are transition networks for the global and individual modalities, respectively.  $\mathcal{R}^g$  is the reward prediction network. All these networks are formed by FC layers and a more detailed architecture description can be found in the supplementary material.
Overall Training Objective Given the above training objectives, the overall loss of HAVE is finally defined by:
$$
\mathcal {L} = \mathcal {L} _ {Q} + \mathcal {L} _ {\pi} + \mathcal {L} _ {a u x}. \tag {13}
$$
By iteratively performing policy evaluation and improvement steps, HAVE can effectively exploit the strengths of both global and distinct value estimates, ultimately converging to an improved policy that can handle the challenges posed by a broad range of environmental modalities.
3.6 Further Analyses