Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving

Authors: kairui  ding
boyuan  chen
yuchen  su
huan-ang  gao
bu  jin
chonghao  sima
wuqiang  zhang
xiaohui  li
paul  barsch
hongyang  li
hao  zhao
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-23 08:24:16

Abstract: End-to-end architectures in autonomous driving (AD) face a significant challenge in interpretability, impeding human-AI trust. Human-friendly natural language has been explored for tasks such as driving explanation and 3D captioning. However, previous works primarily focused on the paradigm of declarative interpretability, where the natural language interpretations are not grounded in the intermediate outputs of AD systems, making the interpretations only declarative. In contrast, aligned interpretability establishes a connection between language and the intermediate outputs of AD systems. Here we introduce Hint-AD, an integrated AD-language system that generates language aligned with the holistic perception-prediction-planning outputs of the AD model. By incorporating the intermediate outputs and a holistic token mixer sub-network for effective feature adaptation, Hint-AD achieves desirable accuracy, achieving state-of-the-art results in driving language tasks including driving explanation, 3D dense captioning, and command prediction. To facilitate further study on driving explanation task on nuScenes, we also introduce a human-labeled dataset, Nu-X. Codes, dataset, and models are publicly available at https://anonymous.4open.science/r/Hint-AD-1385/.

Main paper: 
 introduction: 
 End-to-end perception-planning architecture is critical in autonomous driving (AD) [1, 2] and general embodied intelligence [3, 4] due to its potential for self-supervised training with extensive data. However, these systems face significant interpretability challenges [5, 6] .Interpretability issue [7, 8, 9, 10, 11] is particularly pronounced in embodied intelligence problems such as AD. When an AD system directly outputs control signals, it becomes difficult for human passengers to trust the decisions. To address this, natural language, a highly user-friendly communication medium, has been explored to enhance interpretability through tasks like driving explanation, 3D dense captioning, and visual question answering (VQA). While human driver recognizes the value of the bird's eye view (BEV) trajectories as an explanation of WHAT is happening, language offers a complementary perspective on WHY this is happening. These approaches can be categorized into declarative interpretability and aligned interpretability based on a single criterion: whether the generated language aligns with the intermediate outputs of the AD system (Fig. 1 ).• Declarative interpretability generates natural language directly without intermediate inputs from the AD system, as in recent works on driving explanation [12, 13, 14] , 3D dense captioning [15] , and visual question answering [16, 17, 18, 19] . This approach often suffers from hallucination, as the language is not grounded in comprehensive intermediate outputs, making it mere justification of the driving behavior.• Aligned interpretability requires alignment between language and the inner states of the AD model. To our knowledge, this approach was first explored by [14] , aligning attention states of the AD model with a language decoder. Later works aligned the language decoder with the inner decision states [20] .However, existing research neglects the correspondence between the language decoder and the complete perception-prediction-planning outputs of an AD pipeline, resulting in a discrepancy between the language tasks and AD tasks. The potential to enhance the accuracy of language tasks in driving scenes through intermediate AD pipeline outputs remains unexplored. To this end, we propose Hint-AD, an integrated AD-language framework designed for holistic alignment with the AD model's perception-prediction-planning process and high-accuracy language generation to facilitate the interpretability in autonomous driving.We developed two approaches for the holistic alignment between language and the AD model and the accuracy of language outputs: (a) We develop a holistic token mixer module that adapts the intermediate output tokens from the AD model for the language decoder, focusing on robust feature extraction and fusion; (b) We introduce an alignment task as an online dataset to align the language output with the intermediate outputs of the AD model, requiring the language decoder to interpret intermediate tokens generated during the AD model's inference process throughout training.We implemented Hint-AD on both UniAD [2] and VAD [21] , state-of-the-art (SOTA) AD models utilizing rasterized and vectorized representations, respectively, to demonstrate its generality. Experimental results show that Hint-AD achieves state-of-the-art performance on various language tasks, including driving explanation (20.4% on CIDEr than baseline), 3D dense captioning (185% on CIDErthan baseline), VQA (1.2 percent improvement on accuracy), and driving command prediction (1.2 percent improvement of accuracy). The alignment tasks significantly improved coherence between language outputs and intermediate AD model representations. Additionally, we contributed a human-labeled driving explanation dataset, Nu-X on nuScenes [22] to address the lack of driving explanation data on this widely-used AD dataset. 
related works: 
 End-to-end autonomous driving systems aim to create an architecture that processes sensor data and directly outputs vehicle control signals [23] . These systems have gained research attention due to their ability to address error accumulation problems found in traditional modular designs, where perception and planning are separated into distinct modules [24, 25, 26, 27, 28, 29, 30, 31, 32] . Prominent examples include UniAD [2] and VAD [21] integrate modular perception tasks such as object tracking, map building, motion forecasting, and trajectory planning within a unified framework.Offline datasets for end-to-end autonomous driving have also been developed [22, 33] .Interpretability of AD [5, 6, 7, 8] , the ability to provide a comprehensive explanation for AD planning, is crucial for user trust and system transparency in AD systems [5, 6] . Natural language, as a user-friendly medium for communicating with users, has been explored for improving interpretability of AD through by means like driving explanation [12, 13, 14] , VQA [16, 17, 18, 19] , and 3D dense captioning [15] . Previous work mainly focused on declarative interpretability, For example, [12, 34] realized driving explanation tasks using visual information. But intermediate outputs from AD model is not aligned. [14] raised the concept that the language output should be grounded on the inner states of an AD system. Aligning the decision state between the language decoder and the inner states of AD model has also been explored [35] , but to our knowledge, no previous work has achieved holistic alignment with all perception-prediction-planning process with an AD model. 
methodology: 
 To explore holistic alignment between natural language and intermediate results in end-to-end AD frameworks, we propose a novel framework named Hint-AD, which consists of three modules: a holistic token mixer, a language decoder, and a traditional AD framework. An overview of Hint-AD is shown in Fig. 2 . The existing AD pipeline in Fig. 2 can be any end-to-end AD system that decomposes AD into perception, prediction and planning. Without loss of generality, we implement our method on top of both UniAD [2] (as Hint-UniAD) and VAD [21] (as Hint-VAD), which use rasterized and vectorized representations, respectively.Firstly, we extract intermediate query tokens from the existing AD model of a perception-predictionplanning architecture, yielding track tokens, motion tokens, and planning tokens. Secondly, a holistic token mixer module will adapt the tokens for language decoder input, in which we design an instance mixer to merge instance-level track and motion information of each detected instance. We also introduce BEV blocks and instance blocks for further feature extraction and converting length-variable instance tokens to a fixed length. All the processed tokens are concatenated as context tokens for text generation (see Sec. 3.2). Finally, the context tokens are formulated as prompt tokens and put into the language decoder together with text prompts. We adopt a barbell adaptation paradigm for efficient context understanding of the language decoder (see Sec. Additionally, the training procedures are described in Sec.3.5.The query tokens extracted from the AD pipeline are not directly understandable to the language decoder. Addressing this, we propose a holistic token mixer architecture. The implementation is slightly different between Hint-UniAD and Hint-VAD in detail. We primarily follow the design of Hint-UniAD, while small adjustments for Hint-VAD are provided in Appendix B.3.We start by giving denotations for query tokens extracted from the AD pipeline. For a typical perception-prediction-planning AD pipeline, we can extract the following components: BEV tokensF bev ∈ R H b ×W b ×C, where H b , W b , and C are the height, width, and channels of the BEV field.Track tokens {F i track } N det i=1 ⊆ R D contain position and past trajectory information of each detected object, where N det is the number of detected objects, and D is the dimension of the token vector. Motion tokens {F i motion } N det i=1 ⊆ R D contain predicted future trajectories of each detected object. Planning steps F plan ∈ R Tp×2 would be the future trajectories predicted by the model.To effectively merge tokens into an instance level, we design a novel instance mixer that integrates the track token F i track and the motion token F i motion of each detected instance into an instance token F i instance . This is accomplished through tensor concatenation followed by a Multi-Layer Perceptron (MLP) projector P instance to project tokens of N det detected instances into embeddings with dimension D embed : Feature encoder E is implemented as a multi-layer convolutional network, which extracts the feature and down-scales the BEV to 3 × 3. Following this, an MLP projector P bev is employed to transform the BEV channel dimension C into D embed , yielding F ′ bev = P bev (E(F bev )). BEV blocks and instance blocks employ multi-head self-attention layers to adapt BEV and instance features. For BEV tokens, multi-head self-attention (MHSA) operates among them. Given the variable number of detected instances per frame, N ins-adapt learnable tokens {F ins-adapt } N ins-adapt i=1 are introduced as queries. Multi-head cross-attention (MHCA) is then performed between these learnable tokens and the instance tokens. BEV and instance blocks improve adaptation through improving feature extraction and fusion of BEV and instance tokens, as demonstrated in Sec. 4.4.EQUATIONEQUATIONThe planning steps F plan would be encoded by sinusoidal positional encoding PE and an MLP projector P plan to embedding dimension D embed : F ′ plan = P plan (PE(F plan )). Among all the instances tokens, there's one ego instance token F ego instance representing the ego car [2] . Processed BEV, instance, ego instance, and planning tokens would be concatenated as context tokens for further language generation tasks:EQUATIONTo incorporate high-level reasoning and context understanding ability of Multimodal Large Language Models (MLLMs) in AD relevant language tasks [36, 37, 17, 18, 38, 39] , we employ a LLaMa-Adapter-V2 [40] as our language generator with a pretrained LLaMA-2-7B [41] . For language fine-tuning, learnable adapters {F i adapter }N adapter i=1⊆ R D dec are employed, which serve as extra keys and values in the inserted layers with zero-initialized attention [40] , where N adapter is the number of layers to insert, and D dec is the dimension of tokens in the language decoder.In the original LLaMa-Adapter-V2 strategy, context tokens F context would be inserted in the first layer, and learnable adapters would be inserted in all other N -1 layers, where N is the total number of layers of LLaMA-2-7B. We observed that the adapters, which are for language tuning, tend to dominate the adaptation process and reduce the context-language alignment. This is crucial for AD tasks that requires high-level context understanding ability. Thus we propose a barbell adaptation paradigm (see Fig. 2 ), where learnable adapters are inserted only at the [2, N front + 1] layers (as front adapters {F i front } N front i=1 ) and the [N -N end + 1, N ] layers (as end adapters {F i end } N end i=1 ). The context tokens are inserted at the first layer.The rationale for placing adapters at both the front and end is that front adapters aid in comprehending context information, while end adapters enhance language fine-tuning. This design balances the need for high-level context understanding and precise language adaptation. The effectiveness of this barbell adaptation approach is demonstrated in Sec. 4.4. During training, we employ cross-entropy loss as the captioning loss, with supervision applied exclusively to the answer tokens.To align language with intermediate outputs from AD model, the language decoder needs grounded context understanding of the information contained in each token (i.e. object's position in track tokens) generated in AD model's inference steps. We implemented this by adding an online alignment task dataset in the training process.During the alignment task, given the AD model's intermediate inputs, a set of prompt-answer pairs are generated (see Fig. 3 ). This task includes four types of alignments: (a) Counting alignment, requiring the language decoder to interpret the number of instances of each type detected in the frame according to track tokens; (b) Position alignment, necessitating the model to provide the position of a tracked instance based on a specific instance token; (c) Motion alignment, involving decoding the velocity information contained in an instance token; and (d) Planning alignment, requiring the language decoder to output the future trajectory points contained in a planning token.All the question-answer pairs for alignment tasks are generated online during training. The alignment tasks greatly improve the language decoder's context understanding of the intermediate tokens, thus improving the accuracy of AD captioning by a large margin (see Sec. 4.4).The whole training pipeline of Hint-AD includes two stages. In the first stage, the end-to-end AD model is trained independently. In the second stage, we freeze all parameters of the AD model and the MLLM parameters, updating only the parameters of the holistic token mixer and adapters. The total trainable parameters at the second stage are 87M, more training details are stated in Appendix A. 
experiments: 
 4.1 Datasets and baselines Datasets. Explanation serves as a guide for human learning and understanding [42, 43] . Particularly in the context of end-to-end autonomous driving (AD) systems, human users often seek explanations to bridge the gap between sensor inputs and AD behaviors [14] . Currently, there is no dataset providing such explanations for nuScenes [22] , a widely utilized dataset in AD research. To address this gap and facilitate interpretability-focused research on nuScenes, we introduce Nu-X, a comprehensive, large-scale, human-labeled explanation dataset. Nu-X offers detailed contextual information and diverse linguistic expressions for each of the 34,000 key frames in nuScenes.A sentence of explanation typically comprises narration and reasoning [14] , for instance: "<Narration> The car is merging into the right lane. <Reasoning> To pass the red car in front." In our dataset, each caption encompasses these two components. Detailed examples and key data statistics are presented in Fig. 10 . For an in-depth description of the labelling process and comprehensive data statistics, please refer to Appendix D.All Hint-AD architectures and baselines were trained and evaluated on the following datasets to provide a comprehensive analysis: (1) Alignment task dataset (see Sec. We present examples of the language output generated by Hint-AD across multiple tasks, including driving explanation, 3D dense captioning, VQA, command prediction, and four categories of alignment tasks. Captions that do not match the ground truth are colored in red.annotated with appearance, motion, environment, and inter-object spatial relationships; (3) NuScenes-QA [16] , a VQA dataset covering 34K frames of nuScenes with five question types, including existence, counting, query-object, query-status, and Comparison; (4) Driving command dataset, labeled by us on nuScenes (see Appendix E), composed of direction and velocity commands.Baselines. We have selected benchmark methods that include both key milestones and state-of-theart approaches in language generation within the AD context (more details in Appendix C.4): (1) ADAPT [12] generates sentences with a vision-language transformer in an auto-regressive manner. Cross attention and sparse attention masks are used on text and video tokens; (2) BEV+Adapter [40] takes only BEV features as input and LLaMA-Adapter-V2 (the same as Hint-AD) as the language decoder; (3) BEVDet+MCAN [16] uses a Modular Co-Attention Network (MCAN) [44] with layers of self-attention for separate language and visual understanding. Stacked cross-attention layers are used for cross-model feature interaction. Detection results from a BEVDet [45] is adapted for input; (4) Vote2Cap-DETR [46] has two parallel task-specific heads base on a transformer architecture.The queries are decoupled into localization queries and caption queries; (5) TOD 3 Cap [15] utilizes a query-based detection head to generate a set of 3D object proposals from the BEV features. These features are then processed by LLaMA-Adapter [47] to be prompts for the language model to generate dense captions; (6) GPT-4o is a multimodal model developed by OpenAI, equipped with state-of-theart vision capabilities alongside text generation performance comparable to its predecessor, GPT-4. (7) Gemini-1.5 is a pioneering large language model from Google, specifically designed to handle multimodal inputs with an extended context length.Quantitative results. We present results separately for different input types and backbone modules on four datasets. For Nu-X and TOD 3 Cap datasets, we adopt four standard image captioning metrics, CIDEr (C) [48] , BLEU (B) [49] , METEOR (M) [50] and Rouge (R) [51] . GPT-3.5 scoring (G) is also used for Nu-X' evaluation due to the comprehensive expression in driving explanation (See Appendix C.6 for detail). A threshold of 0.25 is set for matching predicted and ground-truth bounding box while testing TOD Input Method Nu-X TOD 3 Cap NuScenes-QA Command C ↑ B ↑ M ↑ R ↑ G ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ Image + 6-To quantify the alignment between language and the intermediate outputs of the AD model, we evaluate the language decoders' output against the predictions of the AD perception modules, which is generated online on the validation set. Four kinds of disalignment protocols are designed: (a) Counting Disalignment (CD) which measures the discrepancy between the number of instances in each category given by the decoding head and the tracking model, (b) Position Disalignment (PD) which measures the positional difference of a specific instance, (c) Motion Disalignment (MD) which measures the velocity difference, calculated as the mean distance between the velocity in the caption and the velocity predicted by the perception system, and (d) Planning Disalignment (PLD) which measures the discrepancy in trajectory points. For detailed definition, please refer to Appendix C.1.We tested Hint-AD with both aligned interpretability (original design) and declarative interpretability, results are detailed in Appendix C.1. The aligned language decoder performs significantly better than the models operating under the declarative interpretability paradigm, indicating the effectiveness of alignment designs including holistic token mixer and alignment tasks.Effectiveness of holistic alignment. To evaluate the effectiveness of holistic language-AD alignment on language task accuracy, we conducted an ablation study by removing track, motion, and planning tokens from the language decoder's inputs. Results in Tab. 2 show that using all tokens achieves the highest performance. Track tokens enhance 3D dense captioning with positional information, while planning tokens improve command prediction by providing future trajectory data.Table 2 : Ablation on holistic alignment. The performance is highest when all tokens are used, highlighting their importance in enhancing specific tasks. Effectiveness of barbell adaptation. We explore three alternatives: (a) Early Fusion, original design of LLaMA-Adapter-V2 [40] , adapters in each layer; (b) Pyramid adaptation, adapters in the first N front layers; and (c) Hammer adaptation, adapters in final N end layers only. Barbell adaptation performs the best on three datasets (Tab. 4). More adaptation methods in Appendix B.2.Method Input Nu-X TOD 3 Cap NuScenes-QA Command C ↑ B ↑ M ↑ R ↑ G ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ Hint-UniAD W/o↑ B ↑ M ↑ R ↑ G ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ Hint-UniAD W/oTable 4 : Ablation on adaptation strategy. Ballbell adaptation achieves the best scores on all datasets except for TOD 3 Cap, demonstrating our adaptation strategy enables better context understanding and language fine-tuning.Nu-X TOD 3 Cap NuScenes-QA Command C ↑ B ↑ M ↑ R ↑ G ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ Hint- 
conclusions and limitations: 
 We present Hint-AD, an integrated AD-language framework that aligns language generation with the holistic perception-prediction-planning process of AD models, achieving SOTA performance in multiple AD captioning tasks. Meanwhile, as an exploratory research on the implementation of aligned interpretability, the following limitations are waiting to be resolved by further research:• Due to its pipeline-specific nature, any changes in the intermediate output format necessitate modifications in the design of the token mixer. For purely end-to-end models, such as black-box models, adjustments are required to handle latent outputs effectively.• The LLaMA-based language decoder is relatively time-consuming. Further investigation into smaller model alternatives, such as MiniChat-1.5-3B and StableLM-3B-4E1T, is warranted.As LLM's potential to comprehend AD models' intermediate outputs becomes evident, future research can delve deeper into this area and enhance user trust in AD models through aligned interpretability. 
 
 Appendix: The graphics memory usage for the language decoder varies with batch size, ranging from 31GB (batch size = 1) to 78GB (batch size = 24). For Hint-VAD, the memory usage ranges from 26GB (batch size = 1) to 45GB (batch size = 20). Training Hint-UniAD takes 13 hours for 3 epochs on 8 A100 GPUs, while training Hint-VAD takes 7 hours on 8 A100 GPUs. The inference times for the AD module and language decoder are detailed in Tab. 5. In Tab. 6, we present the wall-clock time of the entire system with a batch size of 1, as applied in a real car scenario. This time encompasses data acquisition, pre-processing, and post-processing, all measured in seconds.Data acquisition time includes capturing images from six HIKROBOT MV-CU013-A0UC cameras and transmitting these images from the computer to the online server if using online inference.Pre-processing time involves preparing image sequences before inputting them into the model, which is conducted online if using an online server.Autonomous driving inference time refers to the time taken by UniAD and VAD to perform inferences.LLM inference time is calculated as the time required to generate 26.5 tokens, which is the average generation length observed in our research.Post-processing time includes the duration needed for data transmission if the inference is conducted online.Since users can access language output as soon as the first token is generated, we define interactive latency as the sum of data acquisition, pre-processing, autonomous driving inference, and postprocessing times. Here we provide Tab. 7 to specify parameters used in our architecture and additional explanation.BEV blocks consist of a sequence of 8 transformer encoder blocks. Each block includes a normalization layer, a multi-head self-attention layer with 16 attention heads, a second normalization layer, and a multi-layer perceptron (MLP) with a hidden dimension of 3072.Instance blocks comprise a sequence of 5 transformer decoder blocks with the same internal structure as the BEV blocks. These blocks accept 5 learnable tokens, F ins-adapt , and instance tokens, F instance , as input. A cross-attention mechanism is performed between these inputs, with F ins-adapt serving as the queries and F instance serving as the keys and values. Here we present analysis on why adapter is chosen as tuning method for Hint-AD.We C ↑ B ↑ M ↑ R ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ LoRA (VAD [21] uses the same BEV encoder as UniAD [2] , but vectorizes scene representations for planning and getting rid of dense maps. So the BEV tokens are the same:F bev ∈ R H b ×W b ×C, where H b , W b , and C are the height, width, and channels of the BEV field. But for VAD has different motion prediction and planning phases, so the input tokens are adjusted as follows: In Track tokens{F i track } N i=1 ⊆ R D and Motion tokens {F i motion } N i=1 ⊆ R D , Nis the number of agent queries, not the number of detected objects. We select valid tokens based on the query's classification score, using a threshold of 0.5, thereby only considering queries with a high probability of corresponding to actual objects. Additionally, VAD considers both ego-agent interaction and ego-map interaction. We use updated queries as Ego token F i ego ⊆ R D , which contain both dynamic and static information of the driving scene. Finally, we adopt Planning steps F plan ∈ R Tp×2 , the future trajectories predicted by the model. We adjust the token mixer to accommodate the shape of each token while maintaining the same overall architecture.To quantify the alignment between language and the intermediate outputs of the AD model, we evaluate the output of language decoders against the results predicted by the AD perception modules, which is generated online on the validation set. We define four kinds of disalignment as follows:Counting Disalignment (CD) measures the discrepancy between the number of instances of each category given by the text and the number detected by the AD model. It is defined as: CD = 1 Ncat Ncat i=1 (c (i) text -c (i) AD ) 2, where N cat is the number of object categories, c Motion Disalignment (MD) measures the difference in velocity, calculated as the mean distance between the velocity in the caption v text and the velocity predicted by the AD modelv AD : MD = Mean(|v text -v AD |).Planning Disalignment (PLD) measures the discrepancy in trajectory points. A trajectory is expressed as four future positions of the ego car with 0.5s time steps between each [r (1) , r (2) , r (3) r (4) ], and PLD is defined as the mean distance from the original AD prediction at 2s (L2 loss): PLD = Mean(|rEQUATIONHere we offer a comparison of alignment between Hint-UniAD and various declarative alignment methods, as shown in Tab. 9.• Among the declarative methods, the Hint-UniAD architecture achieved the highest performance, except in the counting disalignment task, demonstrating its overall efficacy.• Methods utilizing a pre-trained LLM as a language decoder performed poorly on the counting task. This poor performance arises from a bias towards certain numbers: for example, LLaMA tends to output specific integers like 10 and 20.Table 9 : Comparison with other declarative alignment models. To standardize the output formats of the language decoders used in these methods, we also tuned the models accordingly. N refers to no, and Y refers to yes. Here we demonstrate more detailed Qualitative results on all four datasets. All predictions are generated by Hint-UniAD model, and we choose TOD 3 Cap as baseline while evaluating results on Nu-X and Command datasets.Our model excels in generating accurate and contextually appropriate predictions across multiple tasks. For driving explanation and command prediction, our model outperforms the baseline by providing more accurate narration, reasoning, and driving commands. In VQA tasks, it excels in accurately describing the surroundings, including the type and number of objects. In 3D dense captioning, our model successfully detects surrounding objects and provides precise status descriptions. However, it occasionally make errors in identifying object colors and estimating distances. In alignment tasks, our model performs well across all directions but struggles with precise positional information due to the limitations of image-based data. Additionally, it tends to undercount objects in counting tasks.We also present some qualitative results from prompting GPT-4o and Gemini-1.5, as shown in Figures 4 to 6 . Hint-AD demonstrates better consistency with ground-truth annotations, achieving higher accuracy in VQA tasks such as determining whether an object is moving. In contrast, GPT-4o and Gemini-1.5 tend to provide more diverse descriptions of the scene and driving behavior. The full prompts are shown in Figure 8 .According to Tab. 1, VAD performs much worse than UniAD on TOD 3 Cap dataset but slightly better on the other three datasets. This can be explained by VAD's restricted perception range of 60m by 30m, whereas objects in TOD 3 Cap often fall outside this range. Since all questions in TOD 3 Cap are object-centered, missed detections significantly lower the scores on this dataset.ADAPT. [12] The original work [12] takes a sequence of raw video frames as inputs, and outputs natural language narration and reasoning. It employs Video Swin Transformer [52] as the video encoder. Two tasks, Driving Caption Generation (DCG) and Control Signal Prediction (CSP), are jointly trained based on the same video tokens.Here we adopt its text generation head, where the sentences are generated by vision-language transformer in an auto-regressive manner. The text inputs are tokenized and padded to a fixed length, and then are embedded with a segment embedding method. While training, we use cross attention mask within text tokens and sparse attention mask between text tokens and BEV features. The vision-language transformer encoder starts with a "[CLS]" token and generates one word token at a time, consuming previously generated tokens as the inputs until it outputs the ending token "[SEP]" or reaches the maximum length.TOD 3 Cap. [15] The original TOD 3 Cap framework is implemented using a BEV detector module that processes both 2D and 2D+3D inputs. We adapted TOD 3 Cap for UniAD and VAD by adding an extra captioning head. The inputs for Image BEV features and object proposals are derived from the BEV tokens and tracking tokens of these two AD models, respectively.Vote2Cap-DETR++. [46] Unlike the traditional "detect-then-describe" methods that first detect objects and then describe them, Vote2Cap-DETR++ uses a transformer-based framework to parallelly decode object localization and caption generation tasks. It uses 3D positional encoding is added to enhance the context-awareness of the generated captions: Absolute Position Token is injected into the caption prefix to indicate the spatial location of the object in the 3D scene and Rank-Based Position Encoding is applied to local context tokens.We have expanded the ablation studies to include multi-module ablation and cross-model consistency, encompassing both Hint-UniAD and Hint-VAD. The ablation results for Hint-UniAD are presented in Tab. 10 and Tab. 11. These results indicate a consistent performance trend between Hint-UniAD and Hint-VAD. Notably, during multi-module ablation, the performance of the model experiences a further decline. C.6 Caption evaluation GPT-3.5 for Logicality and Accuracy Assessment The evaluation of visual language models for automated driving presented in this paper focuses on two main components: the logicality assessment of driving descriptions and the accuracy evaluation of driving behavior predictions.C ↑ B ↑ M ↑ R ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ Hint-C ↑ B ↑ M ↑ R ↑ C ↑ B ↑ M ↑ R ↑ H0 ↑ H1 ↑ All ↑ Acc. ↑ Hint-For the logicality assessment of driving descriptions, we utilized GPT-3.5 to determine the correlation between driving behavior descriptions and the triggers for those behaviors in the model's predictions. Initially, we prompted GPT-3.5 to extract key information from each predicted description, including the car's driving behavior (movement and geographic location) and the causative factors (subject and form). We evaluated the associative relationship between the preceding and following events based on three criteria:• The consistency between the car action implied by the causative factors and the predicted driving behavior.• The logical coherence between the causative factors and the predicted driving behavior.• The consistency between the details in the causative factors and those in the driving behavior.Each metric was scored and summarized for each component.Additionally, we conducted a comparative analysis of the similarity between predicted and manually labeled driving behaviors. Using GPT-3.5, we extracted key information, such as the car's action, purpose, and reason, from both the predicted and manually labeled driving behaviors for comparison. Each pair was scored based on the degree of match (complete, partial, or missing). The primary metrics for this comparison included:• The similarity between driving behavior and geographic location.• The objects and reasons leading to such driving behavior.• Other relevant details.Penalties were applied for instances of fictitious information, which reduced the total score. The mean of these scores constituted the scene description score. The total prompt for GPT-3.5 is shown in Fig. 9 To balance accuracy and effectiveness of annotation, we conducted a process incorporating both human labelling and LLM diversification (see Fig. 11 ). The training / validation branches of nuScenes contain 850 videos, each spanning about 20 seconds, with approximately 40 key frames per video annotated. labelling all frames is inefficient as driving behaviors and road conditions in adjacent frames are often similar. Each video typically features only 3-4 distinct driving behaviors. However, identical annotations across frames risk over-fitting, reducing effectiveness. Therefore, we used MLLM to diversify expressions and add scene-description details to each frame. The annotation process includes the following procedures:Human labelling We employed five professional annotators for a total of 126 hours. All annotators are familiar with US driving rules and have driving experience. They were instructed to describe specific driving behaviors and the potential reasons for these decisions. Annotations for each video include the time intervals of certain driving behaviors, along with narration and reasoning. For narration, annotators focused on at least 11 driving behaviors, such as directional decisions (e.g., go forward, turn left, turn right, U-turn) and velocity commands (e.g., accelerate, decelerate, stop, stay stationary). Descriptors like "slightly" or "sharply" were encouraged.For reasoning, annotators incorporated specific road conditions, key instances (e.g., pedestrians, front cars), and traffic rules (e.g., traffic lights, stop signs, yellow lines).GPT-4v image description Before diversification process, we firstly leverage GPT-4v to generate the image description of multi-view camera images to extract the visual information. Specifically, we provide GPT-4v with each frame of multi-view images in nuScenes and task it with capturing and recording key traffic information from these images. This information includes, but is not limited to, pedestrians, vehicles, traffic signals, and other relevant road signs. Through this process, GPT-4v is able to automatically extract and provide detailed descriptions of the elements and dynamics present in each image frame in a sentence of about 120 word length.GPT-4v auto-diversification With the image description, we utilize GPT-4v to automatically generate diverse outputs based on the human-labeled captions, which is particularly useful for creating varied descriptions and perspectives of the same set of images. We also constraints the text output to follow the original main driving behavior and reasoning of human annotators to ensure the correctness.Human check We conduct a thorough human check to ensure accuracy and relevance after the auto-diversification. This step involves experts reviewing and validating the multiple interpretations generated by GPT-4v, verifying that the descriptions accurately reflect the content of the images. The human check process is essential for maintaining high standards of quality and reliability in our traffic analysis, ensuring that the diversified outputs align with real-world observations.We show some key results of data statistics in Fig. 11 (b)-(d). The average length of a narration is 8.29 words for a sentence, and the average length of reasoning is 11.25 words.The ground truth of the nuScenes dataset provides only three types of directional commands: <TURN LEFT>, <TURN RIGHT>, and <FORWARD>, which lack the necessary velocity information. To address this, we labeled a command dataset using programmed protocols based on the ground truth future steps, incorporating additional velocity commands such as <ACCELERATE>, <DECELER-ATE>, <STATIONARY>, and <KEEP SPEED>.Directional commands were generated by analyzing the trajectory slope, applying a criterion where driving 5 meters with only 1 meter of lateral drift was considered.For velocity commands, polynomial fitting was used to smooth the speed curve derived from the average speed between predicted points. The average acceleration over the next 4 seconds was then used to determine the velocity commands. Given the vehicle's increased sensitivity to speed changes when approaching zero, different thresholds for speed changes were set: 0.36 m/s 2 , 0.12 m/s 2 , and 0.05 m/s 2 for initial speeds of >2 m/s, 1-2 m/s, and <1 m/s, respectively.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: Strengths:
1. Contributes a dataset (nu-X) for explainability in autonomous driving 
2. Extensive quantitative and qualitative evaluations
3. Modular approach to interpretability, a key safety research area

Weaknesses:
1. Modularity for the sake of interpretability may not survive the larger trend of end to end methods taking over. Future value of this work is contentious.
2. Please cover limitations in more detail:
- how well does the method generalize
- lack of enforced alignment
- comparison against multimodal LLMs 
- latency of system for driving


Reviewer HNsL:The reviewer questioned the future value of the modular approach in Hint-AD, given the trend towards fully end-to-end methods in autonomous driving. They also pointed out the system's dependence on specific token formats, which could limit its applicability. The authors clarified that Hint-AD is already built on end-to-end AD systems and that incorporating intermediate outputs is a common practice in such systems. They also highlighted that Hint-AD is not dependent on specific token formats and can work with various types, given sufficient data and a well-designed encoder.

Reviewer Kmwf: The reviewer raised several concerns, including a lack of technical details in certain areas, the absence of comparison with state-of-the-art multi-modal language models, and insufficient discussion of real-world applicability, such as latency. The authors provided more detailed technical explanations, included comparisons with GPT-4 and Gemini 1.5, and added a discussion on real-world applicability, focusing on latency and inference speed.

Reviewer kWqV: The reviewer questioned the limitations of Hint-AD, particularly regarding lingering disalignment and the applicability to non-modular, black-box end-to-end AD systems. The authors discussed the possibility of reducing disalignment through strategic model design and larger data scales. They also addressed the applicability to black-box models, explaining that aligned interpretability could be achieved by passing BEV perception and inner states to the MLLM.

Reviewer dfFt: The reviewer suggested extending the baselines in the experiments and evaluating other adaptation strategies beyond layer-choosing. The authors included additional experimental results featuring GPT-4, Gemini 1.5, LoRA, and DoRA. They also explained why most 3D-LLM baselines are incompatible with their use case.

Reviewer 3Cfu: The reviewer pointed out a lack of clarity in the method presentation, the ablation studies, and the definition of interpretability. The authors provided a more detailed figure, clarified the loss functions and training stages, expanded the ablation studies, and provided a clearer definition of interpretability and how it's measured.

Reviews: review: title: title: Interesting work, decent contributions, but future impact is uncertain
paper_summary: paper_summary: The paper introduces a holistically aligned AD-language system designed to generate explanations for driving behaviors. Specifically, the authors demonstrate that by integrating tokens and employing an alignment procedure across all components of the holistic perception-prediction-planning stack, the LLM model enhances performance in driving language tasks. The effectiveness of this approach is validated on three distinct datasets, including the newly introduced Nu-x dataset, which provides driving explanations aligned with frames from the widely used nuScenes dataset.
main_review: main_review: Praise:
- The paper addresses a critical aspect of interpretability in ADAS systems' behaviors.
- An extensive ablation study across three datasets evaluates the impact of various alignment strategies, the design of a holistic token mixer, and the placement of adaptation injections within the architecture.
- The authors contribute the Nu-x driving explanation dataset, which promises to foster further advancements in model interpretability research.

Concerns and Questions:
- A primary concern is the fundamental approach of the system: as holistic pipelines gradually give way to end-to-end planners, the reliance on perception and prediction outputs becomes problematic. Hence, these trends might strongly limit the potential impact of this work on the future of AV development.
- Another drawback is the system's dependence on specific token formats, potentially limiting its applicability. In holistic pipelines, outputs from perception and prediction systems are tailored to the planner’s requirements, undergoing continuous evolution throughout AV stack development. This necessitates frequent re-training of the proposed DNN with each pipeline modification.
questions: questions: Please, see the review section.
limitations: limitations: 2
review_summary: review_summary: Interesting work, decent experimental session with good ablation studies. But the core of the approach brings questions about future impact of this work. Hence I propose "weak accept"
title: title: Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving
paper_summary: paper_summary: In this paper, the authors introduce a novel approach called "aligned interpretability" for autonomous vehicle (AV) actions. This algorithm/method aims to provide intermediate interpretability to the AV decision-making processes by aligning language generation with the intermediate outputs of the AD model.  Authors have developed a new interpretability method, introduced a new dataset (Nu-X), and created a framework (HINT-AD) that generates language aligned with the perception-prediction-planning outputs of the AD model.
main_review: main_review: ## Quality and Clarity


Strengths:

-  The methodology is thoroughly explained, with clear diagrams illustrating the HINT-AD architecture.
- Clear explanation of the annotation pipeline and benchmarking, including the interoperability and alignment. 

Weaknesses:

- Some technical details, such as the specifics of the rationale for selecting adaptors over SFT, the cross-attention mechanism between the learnable and instance tokens, and masking techniques, are not fully explained, which could hinder reproducibility.
- The paper could benefit from a more detailed discussion of the limitations of the approach.

## Originality


Strengths:

- The token mixer module and the alignment task are innovative in robotics applications.
- The Nu-X dataset addresses a gap in existing autonomous driving datasets.

Weaknesses:

- It builds upon existing work in multi-modal language models such as LLAVA and autonomous driving systems. A more explicit discussion of how this work advances beyond previous approaches would strengthen the paper.

## Significance


Strengths:

-  The HINT-AD framework demonstrates improvements over baseline models.
-  The approach could enhance the explainability of autonomous driving systems.
-  The Nu-X dataset introduced could facilitate further research in this area.

Weaknesses:

- The real-world applicability, impact of the latency from the token generation and scalability of the approach are not thoroughly discussed.
- Authors have not shown how MLLMs such as GPT4 or gemini1.5 perform compared to the proposed methods.
questions: questions: - What is the rationale for not passing planning and ego motion through an attention layer similar to the BEV and instance blocks?
- Explain the why not perform a SFT instead of adaptor, give the task performed is different compared to typical training data in the LLM ?
- Given that MLLM adapter training (e.g., LoRA) is shown to preserve pretrained information and language understanding, why weren't low-rank adapters used instead of full adapters?
- It is highly recommended to include comparisons with multi-modal language models such as GPT-4, LLAVA-v1.6, and Gemini 1.5 for VQA and other tasks. This comparison would help quantify the improvement over out-of-the-box models, especially given that LLAVA is the base MLLM for HINT-AD.
- The declarative alignment results of the HINT-AD models proposed here should be compared with other declarative alignment models. This could highlight whether HINT-AD is increasing overall efficacy of the alignment or specifically improving the efficiency of aligned interpretability.
limitations: limitations: 1
review_summary: review_summary: I recommend a weak accept for this paper. The work presents a novel and potential impactful contribution to the field of interpretable AI with the introduction of the HINT-AD framework and the concept of aligned interpretability. The paper demonstrates clear methodological rigor and fair experimental validation, showing some improvements over baseline models across multiple language tasks. The creation of the Nu-X dataset is also a valuable contribution to the research community and will help advance the field. However, the paper has some limitations that prevent a stronger recommendation. These include the 1. Need for more detailed technical explanations in some areas, 2. A lack of comparison with state-of-the-art multi-modal language models, and 3. Insufficient discussion of real-world applicability such as latency. Despite these shortcomings, the  approach and potential impact of this work make it worthy of inclusion in the conference, with the expectation that the authors will address these issues in the final version or in future work.
title: title: 
paper_summary: paper_summary: In this paper the authors define a distinction between "declarative" and "aligned" interpretability, arguing that aligned interpretability, where language explanations are grounded in the internal state of a system, is necessary to explain _why_ certain decisions are made. This motivates the authors' development of the Hint-AD framework which provides aligned interpretability for modular perception-prediction-planning autonomous driving stacks (specifically UniAD and VAD in this work), and moreover results in improved performance on various vision-language driving tasks including a new human-labeled driving explanation dataset + task Nu-X. These advances are enabled by a new AD-language model architecture that incorporates intermediate module outputs and is trained on various online alignment tasks to encourage the learning of token representations well-adapted to interpretable reasoning.
main_review: main_review: This paper does an excellent job of motivating why aligned interpretability is necessary (i.e., the alternative is post-hoc "justification" at best), demonstrating through disalignment metrics the drawbacks of purely declarative approaches to interpretability, and supporting their design choices through impressively thorough quantitative ablation studies. I am pleased to recommend its acceptance, however there are some questions about the limitations of this work which, if discussed, would help readers better assess the impact of this work:

(1) Lingering disalignment: While processing intermediate outputs through the holistic token mixture model enables the MLLM to access more information, and therefore to provide better grounded explanations, alignment is still treated as an emergent property of training rather than enforced by design. This is evident in the disalignment metrics where, although significantly improved, Hint-AD still displays some degree of mismatch between module outputs and language explanation. How may this be remedied? Is greater model/data scale all that is needed to drive disalignment down to 0? An illustrative comparison might be to contrast this approach to tool-using LLMs, i.e., LLMs that in order to improve QA performance can query specialist models (e.g., planning, prediction, perception modules) and directly include their output in the response.

(2) Discussion of non-modular approaches to end-to-end AD: In this work "end-to-end" refers to "end-to-end differentiable" AD systems, while to some researchers and practitioners it has an alternative connotation of end-to-end "black box" neural network policies that do not have clear submodules. The authors do make it clear that Hint-AD can only be applied to "any end-to-end AD system that decomposes AD into perception, prediction and planning", but many black box policies do train additional output heads (with the claim that this is done "for interpretability") that provide similar perception, prediction, and planning outputs but notably are not "intermediate" in the sense that they are not on the computational path for decision making. One might imagine developing a system similar to Hint-AD for an end-to-end black box AD system with the output of these auxiliary heads. Would such an AD-language system still fall in the category of declarative interpretability? Do the authors believe that aligned interpretability can ever be achieved with models with a black box backbone?
questions: questions: From the comments/questions above, please prioritize:
- Is greater model/data scale all that is needed to drive disalignment down to 0?
- Could aligned interpretability be achieved for black-box models?
limitations: limitations: 2
review_summary: review_summary: I recommend this paper's acceptance at CoRL 2024 as it both presents strong benchmark results/corresponding insights for AD practitioners, in addition to addressing a topic of broader interest/importance in the robot learning community today: the importance of alignment in interpretable AI. However, as noted above, I believe the paper would be improved if the limitations of this work could be more clearly established.
title: title: The authors present a well motivated approach that is extensively evaluated across various benchmarks and metrics and against several methods. The proposed work provides strong technical contributions and the technical decisions made throughout the paper have been explained with clarity.
paper_summary: paper_summary: The paper presents a systems level approach to improving interpretability in Autonomous Driving (AD) systems. The authors argue for an aligned interpretability approach as opposed to declarative interpretability in order to improve interpretability of the decision making process within AD systems. Specifically, they propose the use of intermediate outputs (from perception, prediction and planning) in the AD stack to provide contextual information to a foundation model to achieve this goal. They propose the use of features extracted from multi-view images as well as track, motion, and planning information obtained from the rest of the AD stack and present a method to efficiently and effectively combine this contextual information towards generating better aligned responses. Lastly, they created a dataset by augmenting the nuScenes dataset with detailed contextual information.
main_review: main_review: The work presented by the authors is well motivated and clearly explained throughout the paper. The authors have made strong technical contributions in both designing a system to improve interpretability and also releasing a dataset towards the same.
 
Strengths:
- The authors present the motivation for the work clearly.
- Claims made by the authors are supported through extensive quantitative evaluations, while also evaluating the results qualitatively. Evaluations are made on a wide variety of metrics, datasets and methods.
- Architectural decisions made in the paper (and the reasons behind them) are conveyed clearly and are well justified through ablations.
- The proposed approach is evaluated against several SOTA approaches and shows strong potential.

Weaknesses:
- It is important to consider inference time and model size when building systems for autonomous driving as this ultimately dictates the applicability of a method. Though the authors provide inference time for different batch sizes for the AD system and language decoder, it would be beneficial to also include latency measurements for the entire system.
- The limitations of the proposed approach have not been covered by the authors.
- The authors present a claim that demonstrating their proposed approach on UniAD and VAD demonstrates the methods' generality. However, it is unclear this is the case. It would be beneficial to provide some clarity on the generalizability of the proposed approach. 
- Though it may be implied, it would be helpful to mention that F'_{plan} is projected to D_{embed} for clarity in the methodology section. This is a minor point.
questions: questions: 1. What is the latency of the proposed approach as a whole i.e., of the augmented AD system?
2. What are the limitations of the proposed approach?
3. How does the success of the proposed approach on UniAD and VAD demonstrate the methods' generality? (Claim made in the paper)
limitations: limitations: 3
review_summary: review_summary: The authors have presented their approach and its motivation with clarity and have extensively evaluated it both quantitatively against other methods, across several metrics and benchmarks as well as qualitatively. The paper, however, would benefit from clarifying claims made around generality of the approach and from providing additional information around the limitations of the system. That being said, the proposed approach provides sufficient contributions for me to recommend a Weak Accept.
title: title: Multi-modal language model  for dynamic driving scenes and first annotation data on NuScenes
paper_summary: paper_summary: Multi-modal language model for aligned interpretability in autonomous driving
main_review: main_review: This paper introduces Hint-AD, a method that provides aligned interpretability in autonomous driving scenarios leveraging multi-modality language models(MLLMs). The method can be thought of a variant of 3D-LLM but with autonomous-driving-specific modalities/tokens including features for BEV, tokens for past and future predicted trajectories and tokens for motion planning. 

To incorporate those changes, the authors introduce certain architectures named "holistic token mixer" designs to fuse the information together with multi-head-self-attention and multi-head-cross-attention. To input those driving-related context information into the MLLM finetuning, the authors introduced "Barbell adaptation" that only changes the parameters for the first two and last two layers of the network to avoid over-reliance on language tokens and ignores context information.

Another contribution is the introduction of Nu-X dataset, which is the first language annotation effort on NuScenes, a prominent dataset in autonomous driving. Following common recipes for MLLM training, Nu-X also introduces 4 different tasks including counting, position, motion and planning.

Overall, I believe the idea of "Aligned interpretability" is a better alternative of "Declarative interpretability" by incorporating driving-specific immediate information into the explanation.  I feel that this architecture can be even extended beyond just interpretability research, but in the future, directly produces better track prediction and planning signals. Although interestingly, the current results only support better scene understanding, not particularly strong in terms of numeric tasks like counting or planning (see figure 3). It could be a future direction.
questions: questions: The experiment section could be made stronger:

1. Baselines in table 1 could be extended. For example, although many closed-source VLMs do not support finetuning, they could still be used as baseline including GPT-4V, Gemini-1.5. Other 3D-LLMs that are good at spatial metric prediction including SpatialRGPT, Spatial GPT, Transcrib3D can also be considered adding as baselines.

2. While table 4 investigate the effectiveness of "Barbell adaption" against other approaches in terms of layer-choosing strategies, it would also be helpful evaluate against other adaption strategies beyond layer-choosing, for example, LoRA, DoRA, IA3.

**Update after rebuttal:**
Thank authors for the additional experiments. I believe they made the paper stronger. I have raised my rating on technical quality.
limitations: limitations: 3
review_summary: review_summary: Good extension of 3D-LLM s to dynamic scenes in autonomous driving as well as the contribution of a useful dataset
title: title: Aligning an autonomous driving system with multimodal LLM to improve interpretability
paper_summary: paper_summary: This paper proposes a mixer sub-network to align the intermediate outputs from a BEV perception-prediction-planning system for autonomous driving (AD) with a multimodal LLM (MLLM) for driving explanation, 3D dense captioning, and command prediction.  The contribution of this work is to enhance interpretability in end-to-end AD, achieving better performance than the declarative interpretability that is not grounded in the intermediate outputs of AD systems.
main_review: main_review: Strengths:
The paper shows that aligning the MLLM and AD models can provide better results for driving explanation, 3D dense captioning, and command prediction. The proposed method has been proven effective on two popular end-to-end AD models, UniAD and VAD.

The paper also provided dedicated data called Nu-X, extended from the nuScenes dataset for driving explanation.

Weaknesses:
The presentation of the method lacks a certain level of detail. The pipeline figure (Fig. 2) is very compact and does not provide much detailed information about the core component, the mixer sub-network. There should be a detailed figure to make it clearer.

What are the loss functions to optimize the model in different training stages?

In section 3.5, it is not clear why the MLLM parameters are frozen or whether they are fine-tuned.

The ablation studies in Tables 2 and 3 are not comprehensive. First, each time, only one module is removed. What about the performance when removing more than one of the proposed modules together? Second, are the ablation studies consistent on Hint-VAD?

The paper emphasizes interpretability, but interpretability is never clearly defined in this paper, and how to measure the quality of interpretability is also not clear. A related question concerns the Nu-X dataset for driving explanation. The authors never mentioned how good the annotation scheme of Nu-X is regarding "interpretability."

Some minor issues/errors:

In Fig. 2, layers are subscripted by "early" and "late," but in the text, they are subscripted by "front" and "end."

"MLLM" is never defined before it appears.

There is an error in Table 2; in the TOD2Cap-M column, 48.5 should be boldfaced instead of 48.0.

One more thing, which may be off the focus of this paper: I wonder whether the alignment also has some impact on the performance of the AD tasks.
questions: questions: A more comprehensive ablation study is needed to validate the model's performance.

A better definition of interpretability is helpful to understand the contribution of the proposed approach.
limitations: limitations: 2
review_summary: review_summary: The paper shows some potentials for interpretable autonomous driving using multimodal LLM, but the lack of detailed insights of the model's design and evaluation result in a week rejection.