Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

Authors: jean-baptiste  alayrac
jeff  donahue
pauline  luc
iain  miech
yana  barr
karel  hasson
arthur  lenc
  mensch
ebtesam  almazrouei
hamza  alobeidli
abdulaziz  al- 436 shamsi
alessandro  cappelli
ruxandra  cojocaru
holger  caesar
varun  bankiti
alex h lang
sourabh  vora
venice erin liong
qiang  xu
anush  krishnan
yu  pan
long  chen
oleg  sinavski
jan  hünermann
alice  karnsund
andrew james willmott
danny  birch
daniel  maund
jia  deng
wei  dong
richard  socher
li-jia  li
kai  li
dirk  weissenborn
xiaohua  zhai
thomas  unterthiner
mostafa  dehghani
matthias  minderer
georg  heigold
yifan  du
zikang  liu
junyi  li
wayne  xin
Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-27 00:45:17

Abstract: Vision-Language Models (VLMs) and Multi-Modal Language Models (MMLMs) have become prominent in autonomous driving research, as these models can provide interpretable textual reasoning and responses for end-to-end autonomous driving safety tasks using traffic scene images and other data modalities. However, current approaches to these systems use expensive large language model (LLM) backbones and image encoders, making such systems unsuitable for real-time autonomous driving systems where tight memory constraints exist and fast inference time is necessary. To address these previous issues, we develop EM-VLM4AD, an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations, while also achieving higher BLEU-4, METEOR, CIDEr, and ROGUE scores than the existing baseline on the DriveLM dataset. EM-VLM4AD also exhibits the ability to extract relevant information from traffic views related to prompts and can answer questions for various autonomous driving subtasks. We release our code to train and evaluate our model at https://github.com/akshaygopalkr/EM-VLM4AD.

Main paper: 
 vision-language models: 
 Initially designed to operate on sequence data, Transformers [33] achieved state-of-the-art performance for natural language processing tasks. This propelled the development of Large Language Models, which learn general statistical properties of language through pretraining Encoder[9], Encoder-Decoder [29] , and Decoder [2, 27, 32] Transformer architectures on a large corpus of tokens. These pretrained models can then be finetuned for downstream, more specialized language tasks. Dosovitskiy et al. text and image correlations. This general approach to multimodal learning has sparked a variety of vision-language models. Radford et al. [28] devise a pre-training task of matching text captions with images to develop CLIP, which learns state-of-the-art image representations and exhibits strong zero-shot transfer capabilities for many image classification tasks. BLIP-2 [20] introduces a two stage pretraining process to train a Querying Transformer "QFormer" that serves as a intermediary between a frozen image encoder and language model. This approach outperforms much larger vision-language models such as Flamingo [1] and is capable of zero-shot image-to-text generation. Instruct-BLIP [7] builds off BLIP-2 and is a general-purpose VLM that aggregates public vision-language datasets and transforms them into an instruction tuning format. The VLM most similar to the model introduced in this paper is VL-T5 [6], which extends a pre-trained T5 to learn to generate text labels conditioned on a combination of a text and image embedding. Using a pre-trained LLM as a framework for multi-modal tasks harnesses the text generation ability of these models, critical for the question-answering task in our research. Despite their strong performance across many tasks, deploying these large models, which often exceed one billion parameters, is difficult for real-time appli- In the realm of autonomous driving, modalities beyond text and image such as LiDAR, radar, or video offer important features for many downstream tasks. However, most visionlanguage models are pre-trained for single-image singletext problems, making it unfeasible to directly input multiple images or modalities with a piece of text [37] . Consequently, it is necessary to consolidate multiple modalities and text into a single embedding that can be used by a VLM. DriveGPT4 [38] encodes video input by pooling CLIP visual encodings of each video frame. DriveMLM's[31] multimodal tokenizer uses QFormer to embed video and LiDAR data, and then concatenates these embeddings with text and system message embeddings. Wu et al. [37] find that using gated attention pooling across each individual image embedding helps introduce more non-linearity and extracts visual information across multiple images. Importantly, this gated attention method introduces a negligible amount of computational overhead, rendering it an ideal choice for our model to aggregate multi-view traffic scene images into a unified embedding. 
methods: 
 Our model for Visual Question Answering (VQA) in Autonomous Driving, EM-VLM4AD, consists of a custom image embedding network and a pre-trained T5 language model [29] . We describe these following modules and the overall training process in this section.To tackle multi-view (Front, Front-Left, Front-Right, Back, Back-Left, Back-Right) QA tasks for autonomous driving, individual image embeddings need to be aggregated into a single embedding. This unified embedding can then be concatenated with a text embedding to serve as input to the LM.In typical VLMs, the image embedding process uses models like CLIP or object detection networks, resulting in a slow extraction process. To address this, we adopt the patch pro- This leaves us with 6 distinct individual image embeddings from each view, which now need to be combined. We first flatten each image embedding into a one-dimensional vector and then use gated pooling attention as described by Wu et al. [37] . Given the individual image embeddings V i , gated pooling attention learns a single embedding:227 V = N i=1 α i V i (1)in which α i are weights for the ith image such that 229 N i=1 α i = 1 that are calculated using: 230 α i = exp{w T (tanh(ZV T i ) ⊗ sigm(GV T i ))}This section presents an analysis of the quantitative, qual-294 itative, and computational performance of EM-VLM4AD.We use the following metrics commonly used in image cap-296 tioning tasks to assess the quality of the model-generated answers:• BLEU-4 [26] : Measures how many 4-grams in the generated text match those in the reference text.• ROUGE-L [23] : Calculates sentence similarity scores using the longest common sub-sequence between the generated text and ground-truth text.• METEOR [3]: Considers exact matches, stemming, synonymy, and word order to measure alignment between model outputs and references.• CIDEr [34] : To account for lexical and semantic similarity between the generated and reference text, CIDEr weights n-grams with their corresponding TF-IDF weight. This helps de-emphasize n-grams that commonly occur across all examples that may not have important meaning.For computational analysis, we aim to analyze the memory and computational efficiency of our model, essential aspects in real-time systems where resource constraints exist and inference efficiency is paramount. 
quantitative results: 
 We evaluate the BLEU-4, ROUGE-L, METEOR, and CIDEr scores using the test set of unseen traffic scenes we create. Currently, the only existing approach on the Driv- The superior performance of EM-VLM4AD with the Adding training techniques such as distillation [17] with larger vision-language models, which have a better understanding of grammar rules, will help this smaller model learn these complex rules. EM-VLM4AD also struggles with behavior related questions, where the prompt is "Predict the behavior for the ego vehicle". Adding temporal context through inputting multi-view videos to our network would improve results on this type of question, since behavior related questions often need more than one frame to make accurate predictions. 
conclusion: 
 We  
: 
 language models and multimodal language models tailored for autonomous driving applications [5, 24, 31, 32, 38] .These models cover various aspects of autonomous driving including closed-loop control, perception tasks, and traffic agent behavior analysis.Typically, the process in a VLM is the following: vision and text features are encoded separately, then fused together through a concatenation or projection layer, and then finally fed into an LLM to output some probability distribution over the vocabulary [37] . While generating text embeddings is relatively low-cost, the LM and image embeddings can often entail high computational costs. In real-time systems such as autonomous driving, prioritizing the development of VLMs with efficient inference times is crucial for practical deployment in vehicles. However, current research in applying multimodal language models to autonomous driving predominantly use large models such as BLIP-2 [20] , GPT 3.5 [24] , and LLaMA-7b [32] , all of which contain over one billion parameters. Consequently, these models require expensive hardware and longer inference times, limiting their potential to be applied in current vehicles and accessibility for researchers with limited computational resources. This paper focuses on the development of lightweight vision-language models with less than one billion parameters than can accurately and efficiently answer questions related to autonomous driving safety tasks. We develop the model EM-VLM4AD: Efficient, Multi-Frame Vision-Language Model for Autonomous Driving. We use the DriveLM dataset [31] , which offers real, multi-view traffic scene images paired with question/answer pairs to train this model. Our contributions are as follows:• We develop an efficient, smaller vision-language model EM-VLM4AD that consumes at least 10x less memory and floating point operations (FLOPs) than current AD-VLMs, and can also respond to questions conditioned on multiple frames.• We explore two different lightweight LM backbones for EM-VLM4AD: a finetuned Text-to-Text Transfer Transformer (T5) Base LM and an 8-bit quantized T5-Large LM finetuned using low-rank adaptation (LoRA) [18] .  
 
 Appendix:  



Reviews: review: title: title: Review
main_review: main_review: This paper introduces a few improvements to reduce the memory consumption and training/inference speed of Driving VisionLangauge Models (VLMs). It does so by: 1) removing the ViT encoder while keeping the patch embedding layer, 2) fusing embeddings of multi-cameras into a single embedding using a gated pooling attention method, 3) and using smaller LMs with optional quantization and LORA techniques.

This paper is highly relevant to our workshop and provides empirical results to support its claims. Thus I recommend acceptance.