MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Driving

## Abstract

Abstract Vision-language models (VLMs) serve as general-purpose end-to-end models in autonomous driving, performing subtasks such as prediction, planning, and perception through question-and-answer interactions. However, most existing methods rely on computationally expensive visual encoders and large language models (LLMs), making them difficult to deploy in real-world scenarios and real-time applications. Meanwhile, most existing VLMs lack the ability to process multiple images, making it difficult to adapt to multi-camera perception in autonomous driving. To address these issues, we propose a novel framework called MiniDrive, which incorporates our proposed F eature Engineering Mixture of E xperts (FE-MoE) module and D ynamic I nstruction Adapter (DI-Adapter). The FE-MoE effectively maps 2D features into visual token embeddings before being input into the language model. The DI-Adapter enables the visual token embeddings to dynamically change with the instruction text embeddings, resolving the issue of static visual token embeddings for the same image in previous approaches. Compared to previous works, MiniDrive achieves state-of-the-art performance in terms of parameter size, floating point operations, and response efficiency, with the smallest version containing only 83M parameters.

## 1 Introduction

As large-scale pretraining techniques develop, vision-language models (VLMs), due to their powerful visual reasoning capabilities, become the primary choice for visual question answering tasks across various domains. Similarly, in the field of autonomous driving, question-answering reasoning based on VLMs has the potential to become a new method of interaction between drivers and vehicles. This natural language question-answering approach enhances the interpretability of autonomous driving. VLMs integrate perception, prediction, and decision-making during driving into a unified model within autonomous driving systems, functioning as an end-to-end general model for solving various sub-tasks in autonomous driving. Numerous VLMs applications in autonomous driving systems already exist, where these models begin to perform tasks such as closed-loop control, scene perception, and traffic agent behavior analysis in autonomous systems Chen et al. (); Mao et al. (); Sima et al. (); Xu et al. ().

Figure: Figure 1: (a) shows the input format of MiniDrive and the tasks it can perform. (b) compares the average evaluation of multiple-image inputs on the Drive-LM evaluation system with related models. (c) compares the average evaluation of single-image inputs on the CODA-LM evaluation system with related models. Minidrive outperforms open-source models larger than 7B and approaches the performance of commercial models.
Refer to caption: x1.png

VLMs primarily consist of two main modules, including a vision encoder and an LLM for text generation. This implies that deploying VLMs in a system requires high computational costs and hardware resources. In autonomous driving systems, developing VLMs that consume fewer resources, have lower computational costs, and respond faster becomes a key consideration for practical deployment. However, current research on multimodal large models in autonomous driving mainly focuses on models with over a billion parameters, such as BLIP-2 Li et al. (), LLaMA-7B Touvron et al. (), GPT-3.5, and GPT-4 Achiam et al. (), with the vision encoders relying on pre-trained models based on the Transformer architecture, like CLIP Radford et al. (). This consumes substantial computational resources and hardware costs, and requires longer response times, making them challenging to apply and deploy in practice. Recently, EM-VLM4ADGopalkrishnan et al. () introduces a lightweight architecture, attempting for the first time to apply lightweight models in the field of autonomous driving and achieving excellent results. However, there remains a certain gap in response performance compared to billion-parameter models like DriveLM-Agent Sima et al. (). Additionally, autonomous driving typically involves multiple images from different angles, such as front, front-right, front-left, rear, rear-right, and rear-left. Most existing VLMs are trained on single images, making them unsuitable for inputting multiple driving scene images.

To address these challenges, this paper introduces a novel vision-language model called MiniDrive. Unlike traditional mainstream visual-language models, MiniDrive is not a unified model based on the Transformer architecture.
We use the efficient backbone network model UniRepLKNet Ding et al. (), which is based on large convolutional kernels, as the vision encoder. We propose the Feature Engineering Mixture of Experts (FE-MoE) and the Dynamic Instruction Adapter (DI-Adapter) to sequentially process visual features and obtain visual tokens before inputting them into the language model. Specifically, UniRepLKNet captures the 2D features of images, and FE-MoE processes multiple 2D features, mapping them into text tokens for input into the language model without requiring stage-wise training for cross-modal fine-grained alignment. Additionally, the DI-Adapter is introduced to enable the mapped visual tokens (i.e., text tokens used as input to the language model) to dynamically adapt to user text instructions, effectively aiding cross-modal understanding between text and images.
As shown in Figure 1(a), MiniDrive processes multiple input images along with user instructions to generate natural language responses. It encompasses the most critical capabilities in autonomous driving, including perception, planning, and prediction question-answering abilities. In Figure 1(b), we illustrate that MiniDrive is a lightweight visual-language model with a minimal parameter size, memory footprint, and FLOP count. It can be fully trained with multiple instances on a single RTX 4090 GPU with 24GB of memory. For instance, $\text{MiniDrive}_{224}$ has only 83M parameters and a FLOP count of merely 5.9B, which is significantly lower than current visual-language models used in autonomous driving. In terms of response performance, MiniDrive outperforms a series of previous models in question-answering capabilities. Notably, its response quality exceeds that of models with billions of parameters. Additionally, MiniDrive supports both single and multiple image inputs. In Figure 1(c), MiniDrive outperforms open-source models with 7B parameters and above on the single-image evaluation system CODA-LM Li et al. (), approaching the performance of closed-source commercial models. Here are our main contributions:

- •
We develop autonomous driving VLMs—MiniDrive, which address the challenges of efficient deployment and real-time response in VLMs for autonomous driving systems while maintaining excellent performance. The training cost of the model is reduced, and multiple MiniDrive models can be fully trained simultaneously on an RTX 4090 GPU with 24GB of memory.
- •
MinDrive is attempting for the first time to utilize a large convolutional kernel architecture as the vision encoder backbone for autonomous driving vision-language models, enabling more efficient and faster extraction of 2D features at different image levels. We propose Feature Engineering Mixture of Experts (FE-MoE), which addresses the challenge of efficiently encoding 2D features from multiple perspectives into text token embeddings, effectively reducing the number of visual feature tokens and minimizing feature redundancy.
- •
This paper introduces the Dynamic Instruction Adapter through a residual structure, which addresses the problem of fixed visual tokens for the same image before being input into the language model. The DI-Adapter enables visual features to dynamically adapt to different textual instructions, thereby enhancing cross-modal understanding.
- •
We conduct extensive experiments on MiniDrive, achieving state-of-the-art performance compared to autonomous driving VLMs with multi-view image inputs on Drive-LM. Further, we outperform general open-source VLMs (7B) with single-image inputs on CODA-LM by an average of 13.2 points. We open-source all our resources to foster community development.

Figure: Figure 2: Comparison of the MiniDrive architecture with mainstream architectures. (a) Existing vision-language models primarily use a Transformer-based visual encoder to learn image patches as visual tokens. These visual tokens remain unchanged regardless of the user’s questions. (b) Our architecture employs a more efficient large convolutional kernel as the visual encoder, learning 2D features of the image as visual tokens. These visual tokens change in response to different user questions.
Refer to caption: x2.png

## 2 Related Work

### 2.1 Vision-Language Models

The success of the Transformer architecture Vaswani () drives the development of LLMs. In the field of computer vision, Dosovitskiy et al. () propose the Vision Transformer (ViT), which divides images into patches and processes them based on the Transformer architecture, adapting it to computer vision tasks with success. Both images and natural language can be effectively learned and represented by the Transformer architecture. A pioneering work is CLIP Radford et al. (), which uses contrastive learning for image-text alignment training, demonstrating superior zero-shot capabilities in image classification tasks. Llava Liu et al. () freezes CLIP’s vision encoder (ViT) and adds a linear projection layer between the vision encoder and LLMs, aiming to map visual output representations into textual space. Similarly, BLIP-2 Li et al. () aligns visual and textual representations through a more complex Q-Former. InstructBLIP Panagopoulou et al. () builds on BLIP-2 with instruction fine-tuning on public visual question-answering datasets. MiniGPT-4 Zhu et al. () combines a frozen vision encoder and Q-Former with the similarly frozen LLM Vicuna, aligning them with a single projection layer. Llava-1.5v Liu et al. () achieves state-of-the-art performance in 11 benchmarks by using CLIP-ViT-L-336px with a multilayer perceptron (MLP) projection layer and adding VQA data tailored for academic tasks with simple response formatting prompts, significantly improving data efficiency. Phi-3-mini Abdin et al. () features a default 4K context length and introduces a version extended to a 128K context length using LongRope technology, while employing a block structure similar to Llama-2 and the same tokenizer, enabling a lightweight multimodal model. Despite the powerful capabilities of these multimodal large models and their trend toward lightweight design, their parameter counts exceed one billion, making deployment and real-time use on many hardware platforms challenging. Therefore, research and development of efficient vision-language models with smaller parameter sizes and lower computational costs are necessary.

Figure: Figure 3: MiniDrive Structural Details. In Figure (a), the overall architecture of MiniDrive is presented. The image features from the vision encoder input are processed by the FE-MoE and DI-Adapter with residual connections, resulting in visual token embeddings. These embeddings, along with text embeddings, are then fed into the T5-Small language model, producing the output. In Figure (b), the specific framework of FE-MoE is shown. The image is input into UniRepLKNet, producing feature maps at different levels. These feature maps are then fed into the FE-MoE module, where the Gate network generates weights. The 2D visual features are further assigned to different experts for feature mapping and weighted summation.
Refer to caption: x3.png

### 2.2 Autonomous Driving Based on LLMs

LLMs effectively enhance both the explainability of autonomous driving systems and their interaction with humans Greer and Trivedi (). These advantages lead researchers to incorporate multi-modal data from autonomous driving into LLMs’ training, aiming to build multi-modal large models for autonomous driving. Chen et al. () aligned vectorized modal information with LLaMA-7B Touvron et al. () to train a question-answering model for autonomous driving. The training process follows a two-stage approach: in the first stage, vector representations are aligned with a frozen LLaMA, while in the second stage, LoRA Hu et al. () is used to fine-tune the language model. DriveGPT4 Xu et al. () also employs LLaMA as its large language model, using CLIP as the visual encoder. It generates corresponding answers by inputting both visual and textual information. DriveGPT4 leverages ChatGPT/GPT-4 to generate an instruction dataset and trains on this dataset. However, DriveGPT4 only uses single-perspective images, limiting its ability to handle more comprehensive understanding in autonomous driving scenarios. Wang et al. () developed DriveMLM , which uses LLaMA-7B as the foundational language model and ViT-g/14 as the image encoder. This model processes multi-view images, LiDAR point clouds, traffic rules, and user commands to achieve closed-loop driving. Inspired by the chain-of-thought approach in large language models Wei et al. (), Sha et al. () proposed a chain-of-thought framework for driving scenarios, using ChatGPT-3.5 to provide interpretable logical reasoning for autonomous driving. Mao et al. () introduced GPT-Driver, which uses ChatGPT-3.5 to create a motion planner for autonomous vehicles, and GPT-Driver reframes motion planning as a language modeling task by representing the planner’s inputs and outputs as language tokens. Sima et al. () released the DriveLM dataset, a graphic visual question-answering dataset with question-answer pairs related to perception, behavior, and ego-vehicle planning, based on multi-view image data from the NuScenes dataset Caesar et al. (). To establish baselines, Li et al. () fine-tuned BLIP-2 on this new dataset. EM-VLM4AD Gopalkrishnan et al. () introduced Gated Pooling Attention (GPA), which aggregates multiple images into a unified embedding and connects it with text embeddings as input to LLMs, achieving promising results on the DriveLM dataset.

While existing work provides significant value and demonstrates strong capabilities for autonomous driving, most models have over a billion parameters. They are largely based on large-scale language models such as GPT-3.5 and LLaMA, and rely on vision encoders built on the ViT architecture, such as CLIP, ViT-g/14, and ViT-B/32. This results in high computational costs, making these models unsuitable for online scenarios. Although there is a trend towards developing lightweight models for autonomous driving, their performance still falls short compared to larger models. In Figure 2, we summarize the architectures of the current mainstream vision-language models and compare them with the architecture of MiniDrive. Existing vision-language models primarily divide images into several patches using a Transformer-based visual encoder, learning each patch as tokens for the input language model. Additionally, during inference, the visual tokens remain fixed regardless of how the user’s query changes. Our architecture extracts multi-level 2D features from the image using a vision encoder based on large convolutional kernels (LKNet), further extracting and learning these features to map them as tokens for the input language model. CNNs with large convolutional kernels are more efficient and lightweight (Ding et al., , ). Meanwhile, changes in the user’s query dynamically alter the visual tokens.

## 3 Method

MiniDrive is a vision-language model in the field of autonomous driving, designed to perform visual question answering tasks. It generates text responses by receiving an image and user instruction text as input. In this section, we first provide a detailed introduction to the overall framework of MiniDrive, followed by a specific explanation of the technical details and principles of each module, including the Vision Encoder, Feature Engineering Mixture of Experts (FE-MoE) , and Dynamic Instruction Adapter (DI-Adapter).

### 3.1 Model Architecture

Figure 3 (a) illustrates the overall structure of MiniDrive. In MiniDrive, there are primarily two branches: vision and text. On the vision side, given n images from an autonomous vehicle as input to the visual encoder, $\mathbb{R}^{3\times H\times W}$, each image receives a set of deep 2D feature representations $V_{2D}\in\mathbb{R}^{c\times h\times w}$. These features are then input into the FE-MoE, where multiple experts compress the information along the channel dimension $c$ and expand it along the height $h$ and width $w$ dimensions to generate new 2D feature representations. In the FE-MoE, the Gate network determines which experts are more suitable for processing each image, assigning different weight values to each expert. Finally, the new 2D feature representations are combined through a weighted sum to produce the new feature set $V_{moe}\in\mathbb{R}^{c^{\prime}\times h^{\prime}\times w^{\prime}}$. Flatten $V_{moe}$ to obtain $V\in\mathbb{R}^{l_{1}\times dim_{1}}$, where the length $l_{1}$ corresponds to the previous $c^{\prime}$, and the dimension $dim1$ corresponds to the previous $h^{\prime}\times w^{\prime}$. Then, the Projection layer maps $dim_{1}$ to $dim$, resulting in $V\in\mathbb{R}^{l_{1}\times dim}$.

On the text side, the user’s natural language instruction is processed through a Tokenizer and Embedding layer to obtain the token embeddings of the text $T\in\mathbb{R}^{l_{2}\times dim}$. The embedded sequence of the text $T$ is used as the key (k) and value (v), while the visual embedding sequence $V$ at this stage is used as the query (q). These are fed into the DI Adapter to compute a new visual embedding sequence $V_{1}$, which now incorporates the contextual information from the text embedding $T$, enabling better cross-modal understanding or decision-making. $V_{1}$ is then combined with $V$ through a residual connection to form the sequence $V_{input}$, while $T$ is treated as $T_{input}$. The concatenation $[V_{input},T_{input}]$ is then used as input to the language model. The language model decodes to generate a word sequence with the highest predicted probability. The entire framework efficiently processes multi-image input information, dynamically responding to user queries.

### 3.2 Vision Encoder

As shown in Figure 3(b), the backbone network of the Vision Encoder is based on the large-kernel neural network UniRepLKNet Ding et al. (), which demonstrates excellent performance across multiple modalities. It effectively leverages the characteristics of large-kernel convolutions, enabling a wide receptive field without the need to go deep into the network layers. While maintaining efficient computation, it also achieves or surpasses the performance of current state-of-the-art techniques across various tasks. This generality and efficiency make it a powerful model with potential in a wide range of perception tasks. A brief review of the overall architecture of UniRepLKNet, as shown in Figure 4, reveals that it primarily consists of multiple sequentially connected Stage layers. Each Stage is mainly composed of a series of Lark Blocks and Smak Blocks. In MiniDrive, we use UniRepLKNet as the backbone of the vision network, where an image is input and the output feature map $F1\in\mathbb{R}^{c\times h\times w}$ is obtained from the final Stage n.

Figure: Figure 4: UniRepLKNet generates feature maps. We obtain the set of feature maps from each image propagated to the final stage.
Refer to caption: x4.png

**Table 1: Performance on DriveLM. We compare the response performance of different models on the same test set. Bold indicates the highest value, while an underline indicates the second-highest value.**
| Method | Ref. | DriveLM |  |  |  |
| --- | --- | --- | --- | --- | --- |
| BLEU-4 $\uparrow$ | METEOR $\uparrow$ | ROUGE-L $\uparrow$ | CIDEr $\uparrow$ |  |  |
| $\text{EM-VLM4AD}_{Base}$ | CVPR’ 24 | 45.36 | 34.49 | 71.98 | 3.20 |
| $\text{EM-VLM4AD}_{QLarge}$ | CVPR’ 24 | 40.11 | 34.34 | 70.72 | 3.10 |
| DriveLM-Agent | ECCV’ 24 | 53.09 | 36.19 | 66.79 | 2.79 |
| $\text{MiniDrive}_{224}$ (Ours) | - | 49.70 | 36.30 | 73.30 | 3.28 |
| $\text{MiniDrive}_{384}$ (Ours) | - | 50.20 | 37.40 | 73.50 | 3.32 |

### 3.3 Feature Engineering Mixture of Experts

In Figure 3(b), we present the specific structure of the FE-MoE, which is designed to handle 2D input features from multiple images. Each input image corresponds to a feature map $F_{1}\in\mathbb{R}^{c\times h\times w}$ output by the Vision Encoder. To further process the 2D feature representations of each image efficiently, they are input into the FE-MoE. First, $F_{1}$ is used by the Gate network to obtain the expert selection weights corresponding to the sample. The Gate network mainly consists of convolutional layers, max-pooling layers, and linear layers, as shown in the following equation:

$$ $Weights=Softmax(Gate(F_{1})).$ (1) $$

Then, $F_{1}$ passes through each expert network, resulting in a new feature representation $F_{2}\in\mathbb{R}^{c^{\prime}\times h^{\prime}\times w^{\prime}}$. Each expert network mainly consists of a deconvolutional layer, a ReLU layer, and a convolutional layer. The deconvolutional layer first performs an initial upsampling mapping, increasing the dimensions of the feature map’s width and height to expand the amount of information, facilitating subsequent mapping learning. At the same time, it reduces the number of channels in the original feature map to minimize data redundancy and select the most important 2D feature representation information, significantly simplifying the number of subsequent visual tokens. The convolutional layer further transforms the features to enhance the learning capacity of the experts. The formula is shown as follows:

$$ $F_{2}=Conv(ReLu(Deconv(F_{1}))),$ (2) $$

$$ $\displaystyle F_{1}$ $\displaystyle\in R^{c\times h\times w}\rightarrow F_{2}\in R^{c\downarrow \times h\uparrow\times w\uparrow}$ (3) $\displaystyle=F_{2}\in R^{c^{\prime}\times h^{\prime}\times w^{\prime}},$ $$

where, $c\downarrow$ denotes a decrease in the number of channels, while $h\uparrow$ and $w\uparrow$ indicate an increase in the height and width of the feature map, respectively. In this context, $F_{2}$ represents the output of an individual expert. Given that the weight for the $i$-th expert for an image is $W_{i}$, and the output from this expert is $F_{i}$, with the total number of experts being $N$, the feature $V_{moe}$ of the image after processing by the FE-MoE model is expressed by the following formula:

$$ $F_{i}=Expert_{i}(VisionEncoder(Image)),$ (4) $$

$$ $V_{moe}=\sum_{i=1}^{N}W_{i}\cdot F_{i}.$ (5) $$

Figure: Figure 5: Examples of MiniDrive’s Response. The color blue represents the user command querying for multi-image input. The color red represents the activation response generated by MiniDrive corresponding to the text.
Refer to caption: x5.png

### 3.4 Dynamic Instruction Adapter

In previous vision-language models, image representations are fixed before being input into the language model, and they correspond to various text representations before entering the language model for computation. To enable image representations to dynamically transform according to different text representations before being input into the language model, thereby improving cross-modal understanding, we introduce the Dynamic Instruction mechanism and design the Dynamic Instruction Adapter. We use the text input sequence $T$ as the key ($k$) and value ($v$), and the image input sequence $V$ as the query ($q$). Through cross-attention, we compute the fused sequence $V^{\prime}$ that incorporates textual contextual information. The formula is shown as follows:

$$ $\displaystyle V^{\prime}=\text{CrossAtt.}(q=V,k=T,v=T).$ (6) $$

The sequence in the residual channel is connected via a residual connection with the output sequence of the projection layer, serving as the visual representation prior to the input into the language model. The training of additional language model outputs can be found in the appendix.

## 4 Experiments

In this section, we conduct extensive experiments on MiniDrive and analyze the experimental results, including the analysis of quantitative results, computational efficiency, and examples. Finally, ablation experiments are performed to verify the effectiveness of the module.

**Table 2: The computational analysis of the model includes a comparison of the parameter size, floating point operations (FLOPs), and GPU memory usage.**
| Model | Parameters | FLOPs | Memory (GB) |
| --- | --- | --- | --- |
| DriveMLM | 8.37B | 535B | 36 |
| Drive-GPT4 | 7.3B | 329B | 29.2 |
| LLM-Driver | 7B | 268B | 28 |
| DriveLM-Agent | 3.96B | 439B | 14.43 |
| $\text{EM-VLM4AD}_{Base}$ | 345M | 9.9B | 1.97 |
| $\text{MiniDrive}_{224}$ (ours) | 83M | 5.9B | 1.03 |

### 4.1 Experimental Settings

Datasets  We conduct experiments on the DriveLM dataset. To ensure the fairness of the experiments, we use the same training and evaluation protocol as EM-VLM4AD on the DriveLM dataset, which includes the same training, validation, and test sets. The training set contains approximately 340,184 different multi-view/QA pairs, while the test set and validation set each contain 18,899 different multi-view/QA pairs.
Models  We construct different versions of MiniDrive based on various UniRepLKNet models as the vision backbone, with the main difference being their ability to learn visual token embeddings. We use UniRepLKNet-A as the vision backbone for processing images with a resolution of 224×224, and UniRepLKNet-S for processing images with a resolution of 384×384. We use the T5-small language model as the foundation.
Evaluation metrics  To ensure the fairness and reproducibility of the evaluation, we use the same evaluation method as EM-VLM4AD on the DriveLM dataset, assessing the model from four different perspectives: BLEU-4 Papineni et al. (), ROUGE-L Lin (), METEOR Banerjee and Lavie (), and CIDEr Vedantam et al. ().
Implementation details  Each model is trained on a single RTX 4090 GPU. The vision encoder is frozen, while the other parameters are trained with an initial learning rate of 1e-4 and a weight decay of 0.05. Each model is trained for 6 epochs on the training set. Note that in subsequent experiments, MiniDrive refers to the $\text{MiniDrive}_{224}$ version by default, with the number of tokens per image set to 16 and the number of experts set to 4.

### 4.2 Quantitative Results

In Table LABEL:drivelm, we compare the evaluation results of MiniDrive with previous works on the test set, including EM-VLM4AD Gopalkrishnan et al. () and Drive-Agent Sima et al. (). In terms of overall performance on the metrics, both $\text{MiniDrive}_{224}$ and $\text{MiniDrive}_{384}$ outperform previous methods, although DriveLM-Agent surpasses us in BLEU-4, its parameter count is significantly larger than ours, reaching 3.96B.

**Table 3: Ablation among modules. We compare the response performance of different models on the same test set. Bold indicates the highest value, while an underline indicates the second-highest value.**
| FE-MoE | DI-Adapter | DriveLM |  |  |  |
| --- | --- | --- | --- | --- | --- |
| BLEU-4 $\uparrow$ | METEOR $\uparrow$ | ROUGE-L $\uparrow$ | CIDEr $\uparrow$ |  |  |
| – | – | 45.70 | 34.09 | 69.74 | 3.07 |
| ✓ | – | 48.30 | 35.40 | 72.10 | 3.23 |
| – | ✓ | 48.00 | 35.70 | 72.00 | 3.16 |
| ✓ | ✓ | 49.70 | 36.30 | 73.30 | 3.28 |

### 4.3 Computational Analysis

In this section, we primarily compare the differences between MiniDrive and a range of existing vision-language models in terms of parameter count, Floating Point Operations (FLOPs), and memory usage (GB). The results are shown in Table 2. Using an input image resolution of 224 as an example, MiniDrive demonstrates superior performance in all three aspects.

**Table 4: Performance on CODA-LM. MiniDrive is compared with Multimodal Large Language Models. Bold indicates the highest value, while an underline indicates the second-highest value.**
| Method | Parameters | General$\uparrow$ | Regional Perception $\uparrow$ | Suggestion$\uparrow$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text-Score | ALL | Vehicle | VRU | Cone | Barrier | Other | Text-Score |  |  |
| LLaVA1.5 Liu et al. () | 7B | 22.60 | 34.78 | 40.00 | 28.00 | 32.22 | 24.00 | 10.00 | 14.20 |
| Qwen-VL-Chat Bai et al. () | 7B | 26.00 | 53.33 | 57.76 | 60.00 | 48.89 | 44.29 | 35.71 | 35.40 |
| Qwen-VL-Max Bai et al. () | api-model | 34.60 | 68.17 | 69.83 | 56.00 | 80.00 | 59.29 | 65.71 | 47.40 |
| GPT-4o OpenAI () | api-model | 45.00 | 73.76 | 75.69 | 66.00 | 75.56 | 69.29 | 70.00 | 55.50 |
| $\text{MiniDrive}_{224}$(Ours) | 83M | 21.60 | 62.15 | 62.93 | 36.00 | 86.67 | 59.29 | 48.57 | 45.40 |
| $\text{MiniDrive}_{384}$(Ours) | 137M | 24.60 | 66.34 | 67.41 | 36.00 | 84.44 | 62.86 | 62.85 | 45.44 |

### 4.4 Qualitative Examples

In Figure 5, we present the actual responses of MiniDrive on unseen samples across three different tasks. To provide an interpretability analysis of MiniDrive’s perception of multi-view image inputs, we analyze the activation maps of MiniDrive in various scenarios. In Figure 5 (a), MiniDrive demonstrates perceptual question-answering for multiple image inputs, with the blue box indicating the image referenced by the user’s instruction for the position "back left." The red box corresponds to MiniDrive’s response, primarily focusing on that image, identifying "many cars, one truck, and one pedestrian" at the specified location. In Figure 5 (b), MiniDrive demonstrates planning question-answering for multiple image inputs. Based on the user’s instruction and the spatial term "CAM_FRONT", MiniDrive focuses on the red box on the left side of the corresponding front image. This attention aligns with the elements that humans consider when making planning decisions, including the traffic lane markings and vehicles on the left side of the ego car. In Figure 5 (c), MiniDrive demonstrates predictive question-answering for multiple image inputs. Based on the user’s instruction to predict the movement of the pedestrian in the "front left" position, MiniDrive focuses on the pedestrian in the corresponding positional image, highlighted by the red box. Taken together, the objects that MiniDrive focuses on in the activation map align with the reasoning followed by human drivers during driving, indicating that MiniDrive possesses a certain level of reliability and interpretability.

### 4.5 Ablation Studies

To validate the effectiveness of each module, we design a series of ablation experiments. In Table LABEL:abl1, we investigate the impact of FE-MoE and Dynamic Instruction Adapter (DI-Adapter) on MiniDrive. When FE-MoE and Dynamic Instruction Adapter are introduced separately, the results of various metrics improve, and when both modules are introduced simultaneously, a better effect is achieved. This indicates the effectiveness of the mechanisms between the modules. The details of other ablation experiments can be found in the appendix.

## 5 Further analysis

Although MiniDrive is designed as an autonomous driving question-answering model for receiving multi-image inputs, it extracts, compresses, and re-learns the information from multiple images as Text Tokens for the language model. However, it can still be used for single-image input tasks. We compare it with existing mainstream open-source and closed-source general models on CODA-LM, as shown in Table 4. It is evident that despite MiniDrive having only 83M parameters, it demonstrates superior performance, outperforming open-source models and approaching the performance of closed-source models. Due to the issue with the distribution of the training data, we believe that this is the main factor contributing to MiniDrive’s strong ability to recognize "Cone". Further details can be found in the appendix.

## 6 Conclusion

In this paper, we present MiniDrive, a state-of-the-art lightweight vision-language model for autonomous driving. We introduce the FE-MoE and DI-Adapter mechanisms, proposing a novel approach that maps 2D convolutional features into text tokens for language models. Our model achieves outstanding results on two datasets, DriveLM and CODA-LM. In the future, we aim to develop a real-time response model with video input to further advance autonomous driving technology.

## 7 Limitations

MiniDrive builds VLMs specific to the autonomous driving domain and has achieved excellent results on current mainstream benchmarks. However, it still lacks a certain level of generalization, which we believe is due to the limitations of the training samples. The existing autonomous driving field requires more public datasets and efforts to develop them. Additionally, MiniDrive’s training is primarily focused on instruction-based datasets, and it continues to experience hallucination issues.

## Appendix A Training

Due to the effectiveness of each module in MiniDrive and the consumption of only a small amount of computational resources, the training employs a straightforward full-parameter approach, meaning all parameters in MiniDrive are included in the training process. Meanwhile, MiniDrive is freezed the vision encoder. MiniDrive is supervised by label text, with loss calculated using cross-entropy, which quantifies the difference between the text sequence generated by the model and the target text. The formula is as follows:

$$ $\text{Loss}=-\sum_{i=1}^{n}y_{i}\log(p_{i}),\quad(3)$ $$

where $n$ is the number of tokens, $y_{i}$ is the true label for token $i$, and $p_{i}$ is the predicted probability for token $i$.

## Appendix B More Ablation Studies

We configure the number of tokens per image in $\text{MiniDrive}_{224}$ to 16 and set the number of experts to 2, 4, and 6 for testing on DriveLM. Additionally, we configure the number of experts in $\text{MiniDrive}_{224}$ to 4 and set the number of tokens per image to 8, 16, and 32 for testing on DriveLM. The results are shown in Figure 6. When the tokens become larger, the language model’s ability to learn longer sequences decreases. As the number of experts increases, the training difficulty of the FE-MoE network grows, leading to a decline in learning performance.

Figure: Figure 6: More Ablation Studies on Tokens and Experts
Refer to caption: x6.png

## Appendix C More Examples

In this section, we demonstrate more response instances of MiniDrive. In Figure 7, we showcase question-answering instances on DriveLM. While in Figure 8, we present question-answering instances on CODA-LM. We train on the official training set provided by CODA-LM and conduct testing on the Mini set. The parameter settings are consistent with those described in the experimental section.

Figure: Figure 7: Examples of MiniDrive’s Response on DriveLM.
Refer to caption: x7.png

Figure: Figure 8: Examples of MiniDrive’s Response on CODA-LM.
Refer to caption: x8.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Same as limitations
soundness: 2
strengths: 1. **Efficiency**: *MiniDrive* is a lightweight VLM with low FLOPs, suitable for real-time deployment on limited hardware, making it highly practical for autonomous driving.

2. **Dynamic Adaptation**: The *Dynamic Instruction Adapter* enhances cross-modal understanding by adapting visual tokens to user instructions, improving interaction quality in real-world applications.
summary: The authors proposed MiniDrive framework, a lightweight vision-language model for autonomous driving, optimizing efficiency with reduced parameters. Using the FE-MoE module for visual processing and the DI-Adapter for dynamic instruction response, it achieves competitive performance on the DriveLM dataset while lowering computational costs, making it practical for real-time use on limited hardware​.
weaknesses: This paper, though notable, leans more toward an engineering approach than a research-oriented contribution. I identify the following limitations:

1. **Insignificant Training Cost Reduction**: Reducing training cost is not significant. A comparable 4-bit or 8-bit quantized large language model (LLM) with ~7B parameters can also be fine-tuned on a single RTX 4090 GPU using adapters, which limits the novelty in terms of efficiency.

2. **Limited Benchmarking Scope**: The integration of UniRepLKNet for visual feature extraction and the Mixture of Experts (MoE) design should be evaluated on a broader range of benchmarks beyond autonomous driving (AD) datasets. If the authors focus solely on AD datasets, it would be beneficial to emphasize how the architecture uniquely benefits AD scenarios. Currently, the proposed FE-MoE framework appears generalizable to various visual modality applications, lacking a clear advantage for AD-specific use cases.

3. **Lack of Task-Specific Uniqueness in Dynamic Instruction Adapter**: The Dynamic Instruction Adapter is a promising concept, though it suffers from a similar limitation as (Limination 2) — it lacks specialization for AD tasks, which could limit its applicability in scenarios beyond general-purpose visual adaptation. Also, this idea is not new and similar idea is applied in many other works (e.g. Llama-Adapter [1] and CogVLM [2]).

4. **Ambiguity in the MoE Approach**: The FE-MoE’s primary goal seems to be fusing tokens from different camera sources, yet the reasoning behind using a Mixture of Experts is unclear. In most AD scenarios, information from all cameras is essential. Applying a hard limit (e.g., selecting only the top-k experts, where \( k < 6 \)) risks discarding critical visual data from unselected cameras. Conversely, if \( k = 6 \) (i.e., using all cameras), simpler feature transformation and merging techniques could be more efficient than the current gating + softmax + elementwise weighted merge approach, which substantially increases GPU memory consumption.

5. **Simplistic Illustrative Examples**: Figure 5 does not adequately demonstrate the benefits of MiniDrive over competing frameworks, such as DriveLM-Agent. The examples lack complexity and do not showcase significant advantages.

6. **Incomplete Comparative Evaluation**: In Table 2, models like LLM-Driver and Drive-GPT4 possess explicit waypoint prediction capabilities and are thus evaluated with UniAD metrics. MiniDrive, however, seems like has not implemented waypoint prediction, preventing a direct comparison with these models and leaving its performance on this critical aspect unaddressed.

[1] Zhang, Renrui, et al. "Llama-adapter: Efficient fine-tuning of language models with zero-init attention." arXiv preprint arXiv:2303.16199 (2023).

[2] Wang, Weihan, et al. "Cogvlm: Visual expert for pretrained language models." arXiv preprint arXiv:2311.03079 (2023).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Given the paper’s lack of novelty in its proposed FE-MoE mechanism, absence of real-time performance benchmarks, and incomplete evaluation for autonomous driving environments (e.g., multi-camera setup), its contributions are limited in scope and significance. The paper would benefit from a clearer demonstration of MiniDrive’s practical impact, including concrete real-time performance metrics, and a more thorough investigation of its architectural claims.

1.	Real-Time Performance Validation: Could you provide detailed inference times or benchmarks to clarify MiniDrive’s performance in real-time environments? This would help substantiate the paper’s emphasis on efficiency.
2.	Differentiation of FE-MoE from Existing MoE Architectures: How does FE-MoE specifically improve upon standard MoE frameworks, particularly sparse MoEs, in terms of efficiency for autonomous driving applications?
3.	Multi-Camera Integration Methodology: Could you elaborate on how MiniDrive supports multi-camera setups in autonomous driving, and whether it maintains robust performance across diverse camera angles and resolutions?
4.	Baseline Model Configuration: In Table 3, could you clarify the configuration of the baseline model without FE-MoE, specifically addressing the parameter count and computational requirements to ensure fair comparison?
soundness: 2
strengths: 1.	Goal-Oriented Design for Autonomous Driving: MiniDrive seeks to address the high resource demands of typical vision-language models, specifically aiming to enable real-time processing in the context of autonomous driving.
2.	Efficient Model Parameters and FLOPs: The model claims efficiency in terms of FLOPs and memory usage, potentially supporting multi-instance training on a single GPU, which can be beneficial for applications with limited computational resources.
summary: This paper proposes MiniDrive, a vision-language model optimized for autonomous driving tasks. MiniDrive incorporates the Feature Engineering Mixture of Experts (FE-MoE) and Dynamic Instruction Adapter to dynamically handle visual features and integrate contextual information from text inputs. The paper highlights the model’s lower FLOPs, parameter count, and memory usage compared to larger models, aiming for efficient processing on single GPUs
weaknesses: 1.	Lack of Real-Time Performance Evaluation: Despite the model’s claim of real-time suitability, there is no specific evaluation of inference time or processing speed, which is critical for applications in autonomous driving. The model’s practical performance remains unproven in real-world settings.
2.	Limited Novelty of FE-MoE: 1. The FE-MoE mechanism in MiniDrive employs a continuous weighted-sum approach across multiple experts, similar to the foundational work on Mixture of Experts by Shazeer et al. (2017). In this pioneering study, Shazeer et al. introduced a sparsely-gated MoE structure where multiple experts contribute via a weighted-sum aggregation. While the original model aimed at efficiency by leveraging sparse gating, MiniDrive’s FE-MoE does not implement sparse gating, thus potentially requiring higher computational resources. Given the similarity in the underlying weighted-sum aggregation concept and the absence of a sparse mechanism, the FE-MoE lacks sufficient differentiation from established MoE architectures and does not clearly demonstrate a unique advantage for autonomous driving. Further clarification on how FE-MoE improves upon traditional MoEs would strengthen the claim of novelty.
3.	Insufficient Multi-Camera Environment Evaluation: While the model mentions multi-image processing, there are no evaluations or specific methodologies provided to demonstrate effectiveness in multi-camera setups, which are essential in autonomous driving for comprehensive scene understanding.
4.	Inadequate Control Experiment Details: In Table 3, the comparison between MiniDrive and a baseline without FE-MoE is presented. However, there is insufficient information on the exact parameter count and FLOPs of the baseline model, raising concerns about the fairness and interpretability of these results.
5.	Minor Errors and Inconsistencies: The paper contains typographical errors (e.g., “ReLu” instead of “ReLU” and inconsistent citation formatting). These, along with unclear baseline setup explanations, detract from the paper’s overall clarity and polish.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Most of my questions are included in the Weakness above. I hope the authors can respond concisely to them.

Suggestions: I hope the paper can clarify more about the dataset and task setup.
soundness: 2
strengths: 1.	The research and idea proposed in this paper are very well-motivated, focusing on the efficiency of the VLM-based approach that is critical to deploying the method in practice. The methods and findings in this paper could inspire relevant research in this direction.
2.	The paper applies the convolutional UniRepLKNet as the visual encoder, instead of ViT-based encoders, to efficiently encode image inputs from multiple directions and introduces an MoE to enhance the representations. The ablation study justifies the effectiveness of this model.
3.	The proposed MiniDrive outperforms other recent works on the DriveLM dataset by a large margin despite having significantly fewer parameters and a simpler training process, which demonstrates the potential of this method.
summary: This paper introduces a lightweight and efficient vision-language model for addressing the QA-based autonomous driving task (DriveLM). Specifically, a framework called MiniDrive is proposed, which incorporates a Feature Engineering Mixture of Experts module (FE-MoE) and a Dynamic Instruction Adaptor (DI-Adapter). The FE-MoE applies an efficient visual encoder, UniRepLKNet, built based on large convolution kernels, for efficient encoding of multi-view images, followed by a mixture of experts similar to multi-head convolution to enhance model expressiveness. The DI-Adapter conditions visual features based on the input instruction. The proposed method is tested on DriveLM and CODA-LM datasets to verify the model performance.
weaknesses: 1.	The paper argues that the DI-Adapter is one of its main contributions; although it seems effective, as shown in Table 3, the novelty is very limited. The idea of adapting visual representations conditioned on instruction has been extensively considered in VLM literature, e.g., the classic InstructBLIP.
2.	The proposed method encodes observations from six directions for DriveLM QA. However, it is unclear to me how the model utilizes and benefits from cross-image information.
   - From Section 3.3, each image is modeled independently, and all features are fed to the T5 language model together.
   - From the examples provided in this paper, it seems that all questions (except Figure 7 case 2) clearly specify exactly one view for QA. Hence, it is unclear to me if encoding all views is necessary and how much improvement MiniDrive can gain from this. 
3.	The biggest weakness of this paper to me is the superficial experiments and lack of in-depth analysis of the system and its behavior.
   - The paper argues that the method has significant efficiency advantages and real-time inference, but there is no experiment to compare the speed with previous approaches. E.g., how many frames/questions can the model process per second (FPS)?
   - It is unclear how exactly the proposed method performs in different tasks, e.g., perception, prediction, and planning, as categorized by the DriveLM dataset, and what is the influence of each proposed component in these tasks.
   - The paper and title highlight the use of multi-level 2D features (the UniRepLKNet), but there is no experiment to compare this encoder to encoders applied in previous works.
   - The DriveLM dataset consists of three components for evaluating Motion, Behavior, and P1-3. This paper only studies P1-3, which includes relatively simple one-shot QA instead of Behavior or Motion, where the traffic anticipation and actual rollout of the ego vehicle are needed. I am not convinced that only evaluate on P1-3 can reflect the actual potential of the proposed method in autonomous driving.
   - A relevant question to the above is how to translate the QA responses to actual vehicle control. It is unclear to me how many questions need to be asked and how to integrate those answers before the agent can make a correct control decision in a complex scenario.