Q-VLM: Post-training Quantization for Large Vision-Language Models
Authors: Changyuan Wang
Ziwei Wang
Xiuwei Xu
Yansong Tang
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Post-training quantization
Large vision-language models
Rounding function search
Primary area: deep_learning_architectures

Abstract: In this paper, we propose a post-training quantization framework of large vision-language models (LVLMs) for efficient multi-modal inference. Conventional quantization methods sequentially search the layer-wise rounding functions by minimizing activation discretization errors, which fails to acquire optimal quantization strategy without considering cross-layer dependency. On the contrary, we mine the cross-layer dependency that significantly influences discretization errors of the entire vision-language model, and embed this dependency into optimal quantization strategy searching with low search cost. Specifically, we observe the strong correlation between the activation entropy and the cross-layer dependency concerning output discretization errors. Therefore, we employ the entropy as the proxy to partition blocks optimally, which aims to achieve satisfying trade-offs between discretization errors and the search cost. Moreover, we optimize the visual encoder to disentangle the cross-layer dependency for fine-grained decomposition of search space, so that the search cost is further reduced without harming the quantization accuracy. Experimental results demonstrate that our method compresses the memory by 2.78x and increase generate speed by 1.44x about 13B LLaVA model without performance degradation on diverse multi-modal reasoning tasks.

Q-VLM: Post-training Quantization for Large Vision-Language Models
Changyuan Wang', Ziwei Wang', Xiuwei Xu², Yansong Tang", Jie Zhou', Jiwen Lu²
'Shenzhen International Graduate School, Tsinghua University, China
DDpartmentt of Automation, Tsinghua University, China
School of Electrical and Electronic Engineering, Nanyang Technological University (wangchan22@mal 1s. xxw2 @mai tan..yansongeeeee yansonggsz.z...... ttiinguua..... edu. an; i jzzhou,, luj iwen t iinnaugaa.... edu. enj ziwei www.eiwannnnnue...mmmmmmm edu.
Abstract
In this paper, we propose post-training quantization framework of large vision- language models (LVLMs) for efficient multi-modal inference. Conventional quantization methods sequentially search the layer-wise rounding functions by minimizing activation discretization errors, which fails acquire optimal quan- tization strategy without considering cross-layer dependency. On the contrary, we mine the cross-layer dependency that significantly influences discretization errors of the entire vision-language model, and embed this dependency into optimal observe the
quantization strategy searching with low search cost. Specifically, strong correlation between the activation entropy and the cross-layer dependency concerning output discretization errors. Therefore, we employ the entropy as the proxy to partition blocks optimally, which aims to achieve satisfying trade-offs between discretization errors and the search cost. Moreover, we optimize the visual encoder to disentangle the cross-layer dependency for fine-grained decomposition search space, so that the search cost is further reduced without harming the quan- tization accuracy. Experimental results demonstrate that method compresses the memory by 2.78x and increase generate speed by 1.44x about 13B LLaVA model without performance degradation on diverse multi-modal reasoning tasks]
1 Introduction
have achieved outstanding performance in large number of multi-modal reasoning tasks such as visual question answering 455 28), embodied instruction following [T] and robot navigation [6). which are benefited from numerous network parameters and vast training data. Despite of the high accuracy and generalization ability across different tasks, the extreme computational cost hinders the deployment on resource-limine mobile devices in wide realistic deployment scenarios. Moreover, LVLMSs sequentially generate the response with multiple forward passes, which further increases the computation burden to accomplish the task. Therefore, is highly demanded to reduce the model complexity of LVLMs in practical deployment.
Large vision- laiiunla models (LVLMs) 31 53
To reduce the model complexity, model compression techniques have been presented to accelerate computation and save the storage space including pruning [18 [1], quantization [19 23 low-rank
decomposition (26] (20 and efficient architecture design [38] 17). Among these methods, quantization
replaces the float numbers with quantized ones and substitutes multipyy-accmmulat (MAC) operations with integer arithmetic for significant efficiency enhancement. Due to the intractability of the training data and the unbearable training cost of LVLMs, post-training quantization
13 is leveraged
Code is available at
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to reduce bitwidths of weights and activations, which only searches rounding functions with small calibration set with frozen network parameters. Searching rounding functions that minimize model prediction errors causes extremely high search cost due to the large space, and conventional methods [12 [9] sequentially search the layer-wise rounding functions by minimizing the activation discretization errors. However, ignoring cross-layer dependency of discretization errors fails to acquire the optimal rounding strategy and degrades the performance significantly.
In this paper, we present an accurate post-training quantization framework called Q-VLM to accelerate large -annuge models for efficient multi-modal reasoning. Different from existing methods which sequentially search the layer-wise rounding functions, we mine the cross-layer dependency of output discretization errors across layers, and employ the dependency to efficiently search the optimal rounding functions that minimize the quantization noise of the entire model. More specifically, we observe the significant correlation between the activation entropy and the discretization error dependency with the following layers. We then employ the entropy as the proxy to decompose the large search space from the entire model to smaller blocks containing multiple layers, and rounding functions are searched with the goal of minimizing the block-wise discretization errors. Therefore, the quantized model remains competitive performance with original full-precision counterparts with trivial additional search cost. Moreover, we optimize the visual encoder to disentangle the cross-layer dependency for fine- fine-graing grained search space decomposition, so that precise rounding functions can be acquired with further reduced search cost. Our Q- VLM can still generate plausible response in multi-modal reasoning with 4-bit quantization because of the precise rounding functions, and compresses the memory by 2.78x and increase the generate speed by 1.44x about 13B LLaVA model. We evaluate our method with the L.LaVA and the MoF-ILLaVA models in different bitwidth settings, and the results in various visual question answering datasets demonstrate that our Q-VLM outperforms the state-of-the-ar post-training methods significantly with negligible search overhead.
2 Related Work
2.1 Large Vision-language Model
Large vision-language models (LVLMs) have achieved remarkable performance because of the fast adaptation on different downstream tasks with high generalization ability, which benefits from large- scale image-text pairs [39 [1] and strong generalization capabilities of prrettrained large language models (LLMs)
44] 240 The cnstucctionfoolooinn ability and muuliimodde representations extracted
by LVLMS are general across tasks, which are usually applied in a wide variety of multi-modal reasoning tasks such as visual question answering [45] 28). embodied instruction following (I] and robot navigation
22 16]. Early attempts introduced rich commonsense in L.MSs to vision- language representation learning, which effectively exploits LLMs by treating visual inputs as conditional information. In particular, BLIP [25 leveraged data filtering techniques to enhance performance in tasks such visual question answering (VQA) and image captioning. While these models exhibited extraordinary vision-language reasoning capabilities, their zero-shot abilities were limited due to the absence of explicit instruction during training. Recent studies including LLaVA (31] and InstructBLIP [30) aim to enhance LVLMs' zero-shot capabilities by aligning them more closely with human preferences. They finetuned LVLMs with visual instruction samples where the models were required to complete the human instruction according to the visual information. Despite the notable performance gains from the large model sizes, the computational complexity and the storage cost prohibit LVLMs from being deployed in pesource-limites devices for realistic deployment. Lightweight LVLMs such as TinyGPT-V (49 and TinyLLaVA (53) endeavors explore the extensive domain of large multimodal models focusing leveraging small-scale models and achieve efficient LVLMs architecture designs. MoE-LLaVA (28) constructs spare MoE-based model, which identifies sparse pathway by simultaneously handling image and text features to achieve comparable performance with fewer activated parameters. However, the model inference cost still exceeds the resource budget of mobile devices or robots because of the low compression ratio.
2.2 Post-training Quantization
Network quantization substitutes full-precision tensors with low-precision values and replaces multipty-ccummulte operations with integer arithmetics, which significantly reduces the storage and computational cost of neural networks. Traditional quantization-awwe aware training (QAT) meth-
2

---

Quantize LLaVA VA Model
Mining Cross-Layer Dependency
Small Entropy
Scarch
Search
Layer k+1
Ql:Can
monilis cells make their own food
Layger k
Layer kt
Layye k-
Crow- Jays-lyyrrr Dependency
Jointly Search
Q2.Whsch occan highlighred??
k
VQA Datasets
Large Entropy
Visual Encoder
Lnyer-wise Search
ag ox
ag
Entropy
ax
(x(2-2)
Auto-Regressive Loss
Language Model
Layers Importance
Weights
Discretization Loss
Layer
Language Response
Optimizing Visual Encoder
Figure 1:
The overall pipeline of our method. We employ entropy the proxy to represent cross-layer dependency for efficient block assignment, which decomposes the large search space from the entire model to blocks containing multiple layers. Moreover, the visual encoder is further optimized for fine-grained search space decomposition.
need to finetune network weights with the full training set for rounding, which less practical since the data and resources for training may not be accessible for most users. Recently, post-trane pest-training quantization (PTQ) [T]
ods 6 32
has aroused extensive interest, which lever- search for the optimal threshold in rounding functions with significantly
ages small calibration reduced data demand and optimization cost. Choukroun et al. (7] minimized the l2 distance between quantized and full-precision tensors to mitigate evident task performance degradation, while Zhao al. [52] duplicated channels with outliers and halved their values to reduce clipping loss without amplifying rounding errors. Liu al. vision transformers to mitigate information loss during post-training quantization and explored a mixed-precision quantization strategy based on the nuclear norm of attention maps and features. Zero-shot PTQ further extends the boundaries for efficiently quantizing neural networks without real image data. Cai al. [5] optimized pixel values of generated images to align sample batch statisties with those of batch normalization (BN) layers in full-precision networks. Li et al. [27] extended the PTQ framework transformer architectures by diversifying self-attention across different patches using patch similarity metrics. Meanwhile, deploying PTQ to large language models (LLMs) dynam- ically search the optimal rounding function for each input sample instead of applying learnable one, because the activation distributes very significantly across different samples in large models. LLM.in(8() [ SmoothQuant [46] and ZeroQuant (48) handled activation outliers to achieve accurate quantization function learning by eliminating the extreme values with equivalent transformation. yet encountered challenges in effectively scaling to extremely large models due to the unbearable computational costs. Furthermore, GPTQ [12). AWQ (29), and QLoRA [9] deployed low-precision quantization on weight quantization to further reduce the computational complexity. However, these methods employ layer-wise searching strategy search the rounding functions sequentially, which deviates from the optimal ones due to the lack of cross-bles block dependency.
{ preserved relative ranking orders of self-attention in
Approach
In this section, first introduce the preliminaries of post-training quantization for LVLMs and then detail the cross-layer dependency mining for LVLM quantization. Finally, we demonstrate the visual encoder optimization to minimize the quantization errors with negligible search cost overhead.
3.1 Post-training Quantization for LVLMs
Network quantization decreases the bitwidth of weights and activations to save computation memory and accelerate inference speed. Conventional guantization-awee aware training (QAT) optimize all parame-

---

ters in original full-preed precision networks for optimal quantization, which is unpractical because of the unacceptable training cost and inaccessibility of the large training datasets. Post-training quantization (PTQ) leverages small calibration set X to search the optimal threshold in minor rounding functions with frozen network parameters, which significantly reduces data requirements and optimization costs. Specifically, the optimal solution for quantization function learning is acquired by minimizing the distribution discrepancy between quantized outputs and full-precision ones of the entire model. The optimization objective J can be formulated as follows:
min (Qx) J x
x(m) W X(m) 2
(I) where ( and X mean the quantized weights and activations for the kns layer, and w and x,*) represent their full-precision counterparts. Qk means the rounding function for the by layer and is the total number of the layers in the LVLM. Directly searching the optimal rounding function NP-hard because the search space increases exponentially with the layer number. Therefore, conventional PTQ methods sequentially search the rounding functions by minimizing the quantization errors for each layer the greedy way:
s.t.
x(++1) Qu (W (())
min Qx
J = W
|
X
( (x)
(2)
where the layer index gradually increases search the rounding function from bottom top layers. However, the greedy search ignores the cross-layer dependency of discretization errors, which leads accumulated discretization errors of model output even for the rounding function with small errors in the bottom layers.
3.2 Mining Cross-layer Dependency for LVLM Quantization
Directly search the solution to (I) causes unacceptable search cost, while sequentially search the layer-wise quantization functions results in suboptimal solution. Therefore, we partition the entire model into different blocks consisting of multiple layers. Searching the optimal rounding function by considering the output quantization errors for each block achieves better trade-off between the search cost and the quantization accuracy, which is formulated as follows:
J (44) (L.) ((( x(4))
min (Q.)
s.t. X (k+1) = (W
(k) (k))
(3)
where B, represents the in block in our partition and L is the index of the last layer in block B, Our goal is to obtain the optimal block partition for rounding function search, where we expect the dependency for layers in each block to be strong. Therefore, we can search the rounding functions for layers in each block by minimiz-
Discretization Error Difference
0.1175
R²-0.9718
0.1150
0.1125
ing the discretization errors for the block output, and the output enrors of the entire model are still minimized.
0.1100
0.1075
Explicitly evaluating the cross-layer dependency requires multiple forward pass of LVL.Ms for given input, where the correlation for discretization errors calculated from data statistics. To avoid extremely high cost, we have to explore an efficient proxy to approximate the cross-layer dependency. We leverage Information Entropy of the activation judge the sensitive layers with homogeneous distribution(43] [3]] For sensitive layers, the noise in the former layer caused by the deviation from the global optimal value usually leads to higher deviation for the current layer, so that jointly search these layers decrease the block output quantization errors. result, discretization error difference (DED) between layer-wise search and joint search is obvious
0.1050
1.4
1.6
Entropy
Figure 2: The correlation between dis- cretization error difference (DED) and the activation entropy 15th layer.

---

for sensitive layers with high entropy. In order to reduce the deviation from the optimal rounding points, we should search the quantization function by joint consideration of the current layer and the former one with strong dependency.
Meanwhile, we also empirically verified assumption shown in Figure Figure 2 is produced with the activations in the 15th layer of the LLaVA architectures on SQA dataset. The horizontal axis demonstrates the entropy of the activations, and the vertical axis depicts the discretization error difference (DED) between layer-wise search and joint search. Different markers mean different input multimodal samples, where DED and the activation entropy are strongly correlated. Therefore, the cross-layer dependency D(k, k + 1) between the current layer k and the following layer & + can be defined as follows:
D(k, 1) Ep (k) (k+1) 4as ylogp( T9 (Fg
(k+1) (k)
(4)
where x* represents the set of possible values about the quantized activation x and x*+1 +1 in the keΛ and k + les layer, and p means the probability of the variable. Qs is searched with different quantization levels for rounding function optimization. We leverage uniform quantization round the full-precd pulecipeeiiion tensors due to the high compatibility with real hardware. In realistic implementation, each element is quantized to the nearest rounding point deterministically. In order acquire the entropy of the quantized tensor, we approximate the deterministic quantization process with the following distribution:
exp( exp(-(xx))) (1
9m /2/2)
p(a 9.85
9,15 - 9m
(5)
ΣM exp( - (I 4m )2/A)
where A is the interval between two consecutive rounding points, and å represents the pulse dis- tribution. Om means the meh rounding point in the quantization out M quantization levels. The cross-layer dependency of two consecutive layers can be depicted by the discretization error differ- ence (DED) between the rounding function searched sequentially and jointly, and larger difference indicates the former layer has significant influence on the discretization errors of the following one. demonstrates the positively correlation between DED and the activation entropy across different input samples with high correlation coefficients. Higher conditional entropy indicates that the activation distribution is homogenized with accumulated quantization errors, which represents larger cross-layer dependency for obvious influence with the following layers. To evaluate cross-layer dependency D(k,, of non-consecutive layer h, and layer ke, we consider the summation of entropy in all intermediate layers between them:
Figure 2
D(k,
Σ 15
p(a)k) (K+ (k+ log( (8+1)) (1)) aigs 4.is 9ids
(6)
Since searching rounding functions within blocks with large number of layers, we constrain the maximum layer numbers for each block. We partition blocks in the LVLM based the acquired cross-layer dependency:
B. = B QulD(kr,ke) (Ko kr)
k=
If the average cross-layer dependency between two layers is larger than the threshold hgs in- termediate layers are assigned into single block for joint rounding function search because the discretization errors of the block output are sensitive to former layers within a block. Finally, we search the optimal rounding function by minimizing the output discretization errors of each block, where we select the optimal percentile p 6 of the full-precision tensor distribution as the bounds for the uniform quantization.
3.3 Optimizing Visual Encoders for LVLM Quantization
L.V.Ms leverage a visual encoder to extract informative representations for image input, and align visual embedding and text embedding with projection layer. As the visual encoder significantly
5

---

modifies the distribution of the activations in LVLMs, the rounding function in visual encoder can be optimized minimize the activation entropy to enhance the searching efficiency. As result, there are fewer layers in each block for rounding function search and the search cost remains low.
Simultaneously minimizing the activation entropy and weakening the eross-layer dependency for all layers causes optimization difficulties, because the large number of layers in LLaMA usually provide conflicted supervision for the visual encoder. Since different layers usually have various influence on the quantization errors of the output from the entire model, we assign different importance weights for the entropy minimization objective across layers which are acquired from the Jacobian:
eg
Lent
Evlogh
(1+1) asgi " M (k+1) 1 g995))
(8)
where ((a) represents the quantization errors of the final layer in the model. The Jacobian indicates the influence of the current layer to the quantization emrors of the final output [34]. Larger Jacobian magnitudes represent the higher influence the overall discretization emrrr,, and assigning larger weights to those layers can reduce the cross-layer dependency with fast model convergence. Mean- while, our objective also includes the minimization of discretization errors for both the output of the visual encoder and the LVLM, which can enhance the quantization accuracy for visual representation learning and multi-model reasoning:
Lerr = |X X;II + = -
where Xÿ and X3' respectively represent the quantized and full full-precision output of the visual encoder, and 1) is a hyperparameter to balance the importance between the discretization errors of the visual encoder and the LVLM. Finally, the overall objective for visual encoder optimization can be written as follows with the hyperparameters a and Ag:
L = Lreg + Le + AaLerr
(10)
where Treg means the auto-regressive loss adopted in training original LVLM to minimize the discrepancy of predicted and target tokens. By optimizing the the visual encoder, we can search the rounding function in more fine-grained blocks with fewer layers to reduce the search cost, while the quantization accuracy still remains high due the weak cross-layer dependency.
Experiments
In this section, conduct extensive experiments for LLaVA and MoE-LLaVA benchmarks ScienceQA multi-modal question answering dataset to evaluate the effectiveness our methods. We first introduce the implementation details of our method. We then conduct ablation studies to evaluate the effectiveness of cross-layer dependency mining and visual encoder optimization. Finally, compare Q-VLM with the state-of-the-art post-training quantization methods to show its superiority.
4.1 Implementation Details
We utilize the large vision-language frameworks for post-training quantization including LLaVA 31 and MoE-LLaVA 28 with their pre-trained weights for multi-modal question answering tasks. We set the bitwidth of quantized weight and activation 6 and 4 to evaluate our method in different quality- efficiency trade-offs uniform quantization scheme where the interval between adjacent rounding points was equal. We followed the initialization of the quantization function parameters in QLoRA 9 for the bascline methods and our Q-VLM, where we minimized the Ip distance [37] 29 between the full-precision and quantized activations to optimize the value range for clipping. We set the maximum layer depth to 3 within a block to achieve satisfying trade-offs between the discretization errors and the search cost. In the LVLM quantization exploration, we adjust hyperparameters p of percentile ranging from 1.0 to 0.98 with 0.005 interval for cross-layer dependency mining. Meanwhile, we modified the hyperparameter 1/ to demonstrate the effect of the discretization loss which optimizes the visual encoder in 9 For the parameter learning in LVLMM quantization, we randomly select 64 vision- languag pairs from the datasets for heper-network learning where the batchsize was assigned with 8 for calibration set construction. The quantization function parameters were updated for 10

---

W6A6
W4A4
Method
Memory
Search cost
Accuracy
Memory
Search cost
Accuracy
QLoRA
23.7
88.43
23.5
77.53
+CDM
26.9
88.95
25.9
78.66
9.7G
6.6G
+VEO
24.1
88.72
23.7
78.35
Q-VLM
25.1
89.34
24.6
79.79
Table 1: Effect of different LVLM quantization method we proposed. "CDM" means cross-layer dependency mining and "VEO" stands for visual encoder optimization. We report the result of LLaVA-7B model on ScienceQA dataset.
50
79.5
30.0
PAcc
LLa76-178 Coe
Ond
Acc
Quantzation Error
1.2
Search Cost
40
79
Accuracy
Search Cost
25.0
78.5
Accuracy
30
78.5
22.5
0.6
20.0
78
7B
0.4
10
17.5
0.2
Maximum Depth
pi
Layers
(a)
(b)
(c)
Figure 3: (a)The answering accuracy and searching cost w.r.t. different maximum layer depth within
block. (b) The answering accuracy and searching cost w.r.t. different hyperparameters across various vision-language models. (c) Quantization errors w.r.t. different maximum layer depth across various
layers.
epochs in searching process, and the acquired discretization function was directly employed for multi-modal question answering. The multi-modal answer reasoning dataset is ScienceQA [35], which contains 21k vision-language multiple choice questions. We also contain VizWiz [5] and
VQA-v2 14 datasets.
4.2 Ablation Study
Since previous layer-wise searching methods ignore cross-layer dependency, we employ joint search- ing strategy with dependency mining. In order to investigate the influence of the block-wise searching strategy, we vary the maximum number of layer depths contained in a single block with different trade-off between discretization errors and searching cost. Meanwhile, we adjust the searching space of percentile by modifying the hyperparameter and 17 for cross-layer dependency mining and visual encoder optimizing to select the optimal clipping range in quantization function. Finally, we randomly select 64 vision-language pairs from ScienceQA dataset to finetune quantized models and demonstrate the effectiveness the proposed cross-layer dependency mining and visual encoder optimizing method. All experiments in the ablation study were conducted with ScienceQA dataset and the LLa V.-.VA-v..3 11.3-1111 framework.
Performance w.r.t. the maximum layer depth within block: Integrating multiple layers with cross-layer dependency into single block can exert considerable influence on the clipping and rounding errors throughout the entire model, albeit at the expense of exponentially escalating search costs. Figure 5a illustrates the answering accuracy for our method that astrict different maximum joint layers, where the performance enhancement for layer depth exceeds 3 is slight with significantly increased complexity overhead. To ensure efficient quantization of LVLMs with sizable accuracy increase, assign the maximum layer depth within block to 3 in subsequent experiments.
To verify the effectiveness of different method we proposed in LVLM quantization, we conduct the ablation study on ScienceQA dataset under different bitwidth. Table illustrates memory usage inference. searching cost in calibration, and answering accuracy for our method under different bitwidth for I.LVAAv113--BB model. Observing the second rows, the cross-layer dependency mining(CDDM) module is important for the final performance, because mining the cross-layer dependency and
Performance w.r.t. different method we proposed question answering process:

---

INT4 AWQ
The Southern Ocean reaches from the shores of Antarctica South latitude. The answer
INT4 Q-VLM
Occans huge bodics of salt water. The world has five occans. This is the Indian Ocean. The answer
Which ocean highlighted?
X
INT4 AWQ
Animals get their food by digesting other organisms. But in the 1950s, scientists discovered that animals can make their own food. Fromia monilis cells use chemosynthesis make their food. The answer
INT4 Q-VIM
Today, many scientists classify organisms into six broad kingdoms. The table below shows some traits used to describe each kingdom. Fromia monilis is animal.
Can Fromia monilis cells make their own food? Options: (A) yes (B)
Animal cells cannot make their own food. The answer is B.
Bacteria
Archaea
Protists
Fungi
Animals
Plants
How many cells do they have?
one
one
one many one many
many
many
Do their cells
have a
no
no
yes
yes
yes
yes
nuelcus?
Can their cells make
some species
some species some species
no
no
yes
food?
can
can
can
Figure 4: Visual reasoning examples from LLa 13B model. Q-VLM improves over the AWQ baseline for W4A4 quantization, reducing quantization errors and providing more reasonable answers. We color the text to show the correct or wrong responses.
leveraging block-wise optimization strengthen the cooperation between layers and minimize the overall quantization errors. Visual encoder optimization (VEO) without cross-layer dependency also significantly modifies the distribution of the activations LVLMs for discretization function learning. Q- D-VLM disentangles the cross-layer dependency for fine-grained search space decomposition, so that precise rounding functions can be acquired with further reduced search cost.
Performance w.r.t. hyperparameters and 7): The hyperparameters p control the search space for cross-layer dependency mining and rj balances the importance between the discretization errors the visual encoder and the LVLM. Larger leads to reduced rounding errors by diminishing the influence of outliers, although excessively large values result excessive clipping of significant information in data distribution. Figure 5b depicts the answering accuracy for different hyperparameter settings, where the medium value for both parameters achieves highest performance and achieves trade-off between quantization errors and searching space.
Visualization reasoning examples: We further provide qualitative visual reasoning example of the LLaVA-v 1.B-13B model in Figure from ScienceQA dataset. Q-VLM improves the responses compared to AWQ baseline for W4A4 quantization setting, leading more reasonable and particular answers for vision- lssion-langugge question pairs. In this first example, Q- Q-VLM correctly answers the question about the highlighted ocean, while AWQ appears to have limited comprehension of the image information. For the second example, AWQ under W4A4 lost substantial information to produce sound reasoning about whether Fromia monilis cells make their own food, while Q-VLM answered correctly and even produced table for precise classification about the six broad organisms called kingdom. Q-VLM improves the visual reasoning ability of LVL.Ss by reducing factual errors in the responses.

---

Subject
Context Modality
Bits
Method
Average
NAT
SOC
LAN
TXT
IMG
NO
FP
89.39
96.06
85.64
88.71
87.65
88.50
89.81
AWQ
85.39
92.01
83.27
84.80
83,54
85.99
86.23
LLaVA-7B
W6A6
QLoRA
88.45
94.71
84.45
87.63
86.07
87.87
88,43
Q-VLM
89,43
95.73
84.00
88.71
87.51
87.25
89.34
AWQ
74,33
72,22
74.82
73.41
67.13
77.98
74.02
W4A4
OLoA
77.53
75.48
79.18
76.64
70.70
81.95
77.53
Q-VLM
80.86
75.93
80.73
80.01
72.48
83.90
79.79
FP
90.19
93.14
87.09
89.39
87.06
89.83
90.00
AWQ
88.03
92.60
84,00
86.02
85.18
86.41
87.57
LLaVA-13B
W6A6
QLoRA
88.87
92.89
85.64
87.59
86.56
87.53
88.87
Q-VLM
89.54
93.18
86.50
88.12
87.01
88,85
89.70
AWQ
80.71
70.61
78.49
79,46
70.76
81,82
77.91
W4A4
QLoRA
79.62
71.43
82.45
78.25
68.42
85,30
78.64
Q-VLM
82.55
73.32
83.18
81.03
70.82
86.74
80.78
MoE-LLaVA-1.6 66
FP
64.01
58.57
63,30
62.80
54.78
66.97
62.68
AWQ
59.83
57.78
61.48
58.94
52.95
64.25
59.83
W6A6
QLoRA
62.98
57.78
61.85
61.87
53,99
65.37
61.60
Q-VLM
64.14
58.68
62.85
62.80
55.23
66,69
62.46
AWQ
53.69
49.58
54.27
52.52
47.81
57.65
52.98
W4A4
QLoRA
54.48
49.69
55.55
53.20
47.30
57.58
53.24
Q-VLM
55.06
51.94
56.27
54.57
48.03
58.34
54.72
Table 2: Comparisons with the state-of-the-arts post-training quantization methods for LLaVA-v1.3 and MoE-LLaVA models across bitwidth setting. Results (accuracy) on Science QA dataset. Question classes: NAT natural science, SOC social science, LAN language science, TXT text context, IMG image context, NO = no context.
4.3 Comparison with the State-of-the-art Methods
In this section, we compare our proposed method with the state-of-the-ar post-training quantization frameworks. As far as know,
the first complete multi-modal LVLMs under W4A4 setting, so we conduct a series of baseline methods by combining the conventional state-of-the-ar post-training quantization methods for fair comparison. For weight quantization, experiment setting of AWQ [29] and QLoRA [9] Meanwhile, as the activations in language modal exhibit significant variations in value range
follow the
different channels, reproduce RPTQ (50) with per-channel activation quantization. As the activation distribution appears larger variance across different tokens in vision modal, we utilize per-token quantization following Outier Suppression (44). The answering accuracy of the baseline methods is acquired by implementing the officially released code and pre-trained model.
Table shows the comparison of top-1 accuracy different post-training quantization methods across various LVLMs architectures including V.-Vv1.3-777 LLaVA-v1.3-13B [31] and MoE-LLaVA. 1.6B (28). where bitwidths weights for quantized layers select from 4 and 6. AWQ searched the optimal scale protect the salient weight channels and decreased the quantization
for weight quantization, while weight-only quantization scaling significantly increased outlier in activation and led significant quantization loss. QLoRA with RPTQ employs sequentially search the layer-wise rounding functions by minimizing activation discretization errors. However, ignoring cross-layer dependency of discretization errors fails acquire the optimal rounding strategy and degrades the performance significantly. On the contrary, our Q-VLM mines the cross-layer dependency of output distribution across layers and decomposes the large search space from the entire model blocks containing multiple layers, which minimizes the block-wise discretization errors to avoid suboptimal quantization, and further optimizes the visual encoder to disentangle the cross- cross-layer dependency for fine- aee-grained search space decomposition. As result, our method outperforms QLoRA by 2.26 (79.79 vs. 77.53) for answering accuracy in ScienceQA dataset under 4-bit in LLa VA--A model. The computational cost remains the same for baseline methods and our Q-VLM due to the stored rounding parameters. The advantage of our method becomes more obvious for 4-bit LVLMS because quantization errors and cross-layer dependency are more important for networks with low capacity.

---

FP
W6A6
W4A4
Model
Dataset
AWQ
QLoRA
Q-VLM
AWQ
QLoRA
Q-VLM
SQA
66.79
65.87
66.16
66.67
56.73
56.50
57.70
LLaVA-7B
VizWiz
49.87
48.51
49.23
49.86
44.29
44.73
45.82
VQA v2
78.50
77.51
77.58
78.52
71.89
72.02
72.51
SQA 71.61
71.37
71.52
72.27
68.13
68.04
68.84
LLaVA-13B
VizWiz 53.63
52.35
52.85
53.69
47.55
47.97
49.28
VQA v2
79.94
78.83
79.25
79.65
71.55
72.19
73.02
Table 3: Comparisons with the state-of-the-ark post-training quantization methods for LLaVA-v1.5 models in various VQA datasets across bitwidth setting.
FP
W8A8
W4A4
Method
Time
Memory
Accuracy
Time
Memory
Accuracy
Time
Memory
Accuracy
QLoRA
16.7h
16.5G
89.32
17.0h
10.7G
78.64
AWQ
12.9h
24.0G
90.00
11,2h
17.2G
88.94
8.9h
11.2G
77.91
Q-VLM
11.2h
15.7G
89.82
8.9h
9.6G
80.78
Table 4: Comparisons with the state-of-the-ard post-training quantization methods for LLaVA-v1.3- 13B models about inference time, memory and accuracy in Science QA dataset.
We also evaluate method on other datasets with different architectures verify the generalization ability. Table demonstrates the accuracy of different post-training quantization methods of LVLMs on the VQA dataset. Our method achieves the highest accuracy different datasets, which means that our method can be robustly deployed in diverse downstream tasks. Table depictss the efficiency and accuracy of different methods. Both AWQ and our method can reduce the search time and the inference memory compared with the original full-precision LVL.Ms, while our method can further reduce the memory because of the additional quantization of the CLIP. When deploying the quantized LVLMs on mobile devices, our method is more practical due to the limited memory footprint.
Conclusion
In this paper, we have presented novel post-training quantization framework large vision-language models for efficient multi-modal reasoning. Different from conventional methods which sequentially search the layer-wise rounding functions without considering cooperation between layers, we mine the cross-layer dependency and decompose the large search space from the entire model to blocks containing multiple layers. The proxy of entropy prompts the efficient and optimal block partition for rounding function search with the goal of minimizing the block-wise discretization error. Therefore, the quantized model remains competitive performance with original full- cell-precision counterparts while the search cost is low. Moreover, optimize the visual encoder disentangle the cross-layer dependency for fine-grained search space decomposition, so that precise rounding functions can be acquired with further reduced search cost. Extensive experiments demonstrate that methods highly compress the large vision-lant vision-language models without performance degradation and achieve higher answer reasoning ability than the state-of-the-art post-training quantization methods across large vision-language models with various architectures even under W4A4.
One limitation of work is that applying our framework to the setting of extremely low bitwidths degrade the performance very significantly, which is also a common issue in post- training quantization of foundation models. As foundation models much larger than the resource limit of mobile devices, we will design efficient post-training quantization method to deploy the LVLMMs embedded equipment such cellphones and wearable devices.
Limitations:
Acknowledgement
This work was supported in part by the National Key Research and Development Program of China under Grant YFF (05101 and part by the Beijing Natural Science Foundation under Grant No. L247009.
10

---

References
[1] Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169 (2022)
Gopalakrishnan, K., Hausman, K., al.: Do
[2] Anderson, P., Wu, Q. Teney, D., Bruce, J., Johnson, M.. Sünderhauf, N., Reid, I., Gould, S., Van Den Hengel, A.: instructions in real environments, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674-366 3683 (2018)
navigation: Interpreting disually-roounde navigation
[3] Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre. S., Sagawa, S., al.: Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.011 30..0108891000 (2023)
[4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877-1901 (2020)
[5] Cai, Y., Yao, Z., Dong, Z., Gholami, A.. Mahoney, M.W., Keutzer, K.: Zeroq: A novel zero shot quantization framework. In: CVPR. pp. 13169-13178 (2020)
[6] Choi, J., Wang. Z., Venkataramani, S., Chuang, P.I.,, Srinivasan, V., Gopalakrishnan, K.: Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.0600 (2018)
[7] Choukroun, Y., Kravchik, E., Yang, F., Kisilev, P:: Low-bit quantization of neural networks for efficient inference. In: ICCVW, pp. 3009-3018 (2019)
[8] Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 (2022)
[9] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetuning of quantized llms. Advances Neural Information Processing Systems 36 (2024)
[10] Ding, X., Liu, X., Zhang, Y., Tu, Z., Li, W., Hu, J., Chen, H., Tang, Y., Xiong, Z., Yin, B., et al.: Chq: Cross-block quantization for large language models. arXiv preprint arXiv:2312.07950 (2023)
[11] Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., Modha, D.S.: Learned step size (2019)
quantization. arXiv preprint
[12] Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D.: Gptq: Accurate post-training quantization for generative pre- pre-trained transformers. arXiv preprint arXiv:2210.17322 (2022)
[13] Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., Yan, J.: Differentiable soft quantiza- tion: Bridging full-precision and low-bit neural networks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4852-4861 (2019)
[14] Goyal, Y., Khot, T., Summers-Star D., Batra, D., Parikh, D.: Making the in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6904 (2017)
[15] Gurari, D., Li, Q., Stangl, A.J., Guo, A., Lin, C., Grauman, K., Luo. J., Bigham, J.P.: Vizwiz grand challenge: Answering visual questions from blind people. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3608-3617 (
[16] Hao, W., Li, C., Li, X.. Carin, L., Gao, J.: Towards learning a generic agent for vision-and- language navigation via pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13137-13146 (2020)
[17] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint MxXiv1170.0488 (2017)

---

[18] Hu, H., Peng, R., Tai, Y.W.. Tang, C.K.: Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint 1.Xi::6677003500 (2016)
[19] Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. NeurlPS 29 (2016)
[20] Idelbayev, Y., Perreina- M.A.: Low-rank compression of neural nets: Learning the rank of each layer. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 8049-8059 (2020)
[21] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q. Sung, Y.H., Li, Z., Duerig, T:: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference machine learning. pp. 4904-4999 1904-4916. PMLR (2021)
[22] Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D.: The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems 33, 261 1-2624 (2020)
[23] Lee, J., Kim, D., Ham, B.: Network quantization with element-wise gradient scaling, In: CVPR. pp. 6448 6457 (2021)
[24] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arX1v:2301.12597 (2023)
[25] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In: International Conference Machine Learning. pp. 12888-12900. PMLR (2022)
[26] Li, Y., Gu, S., Mayer, C., Gool, L.V., Timofte, R.: Group sparsity: The hinge between filter pruning and decomposition for network compression. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp.
(2020)
[27] Li, Z., Ma, L., Chen, M., Xiao, J., Gu, Q.: Patch similarity aware data-free quantization for vision transformers. In: ECCV. pp. 154-170 (2022)
[28] Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., Yuan, L.: Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 (2024)
[29] Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S.: Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0097. (2023)
[30] Liu, H., Li, C., Li, Y., Lee, YJ.:: Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 (2023)
[31] Liu, H., Li, C., Wu, Q. Lee, YJ..: Visual instruction tuning. Advances in neural information processing systems 36 (2024)
[32] Liu, 7., Luo, W., Wu, B., Yang, X., Liu, W., Cheng, K.T.: Bi-real net: Binarizing deep network towards real-network performance. IJCV 128, 202-219 (2020)
[33] Liu. Z., Wang, Y., Han, K., Zhang, W., Ma. S., Gao, W.: Post-training quantization for vision transformer. NeurIPS 34, 28092-28103 (2021)
[34] Lopes, V., Alirezazadeh, S., Alexandre, L.A.: Epe-nas: Efficient performance estimation without training for neural architecture search. In: International conference artificial neural networks. pp. 552-563. Springer (2021)
[35] Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507-2521 (2022)
[36] Nagel, M., Amjad, R.A., Van Baalen, M., Louizos, C., Blankevoort, T.: Up or down? adaptive rounding for post-training quantization. In: ICML. pp. 7197-7206. PMLR (2020)
12

---

[37] Nahshan, Y., Chmiel, B., Baskin, C., Zheltonozhskii, E., Banner, R., Bronstein, A.M., Mendel- son. A.: Loss aware post-training quantization. Machine Learning 110(11-12), 3245-3262 (2021)
[38] Qin, Z., Li, Z., Zhang, Z., Bao, Y., Yu, G., Peng, Y., Sun, J.: Thundernet: Towards real-time generic object detection on mobile devices. In: Proceedings of the IEEE/CVF international conference computer vision. pp.
(2019)
[39] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
[40] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971 (2023)
[41] Wang, C., Wang. Z., Xu, X., Tang. Y., Zhou, J., Lu, Towards accurate post-training quantization for diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16026-16035 (2024)
[42] Wang, Z., Wang, C., Xu, X., Zhou, J., Lu, J.: Quantformer: Learning extremely low-precision vision transformers. IEEE Transactions Pattem Analysis and Machine Intelligence (2022)
[43] Wang, Z., Wu, Z., Lu, J., Zhou, J.: Bidet: An efficient binarized object detector. In: Proceedings of the IEFE/CVF conference on computer vision and pattern recognition. pp. 2049-2058 (2020)
[44] Wei, X., Zhang, Y., Zhang. X., Gong, R., Zhang, S., Zhang, Q., Yu, F,, Liu, X.: Outlier suppression: Pushing the limit of liw-bit transformer language models. Advances in Neural Information Processing Systems 35, 17402-17414 (2022)
[45] Wu, C., Yin, S., Qi. W., Wang, X., Tang, Z., Duan, N.: Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.066) 036677 (2023)
[46] Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S.: Smoothquant: Accurate and efficient post-training quantization for large language models. In: International Conference Machine Learning. pp. 38087-38099. PMLR (2023)
[47] Yao, Z., Wu, X., Li, C., Youn, S., He, Y:: Zeroquant-v2: Exploring post-training quantization in Ilms from comprehensive study to low rank compensation. arXiv preprint arXiv:2303.08302 (2023)
[48] Yao, 7., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., He, Y.: Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems 35, 27168-27188 (2022)
[49] Yuan, Z., Li, Z., Sun, L.: Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.1666) (2023)
[50] Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q. Wu, J., Wu, B.: Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.0100 (2023)
[51] Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., Tian, Q.: Variational convolutional neural network pruning. In: CVPR. pp. 2780-276 2789 (2019)
[52] Zhao, R., Hu, Y., Dotzel, J., De Sa, C., Zhang, Z.: Improving neural network quantization
without retraining using outlier channel splitting. In: ICML. pp. 7543-7552 (2019)
[53] Zhou, B., Hu, Y., Weng, X., Jia, J. Luo, J., Liu, X., Wu, J.. Huang, L.: Tinyllava: A framework of scall-scale large multimodal models. arXiv preprint arXiv:2402. 14229 (2024)
13

---

Difference
R²-0.8123
Og
Discreticttoo Errur Differenee
0.155
12-00.9765
Diveretization Error Differnne
R²-0.9586
0.925
0.125
6.11
0,0775
0.19
0.7790
0.0255
0J675
0.08
0.09
0.10
8.11
612
0.13
1.2
1.8
1.8
13
14
Quantization Heror
Eatropy
Eatropy
(b)
(c)
Figure 5: (a) The correlation between discretization error difference (DED) and the quantization errors in 15th layer. (b) The correlation between DED and the entropy 5th layer and 25th layer.
W6A6
W4A4
Model
Dependency Proxy
Accuracy
Search Cost
Accuracy
Search Cost
Quantization Errors
88.59
43.7
77.97
41.2
I.L.VA-BB
Entropy
88.95
26.9
78.66
25.9
Quantization Errors
88.96
58.6
78.82
54.2
LLaVA-13B
Entropy
89.04
33.1
79.89
31.8
Table 5: Comparisons with different proxy for mining cross-layer dependency for LLaVA-v1..3 models in ScienceQA dataset across bitwidth setting.
A Why leveraging entropy as proxy:
The benefit and motivation for using entropy rather than quantization errors proxy for block-wise searches lie in several key considerations. We analyze that larger entropy indicates more homogeneous data distribution, which a well-established principle information theory. Consequently, DED and activation entropy are strongly correlated with an value of 0.97. However, greater quantization error does not necessarily imply more homogeneous data distribution and does show a positive correlation with DED, having an value of 0.81, which is empirically verified in the figure
Meanwhile, the search cost of quantization errors doubles compared with entropy proxy, as the calculation of quantization errors requires multiple forward passes for both the FP model and the quantized model. The weak correlation and the unbearable search cost render quantization error unsuitable as metric for measuring cross-layer dependency.
Furthermore, conducted experiments comparing the proxy effectiveness of quantization error and entropy across different models under various bitwidths Table(5) Entropy outperformed quantization errors by significant margin (78.66 vs. 77.97), showing strong cross-layer dependency within each block. This allowed us to achieve optimal block partitioning by mining the cross-layer dependency.
B Performance on more baseline methods
We have extended our experiments to additional baseline method ZeroQuant-V2 47 and compared it against our proposed methods in Table 6 ZeroQuant-V leverages per-token quantization with different rounding functions to minimizing activation discretization errors. However, ignoring cross-layer dependency of discretization errors fails to acquire the optimal rounding strategy with severe outliers under low bitwidth and degrades the performance significantly. On the contrary, our Q-VLM mines the cross-layer dependency of output distribution across layers, minimizing the block-wise discretization errors to avoid suboptimal quantization. We further optimize the visual encoder disentangle the cross-layer dependency for fine fine-grained search space decomposition. As
14

---

W8A8
W4A4
Model
Quantization Method
Accuracy
Inference Time
Accuracy
Inference Time
ZeroQuant-V2
89.04
10.7h
78.08
7.3h
LLaVA-7B
Q-VLM
89.58
8.3h
79.79
6.1h
ZeroQuant-V2
89.13
12.6h
78.81
9.7h
LLaVA-13B
Q-VLM
89.81
11.2h
80.78
8.9h
Table 6: Comparisons with different quantization methods for 7B and 13B models across W6A6 and W4.4 bitwidth settings.
8bit
4bit
Dataset
Shots
FP
Q-LoRA
Q-VLM
Q-LoRA
Q-VLM
0
23.79
21.24
21.47
17.62
18.69
Vizwiz
4
27.05
25.83
26.59
24.17
24.55
32
39.76
36.38
37.60
31.64
35.52
0
50.23
47.75
49.12
43.86
44.22
Hateful Memes
4
50.10
48.62
49.55
45.12
45.26
32
50.27
50.02
51.05
45.76
47.84
Table 7: Performance comparison on Vizwiz and Hateful Memes datasets across FP, 8bit, and 4bit quantization methods with different shot settings.
result, our method outperforms ZeroQuant-V2 by 1.71 (79.79 vs. 78.08) in answering accuracy on ScienceQA dataset under 4-bit in LLaVA-7B model. Additionally, our method enhances inference speed, exceeding ZeroQuant-V2 by 1.2h (6. 1h vs. 7.3h) due to utilizing stored rounding parameters instead of dynamic per-token quantization. The additional bascline provides a more comprehensive evaluation framework to highlight the strengths of our approach.
C Performance on other multi-modal architectures
We also explored the multi-modal architecture OpenFlamingo to ensure the robustness and generalizability of our methods 7 We deploy our method on OpenFlamingo 3B model using Vizwiz and Hateful Memes [22] datasets, selecting bitwidths of and for quantized layers. Q- VLM designed in LLaVk-like architectures can be effectively adapted to cross-attention based VLMs due to the consistent core mechanism cross-attention and the robust multimodal alignment capabilities treitrained on large-scale dision-language pairs. Since OpenFlamingo is tross-attention based VLM, exploiting cross-layer dependency is particularly suitable. Our method outperforms Q-LoRA by 1.22 (37.60 36.38) under 8- bit in OpenFlamingo-3B model. The advantage of our method becomes more obvious for 4-bit 3B LVLMs because quantization errors and cross-layer dependency play
significant role in networks with low capacity. These results underscore the robustness and generalizability of our approach across different tasks, model architectures and datasets, demonstrating its effectiveness in diverse scenarios.
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We claimed it in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We claimed the limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We have the theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The experiments can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[No]
Justification:
We will release code as soon as the article is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.. cc/
pub ic quides Codeeubmmes ss ssionns aisionoolico icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/
wwwblic//uideee / //odiddee/Cdeeeeeeee Cooeeeooeeeeeeeeee SudeSuumiiee Swwmisssiiii ssssnoooiccyyyycco for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We claimed the experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not get the experiment statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
We the RTX 3090 which do provide in the article.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not have broader impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have claimed the licenses for existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssw Ssperswithhooos com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer:
[NA]
Justification:
We do not have new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have this experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have this.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "hH10TEewEr", "reviewer": "Reviewer_TNuV", "summary": "The paper introduces Q-VLM, a PTQ framework for VLMs that leverages entropy as a proxy to manage cross-layer dependencies for both language model and visual encoder. Experimental results on ScienceQA, VizWiz, VQAv2 datasets and LLaVA variant architectures validate the efficacy of the proposed method.", "questions": "Suggest to clarify the concerns in the weakness section", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Q-VLM achieves good performance with W6A6 on ScienceQA, VizWiz, VQAv2 compared to the FP counterparts. \n* Ablation studies in Table 1 and Section 4.2 are helpful for understanding the importance of each component of Q-VLM, such as cross-layer dependency mining and visual encoder optimization.\n* Experiments across LLaVa model sizes and bit-width demonstrates the robustness of the proposed method.", "weaknesses": "There are several concerns:\n* The benefit and motivation for using entropy as a proxy are unclear. Why not directly use the quantization error as a metric for performing block-wise searches to mine cross-layer dependency? Are there any accuracy or inference efficiency gains by using entropy as a proxy?\n* The idea of mining cross-layer dependency is not new. Existing works, such as BRECQ, already address block-wise quantization.\n* Important baseline methods are missing in the comparison results. For a more solid comparison, include other state-of-the-art PTQ methods for the vision branch and more recent works such as SmoothQuant and ZeroQuant variants for the language branch. Comparing with the previous best method on the language model and another prior method on the vision model would provide a clearer picture of the overall improvement.\n* The proposed method appears to be a general approach for both language and vision models. It would be interesting to demonstrate its individual effect on either the language or vision model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate the opportunity to clarify the points you raised regarding our methodology and its contributions.\n\n**Q1: The benefit and motivation for using entropy as a proxy are unclear.**\n\n**[Reply]** We are sorry for the confusion. For block-wise quantization, we partition the entire model into different blocks and search the optimal rounding function by considering the output quantization errors for each block achieves better trade-off between the search cost and the quantization accuracy. Our goal is to obtain the **optimal block partition** for rounding function search, where we leverage **Information Entropy** to judge the sensitive layers with homogeneous distribution. For sensitive layers, the noise in the former layer caused by the deviation from the global optimal value usually leads to higher deviation for the current layer, so that jointly search these layers decrease the block output quantization errors. As a result, discretization error difference (DED) between layer-wise search and joint search is obvious for sensitive layers with high entropy. The benefit and motivation for using entropy rather than quantization errors as a proxy for block-wise searches lie in several key considerations. \n\nFirst, we analyze that larger entropy indicates more homogeneous data distribution, which is a well-established principle in information theory. Consequently, DED and activation entropy are strongly correlated with an $R^2$ value of **0.97**. However, greater quantization error does not necessarily imply more homogeneous data distribution and does not show a positive correlation with DED, having an $R^2$ value of **0.81**, which is empirically verified in the figure presented in the Author Rebuttal.\n\nMeanwhile, the search cost of quantization errors doubles compared with entropy as a proxy, as the calculation of quantization errors requires multiple forward passes for both the FP model and the quantized model. The weak correlation and the unbearable search cost render quantization error unsuitable as a metric for measuring cross-layer dependency.\n\nFurthermore, we conducted experiments comparing the proxy effectiveness of quantization error and entropy across different models under various bitwidths. Entropy outperformed quantization errors by a significant margin **(78.66 vs. 77.97)**, showing a strong cross-layer dependency within each block. This allowed us to achieve optimal block partitioning by mining the cross-layer dependency.\n|Model|6bit Method|6bit Accuracy|6bit Search Cost|4bit Method|4bit Accuracy|4bit Search Cost|\n|---|---|---|---|---|---|---|\n|7B|Errors|88.59|43.7|Errors|77.97|41.2|\n||Entropy|88.95|26.9|Entropy|**78.66**|**25.9**|\n|13B|Errors|88.96|58.6|Errors|78.82|54.2|\n||Entropy|89.04|33.1|Entropy|**79.89**|**31.8**|\n\n**Q2: The idea of mining cross-layer dependency is not new.**\n\n**[Reply]** Existing works such as BRECQ incorporates Fisher information and jointly optimizes two layers within each residual block, it does not capture interactions across neighboring residual layers. Additionally, they do not provide the optimal configuration of the reconstruction granularity, with their choice of block-wise optimization stemming solely from experimental results. Consequently, these methods leverage fixed block partitions for rounding function search, leading to suboptimal performance in large models. We conduct experiments on LLaVA model with 7B and 13B parameters under 4-bit in ScienceQA dataset. Our cross-layer dependency mining method outperforms BRECQ by **0.56 (78.66 vs. 78.10) and 1.04 (79.89 vs. 78.85)** in the 7B and the 13B models respectively. Conventional quantization methods[1-3] for LLMs indicate that as model scale increases, systematic outliers with large magnitude emerge in activations, leading to significant quantization errors and accuracy degradation. Therefore, BRECQ's fine-grained block partitioning cannot effectively address outliers in LVLMs under low-bitwidth. On the country, our cross-layer dependency mining method which leverage Information Entropy achieve optimal block partition and sufficiently utilizes the cross-layer dependency.\n\n**Q3: Performance on more baseline methods.**\n\n**[Reply]** In response to **Overall Author Rebuttal Q1** as well as Q2 from your previous responds, we have conducted experiments with additional state-of-the-art PTQ methods including BRECQ and QLoRA for the vision branch and more recent methods such as ZeroQuant-V2 for the language branch. We further performed experiments combining ZeroQuant-V2 and BRECQ as baseline methods. Our Q-VLM method outperforms the baseline method by 1.71 (79.79 vs. 78.08) for answering accuracy in ScienceQA dataset under 4-bit in LLaVA-7B model. Both ZeroQuant-V2 and BRECQ are insufficient handling severe outliers in LVLM under low bitwidth which significantly degrades performance. In contrast, our Q-VLM method effectively mines cross-layer dependency by leveraging Information Entropy and utilizing Jacobian matrices to assign various importance weights to different layers for vision-language synergistical optimization.\n\n**Q4: Why the proposed methods are particularly suitable for LVLM.**\n\n**[Reply]** According to the **Overall Author Rebuttal Q2** and the ablation study in Table 1, we conducted experiments to demonstrate individual effect of cross-layer dependency mining method on both LLaMA language model and the vision model. Solely deploy cross-layer dependency mining method indicate limited performance improvement. As the visual encoder optimization method with vision-language synergistic optimization, we conclude our Q-VLM are particularly suitable for LVLM.\n\n[1] Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.\n\n[2] Smoothquant: Accurate and efficient post-training quantization for large language models.\n\n[3] Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BtYMgsIBR", "reviewer": "Reviewer_ff1U", "summary": "The authors propose a new post-training quantization method for LVLMs. The authors separate several layers in a LVLM into blocks and search for the optimal quantization bitwidth for each block individually. The authors also introduced a new objective function for quantizing the vision encoder. The extensive experimental results show the proposed methods outperform the SOTA post-training quantization methods on LLaVA models.", "questions": "See weaknesses", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The authors provide a relatively comprehensive related work discussion in the paper. \n- The paper is well-motivated. Post-training quantization on LLM / VLM is critical for deploying these models due to their extensive computation consumption during training. \n- The authors provide a large amount of evaluations and ablations in the experimental section to help other researchers better understand their model performance.", "weaknesses": "- The overall methods part is difficult to follow. It is not clear if the authors adopt different quantization strategies in visual encoders and language model and the projects in LLaVA. Also, it is not clear to me why the authors design those specific methods for LVLM, i.e., why the proposed methods are particularly suitable for LVLM compared to LLM, CNN, etc. It would be better that the authors can discuss more details about their insight or motivation for those designs.\n- The main advantages of post-training quantization compared to quantization-aware training is that it does not need the entire training data and requires less computation. Therefore, it would be better that the authors can provide some ablations in terms of the amount of training data used during the post-training quantization and the number of computation needed during quantization. Otherwise, it is difficult to position this paper w.r.t. other relevant work.\n- The authors claimed the prior quantization methods are suboptimal. However, adopting entropy as proxy and separating layers into blocks are also sub-optimal solutions. Then, there exists another question to be answered, why the sub-optimal solutions proposed in the paper are better than the others?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the careful reading and valuable comments. We address the questions and clarify the issues accordingly as described below.\n\n**Q1: Confusion about the quantization strategies.**\n\n**[Reply]** We apologize for the confusion. The detailed construction of the baseline quantization methods is described in Section 4.3 as we leverage RPTQ and Outlier Suppression for activation quantization in single language and vision modality model. For our methods, the language model employs cross-layer dependency mining method exclusively, while the vision model not only utilizes cross-layer dependency mining but also incorporates additional optimization for vision encoder with vision-language synergistic optimization. Our vision encoder optimization method utilizes Jacobian matrices to assign various importance weights to different layers for vision-language synergistical optimization. We did not quantize the projector, as it contains only 20M parameters which is negligible compared to the 7B parameters of the entire model. Additionally, remaining the projector to FP would not result in significant increase in inference speed by 0.06h (6.13h vs. 6.07h) compared with 4-bit projector.  \n\n**Q2: Explain why the proposed methods particularly suitable for LVLM.**\n\n**[Reply]** According to the **Overall Author Rebuttal Q2**, we point out our visual encoder optimization method is designed specifically for LVLMs due to the multi-modal data distribution compared with LLM. CNNs primarily rely on the layer-by-layer transmission of local features within the receptive field and the pooling operations that gradually lose detailed information. This makes it challenging for them to achieve global information capture and direct associations like the self-attention mechanism of LVLMs.\n\n**Q3: Ablation study about the number of training data.**\n\n**[Reply]** Thanks for your advice. We have conduct ablation studies to evaluate the effectiveness of varying amounts of calibration images used during the post-training quantization on both accuracy and calibration cost. Leverage more calibration images can reduce the rounding function overfitting, while may result in extreme distribution data selection and increased calibration time overhead with large-scale learnable parameters. The results are presented as follows:\n|Model|Calibration Images|Accuracy|Calibration Cost|\n|---|---|---|---|\n|7B|16|78.85|2.1|\n||64|**79.79**|**8.8**|\n||256|79.61|34.9|\n|13B|16|79.06|3.4|\n||64|**80.78**|**14.2**|\n||256|80.96|56.4|\n\nObserving the accuracy and calibration cost for different amounts of calibration images settings, we analyze that medium calibration images achieve the optimal performance. Limited amounts of calibration images face the challenges of overfitting, while high calibration images select extreme distribution data with marginal improvements in accuracy and substantially increased calibration cost due to large-scale learnable parameters. We suggest that utilizing 64 calibration images achieve optimal trade-off between quantization accuracy and rounding function generalizability.\n\n**Q4: Confusion about the mining cross-layer dependence.**\n\n**[Reply]** We detailly describe the significance of leveraging entropy as a proxy which identifies sensitive layers and facilitates optimal layer allocation through efficient cross-layer dependency correlation in **Rebuttal for TnuV Q2**. In data-limited PTQ quantization, it is impractical to minimize discrepancies by reconstructing the network final output to achieve the optimal solution for large-scale second-order errors. Directly searching the optimal rounding function is NP-hard because the search space increases exponentially with the layer number. Compared with conventional layer-wise quantization which minimizes quantization errors for each layer in the greedy way to solve that NP-hard problem, our method considers cross-layer dependency and employs entropy as a proxy for block-wise quantization to achieves a satisfying trade-off between discretization errors and search cost. Our cross-layer dependency mining method outperforms BRECQ by **0.56 (78.66 vs. 78.10) and 1.04 (79.89 vs. 78.85)** in the 7B and the 13B models respectively. Consequently, our method achieves optimal block partitioning and effectively utilizes cross-layer dependency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zw0xjlunvO", "reviewer": "Reviewer_EGvn", "summary": "This paper proposes Q-VLM, a post-training quantization framework for Large Vision-Language Models (LVLMs). It aims to reduce the model complexity of LVLMs for practical deployment by replacing float numbers with quantized ones and substituting multiply-accumulate operations with integer arithmetic. The key innovation lies in mining cross-layer dependency to efficiently search for optimal rounding functions that minimize quantization noise across the entire model. The authors also optimize the visual encoder to further reduce search costs and maintain quantization accuracy. Experimental results on LLaVA and MoE-LLaVA models demonstrate significant memory compression and speed increases without severe performance degradation.", "questions": "1. Can the authors provide more experiments on other tasks/datasets to demonstrate the generalizability of their method beyond ScienceQA?", "limitations": "Not found.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Novel Approach to Quantization**: The paper introduces a novel approach to post-training quantization that considers cross-layer dependencies, which is a significant departure from traditional layer-wise methods. This approach has the potential to improve the efficiency and accuracy of quantized LVLMs.\n2.  **Clear Presentation**: The paper is well-organized and clearly written, making it easy to follow the authors' thought process and understand their contributions.", "weaknesses": "1. **Limited Evaluation on Diverse Datasets**: The majority of experiments are conducted on the ScienceQA dataset, which may not fully represent the diverse range of tasks and challenges that LVLMs encounter in real-world applications. Evaluating the method on a wider range of datasets would provide a more comprehensive assessment of its effectiveness and generalizability.\n2. **Marginal Performance Improvement**: The observed improvements in some cases are relatively small, potentially due to the specific characteristics of the ScienceQA task. It would be beneficial to investigate whether the method's impact is consistent across different tasks and datasets, or if its effectiveness is limited to specific scenarios.\n\nSome typo:\n\nIn the abstract and introduction, the phrase \"compresses the memory by 2.78x and increase the generate speed by 1.44x **about** 13B LLaVA model\" may be unclear. It could be revised to \"compresses the memory by 2.78x and increases the generation speed by 1.44x **for** the 13B LLaVA model\" for improved clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows:\n\n**Q1: Evaluation on Diverse Datasets.**\n\n**[Reply]** LVLMs encounter diverse range of tasks and challenges in real-world applications. We evaluate our proposed method on various datasets with different architectures to verify the effectiveness and generalizability. The experiments are presented in Table 3 and as follows:\n|Model|Dataset|FP|6bit AWQ|6bit Q-LoRA|6bit Q-VLM|4bit AWQ|4bit Q-LoRA|4bit Q-VLM|\n|---|---|---|---|---|---|---|---|---|\n|LLaVA-7B|MM-Vet [1]|31.40|30.79|31.03|**31.59**|28.17|28.42|**29.28**|\n||ScienceQA|66.79|65.87|66.16|**66.67**|56.73|56.50|**57.70**|\n||VizWiz|49.87|48.51|49.23|**49.86**|44.29|44.73|**45.82**|\n||VQA v2|78.50|77.51|77.58|**78.52**|71.89|72.02|**72.51**|\n|LLaVA-13B|MM-Vet [1]|36.07|34.78|34.67|**35.83**|30.16|30.71|**31.64**|\n||ScienceQA|71.61|71.37|71.52|**72.27**|68.13|68.04|**68.84**|\n||VizWiz|53.63|52.35|52.85|**53.69**|47.55|47.97|**49.28**|\n||VQA v2|79.94|78.83|79.25|**79.65**|71.55|72.19|**73.02**|\n\nScienceQA challenges LVLMs with domain-specific question answering requiring deep understanding of scientific concepts and visual data interpretation. VQA v2 and MM-Vet[1] tasks LVLMs with the need to accurately answer open-ended questions about diverse images, testing their ability to integrate visual and linguistic information. VizWiz leverage LVLMs to interpret and answer questions about everyday images captured by visually impaired users, often dealing with low-quality and diverse real-world content. Our method outperforms Q-LoRA by **1.31 (49.28 vs. 47.97)** under 4-bit in LLaVA-v1.5-13B model on VizWiz dataset, which shows the superiority of mining cross-layer dependency to effectively reduces quantization errors and enhances generalizability for low-quality and diverse real-world content. Q-VLM achieves the highest accuracy among various post-training quantization methods for LVLMs across different VQA datasets, indicates that our method can be robustly deployed in diverse downstream tasks.\n\n**Q2: Evaluation on Different Tasks.**\n\n**[Reply]** Our Q-VLM has achieved the highest accuracy across different visual question answering tasks including ScienceQA, VizWiz, VQA v2 and MM-Vet, where our Q-VLM outperforms Q-LoRA by 0.93 (31.64 vs. 30.71) under 4-bit in LLaVA-v1.5-13B model on MM-Vet dataset. \n\nAdditionally, we conduct experiments on other multi-modal architecture OpenFlamingo with Hateful Memes dataset for classification task to further demonstrate effectiveness and generalizability of our proposed method in the **Overall Author Rebuttal Q1**. Our method outperforms Q-LoRA by 2.08 (47.84 vs. 45.76) and 1.03 (51.05 vs. 50.02) in OpenFlamingo-3B model under 4-bit and 6-bit respectively. The Hateful Memes dataset is a collection of multimodal data containing images paired with text captions, specifically designed for research in detecting hate speech in memes. Our Q-VLM excels in the classification task on the Hateful Memes dataset by effectively combining vision-language information to accurately identify hate speech in multimodal memes.\n\n**Q3: Correcting Some Typos.**\n\n**[Reply]** Thank you for your valuable comments. We have thoroughly revised the paper, carefully improving the writing, and correcting misleading wording and grammatical errors. We hope these revisions make the writing more professional and easier to understand.\n\n[1] Yu, Weihao, et al. \"Mm-vet: Evaluating large multimodal models for integrated capabilities.\" arXiv preprint arXiv:2308.02490 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GdyuKHbioI", "reviewer": "Reviewer_rcqj", "summary": "The authors present a novel post-training quantization framework for large multimodal language models to enhance inference speed. This method accounts for cross-layer dependencies that significantly impact overall model discretization errors and leverages activation entropy to effectively balance these errors with search costs. Extensive experimental results demonstrate that this framework can reduce memory usage and improve generation speed by 1.44x on the 13B LLaVA model, while maintaining comparable performance across diverse multimodal downstream tasks.", "questions": "1. Does this quantization method only work for LLaVA-like architectures? How does it apply to cross-attention based VLMs such as Flamingo [1] and Otter [2]?\n\n[1] Alayrac, Jean-Baptiste, et al. \"Flamingo: a visual language model for few-shot learning.\" Advances in neural information processing systems 35 (2022): 23716-23736.\n\n[2] Li, Bo, et al. \"Mimic-it: Multi-modal in-context instruction tuning.\" arXiv preprint arXiv:2306.05425 (2023).", "limitations": "The authors have addressed the limitation in the manuscript.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper considers cross-layer dependencies during the quantization process for the first time, which is novel.\n\n2. The proposed quantization method is simple and effective. It significantly reduces memory usage and improves inference speed, which is crucial for MLLM deployment. \n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. Is there any quantization for the adapter between the vision encoder and the LLM? With various types of adapters such as MLP for LLaVA, Q-former for BLIP-2 [1], and Visual Abstractor for mPLUG-Owl [2], does the proposed method apply to models with different adapters?\n\n2. Instead of comparing with outdated methods such as Q-LoRA and AWQ, it would be beneficial to compare with more advanced techniques like ZeroQuant-FP [3] and include a more in-depth discussion.\n\n[1] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" International conference on machine learning. PMLR, 2023.\n\n[2] Ye, Qinghao, et al. \"mplug-owl: Modularization empowers large language models with multimodality.\" arXiv preprint arXiv:2304.14178 (2023).\n\n[3] Wu, Xiaoxia, Zhewei Yao, and Yuxiong He. \"Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats.\" arXiv preprint arXiv:2307.09782 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your suggestion and agree that comparisons with more advanced techniques and cross-attention based VLM architectures would provide a deeper insight into the effectiveness of our method. Below are our detailed responses.\n\n**Q1: Ablation study about the projectors.**\n\n**[Reply]** For quantization about the projector between the vision encoder and the LLM, we did not quantize it as it contains only 20M parameters which is negligible compared to the 7B parameters of the entire model. Additionally, remaining the projector to FP would not result in significant increase in inference speed by **0.06h (6.13h vs. 6.07h)** compared with 4-bit projector. We conduct experiments on OpenFlamingo architecture with encoder-decoder structure which emphasize the leverage of the model inherent architecture for multimodal feature alignment and our Q-VLM still achieves outstanding performance compared with baseline methods. The results are presented as follows based on Tables in Overall Author Rebuttal Q1:\n|Model|Dataset|FP|8bit ZeroQuant-V2|8bit Q-VLM|4bit ZeroQuant-V2|4bit Q-VLM|\n|---|---|---|---|---|---|---|\n|LLaVA-7B|ScienceQA|89.81|89.04|89.58|78.08|**79.79**|\n|LLaVA-13B|ScienceQA|90.00|89.13|89.81|78.81|**80.78**|\n|**Model**|**Dataset**|**FP**|**8bit Q-LoRA**|**8bit Q-VLM**|**4bit Q-LoRA**|**4bit Q-VLM**|\n|OpenFlamingo-3B|Vizwiz|39.76|36.38|37.60|31.64|**35.52**|\n||Hateful Memes|50.27|50.02|51.05|45.76|**47.84**|\n\nWe conclude that our Q-VLM designed in LLaVA-like architectures can be effectively adapted to other transformer-based VLM architectures with different projectors due to the effectively vision-language alignment information preserved in projectors.\n\n**Q2: Performance on more baseline methods.**\n\n**[Reply]** In response to **Overall Author Rebuttal Q1**, we have conducted experiments and include a more in-depth discussion with more advanced techniques ZeroQuant-V2 compared with proposed Q-VLM. ZeroQuant-V2 fails to acquire the optimal rounding strategy with severe outliers under low bitwidth led to significant quantization loss. On the country, our Q-VLM leverages Information Entropy as a proxy and mines the cross-layer dependency to achieve optimal block partitioning. As a result, our method outperforms ZeroQuant-V2 by **1.71 (79.79 vs. 78.08)** in answering accuracy on ScienceQA dataset under 4-bit in LLaVA-7B model. Additionally, our method enhances inference speed, exceeding ZeroQuant-V2 by 0.06h (6.13h vs. 6.07h) due to utilizing stored rounding parameters instead of dynamic per-token quantization. The additional baseline provides a more comprehensive evaluation framework to highlight the strengths of our approach.\n\n**Q3: Performance on other multi-modal architectures.**\n\n**[Reply]** In response to **Overall Author Rebuttal Q1**, we additionally explored our Q-VLM into another multi-modal architecture OpenFlamingo. Q-VLM designed in LLaVA-like architectures can be effectively adapted to cross-attention based VLMs due to the consistent core mechanism of cross-attention and the robust multimodal alignment capabilities pre-trained on large-scale vision-language pairs. Since OpenFlamingo is a cross-attention based VLM, exploiting cross-layer dependency is particularly suitable. Our method outperforms Q-LoRA by **2.08 (47.84 vs. 45.76)** under 4-bit in OpenFlamingo-3B model. Q-VLM achieves high accuracy on different cross-attention based multi-modal architectures, which means that our method maintain effectiveness and generalizability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the valuable feedback and insightful questions provided by the reviewers. Below are our detailed responses to two common concerns raised by multiple reviewers. Detailed responses to other specific comments are provided under each individual reviewer's comments.

**Q1: Performance on more baseline methods and different multi-modal architectures.**

**[Reply]** Thanks for suggestions which would provide a deeper insight into the effectiveness and generalizability of our method. We have extended our experiments to an additional baseline method ZeroQuant-V2[1] and compared it against our proposed methods. ZeroQuant-V2 leverages per-token quantization with different rounding functions to minimizing activation discretization errors. However, ignoring cross-layer dependency of discretization errors fails to acquire the optimal rounding strategy with severe outliers under low bitwidth and degrades the performance significantly. On the contrary, our Q-VLM mines the cross-layer dependency of output distribution across layers, minimizing the block-wise discretization errors to avoid suboptimal quantization. We further optimize the visual encoder to disentangle the cross-layer dependency for fine-grained search space decomposition. As a result, our method outperforms ZeroQuant-V2 by **1.71 (79.79 vs. 78.08)** in answering accuracy on ScienceQA dataset under 4-bit in LLaVA-7B model. Additionally, our method enhances inference speed, exceeding ZeroQuant-V2 by **1.2h (6.1h vs. 7.3h)** due to utilizing stored rounding parameters instead of dynamic per-token quantization. The results are presented as follows:
|Model|8bit Method|8bit Accuracy|4bit Method|4bit Accuracy|
|---|---|---|---|---|
|7B|ZeroQuant-V2 [1]|89.04|ZeroQuant-V2 [1]|78.08|
||Q-VLM|**89.58**|Q-VLM|**79.79**|
|13B|ZeroQuant-V2 [1]|89.13|ZeroQuant-V2 [1]|78.81|
||Q-VLM|**89.81**|Q-VLM|**80.78**|

The additional baseline provides a more comprehensive evaluation framework to highlight the strengths of our approach. We also explored the multi-modal architecture OpenFlamingo [2] to ensure the robustness and generalizability of our methods. We deploy our method on OpenFlamingo 3B model using Vizwiz and Hateful Memes[3] datasets, selecting bitwidths of 4 and 8 for quantized layers. The experiments are presented as follows:
|Dataset|Shots|FP|8bit Q-LoRA|8bit Q-VLM|4bit Q-LoRA|4bit Q-VLM|
|---|---|---|---|---|---|---|
|Vizwiz|0|23.79|21.24|21.47|17.62|18.69|
||4|27.05|25.83|26.59|24.17|24.55|
||32|39.76|36.38|**37.60**|31.64|**35.52**|
|Hateful Memes [3]|0|50.23|47.75|49.12|43.86|44.22|
||4|50.10|48.62|49.55|45.12|45.26|
||32|50.27|50.02|**51.05**|45.76|**47.84**|

Q-VLM designed in LLaVA-like architectures can be effectively adapted to cross-attention based VLMs due to the consistent core mechanism of cross-attention and the robust multimodal alignment capabilities pre-trained on large-scale vision-language pairs. Since OpenFlamingo is a cross-attention based VLM, exploiting cross-layer dependency is particularly suitable. Our method outperforms Q-LoRA by **1.22 (37.60 vs. 36.38)** under 8-bit in OpenFlamingo-3B model. The advantage of our method becomes more obvious for 4-bit 3B LVLMs because quantization errors and cross-layer dependency play a more significant role in networks with low capacity. These results underscore the robustness and generalizability of our approach across different tasks, model architectures and datasets, demonstrating its effectiveness in diverse scenarios.

[1] Yao, Zhewei, et al. "Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation." arXiv preprint arXiv:2303.08302 (2023).

[2] Awadalla, Anas, et al. "Openflamingo: An open-source framework for training large autoregressive vision-language models." arXiv preprint arXiv:2308.01390 (2023).

[3] Kiela, Douwe, et al. "The hateful memes challenge: Detecting hate speech in multimodal memes." Advances in neural information processing systems 33 (2020): 2611-2624.

**Q2: Why are the proposed methods particularly suitable for LVLM?**

**[Reply]** We appreciate the reviewer's interest in understanding the suitability of our proposed methods for LVLM. Our cross-layer dependency mining method is also suitable for single modality model. Specifically, utilizing entropy as proxy to assess layer sensitivity and achieve optimal block partitioning is not exclusively beneficial for LVLMs. For single language modality, we deploy our CDM into the LLaMA-7B[1] model and evaluated it on the PIQA[2] dataset. Our method achieved improvement compared to QLoRA by **0.51 (72.23 vs. 71.82)** in 4-bit precision. For single vision modality, we incorporated our CDM into the CLIP only quantized LLaVA-7B model and observed enhancement of about 0.23 (84.15 vs. 83.92) in answering accuracy on the ScienceQA dataset under 4-bit precision. 

However, ablation studies in Table 1 indicate limited performance improvement by 1.13 (78.66 vs. 77.53) when solely using cross-layer dependency mining without visual encoder optimization method. The proposed visual encoder optimization method is unsuitable for single-modal models from the perspective of vision-language synergistic optimization to achieve fine-grained blocks with search cost and cross-layer dependency trade-off. Significant performance gains by **2.26 (79.79 vs. 77.53)** are achieved by jointly employing cross-layer dependency mining and visual encoder optimization method. Our approach achieves vision-language synergistic optimization, resulting in superior performance in LVLM tasks.

[1] Touvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).

[2] Bisk, Yonatan, et al. "Piqa: Reasoning about physical commonsense in natural language." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020.

Meta-review: Previous post-training quantization is to search the layer-wise rounding functions sequentially. This paper claims it is suboptimal, and proposes to leverage some cross-layer dependency and search block-wise rounding functions. It also proposes to use entropy as the proxy to partition the blocks optimally. The results have shown no much performance regression when quantizing LLava model. 

Most of the reviews stay positive on this paper due to 1) it is reasonable and well motivated to search block-wise rounding functions; 2) the results has very minor performance regression. But he common concerns are: 1) the motivation to use entropy as proxy; 2) generalization to single-modality models; 3) missing more evaluation on other benchmarks; 4) more fair comparison with baselines. After the rebuttal, most of the concerns are resolved, but some are still remained, e.g. missing stronger baselines, the method is not specifically for VLM but also applicable to single-modality models. The authors should reflect the rebuttal and further improve the remaining issues in the final version.
Decision: Accept (poster)