ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
Authors: Yi-Lin Sung
Jaehong Yoon
Mohit Bansal
Venue: ICLR 2024 poster (2024)
Keywords: multi-modal learning
model pruning
layer-wise pruning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities, achieving remarkable performance improvements on various multimodal downstream tasks. However, deploying LVLMs is often problematic due to their massive computational/energy costs and carbon consumption, making it infeasible to adopt conventional iterative global pruning, which is costly due to computing the Hessian matrix of the entire large model for sparsification. Alternatively, several studies have recently proposed layer-wise pruning approaches to avoid the expensive computation of global pruning and efficiently compress model weights according to their importance within a layer. However, these methods often suffer from suboptimal model compression due to their lack of a global perspective. To address this limitation in recent efficient pruning methods for large models, we propose Efficient Coarse-to-Fine Layer-Wise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs. We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients. Then, the multimodal model performs layer-wise unstructured weight pruning. We validate our proposed method across various multi-modal and single-modal models and datasets, demonstrating significant performance improvements over prevalent pruning techniques in the high-sparsity regime.

Published as a conference paper at ICLR 2024
ECOFLAP: EFFICIENT CO-RSE-TO-FINE LAYER-WISE PRUNING FOR VISION-LANGUAGE MODELS
Yi-Lin Sung Jachong Yoon Mohit Bansal Department of Computer Science, UNC Chapel Hill (ylsung, jhyoon, mbansal @cs. wnc. edu
ABSTRACT
Large Vision- LanguaLa Models (LVLMs) can understand the world comprehen- sively by integrating rich information from different modalities, achieving remark- able advancements on various multimodal downstream tasks. However, deploying LVLMs is often problematic due to their massive computationall/enrgy costs and carbon consumption. Such issues make it infeasible to adopt conventional iterative global pruning, which costly due computing the Hessian matrix of the entire large model for sparsification. Alternatively, several studies have recently proposed layer-wish pruning approaches to avoid the expensive computation of global prun- ing and efficiently compress model weights according to their importance within a layer. However, they often suffer from suboptimal model compression due their lack of a global perspective. To address this limitation in recent efficient pruning methods for large models, we propose Efficient Coarse-to-Fil Fine Layer- Wise Pruning (ECoFLaP)) two-stage coarse-to-fing weight pruning approach for LVLMs. We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients. Then, the model performs local layer-wise unstructured weight pruning based on globally-informed sparsity ratios. We validate our proposed method across various multimodal and unimodal models and datasets, demonstrating significant performance improve- ments over prevalent pruning techniques in the high-sparsity regime.
INTRODUCTION
Deep learning models have been increasingly growing in size (Radford al., 2018; Brown et al., 2020; Liu et al., 2023; Zhu et al., 2023) in order to have a sufficient capacity to learn challenging tasks in the real world. However, this exorbitant model size requires significant computations and memory to deploy, which limits their applicability in many resource-constrainee environments. Model compression, a strategy to reduce the size of neural networks while preserving their capabilities (Dai et al., 2018; Park et al., 2022; Xiao et al., 2023; Fang et al., 2023), has gained popularity in tackling this problem. In order to build lighter, faster, and more interpretable models, various directions have been studied in parallel, including model pruning (Dai et al., 2018; Fang et al., 2023; Frantar & Alistarh, 2023), quantization (Park et al., 2022; Xiao et al., 2023), layer drop (Sajjad et al., 2023), and token merging (Bolya et al., 2022; Bolya & Hoffman, 2023). Among them, in this paper, we study an unstructured model pruning approach that has strong potential to preserve model performance, even with high compression rate of large vision-language models.
While most model pruning approaches tackle the problem of rision-based tasks (Liang et al., 2022b; Kong et al., 2022; Wei al., 2023) and a few recent works have studied reduce the size of language models (Ma et al., 2023; Sun et al., 2023a; Frantar & Alistarh, 2023), efficient pruning for multimodal models like Large Vision- Visguu-Lan Models (LVLMss) (Ye al., 2023; Li al., 2023a) have been understudied due to their architectural complexity and the disparities in data characteristics between different modalities (Shi et al., 2023). Unlike unimodal learning methods, network architectures for multimodal learning (Li et al., 2022; 2023c) are often composed of modality-speciff sub-modules to capture the knowledge ge for each modality, which substantially expands the scale of multimodal models.
Our project page and code are available httpp:://eeat /appe//////f/p/p/e/eoo/ thub. io/

---

Model pruning for transformers. Efficient model inference via pruning (Yoon & Hwang, 2017; Dai et al., 2018; Zhang et al., 2022a; Fang et al., 2023) is one of the long-standing topics in the community and is coming to the fore as transformer- based models grow in size. Model pruning is typically categorized into structured praning and unstructured pruning. The former (McCarley et al., 2019; Wang al., 2019; Kwon et al., 2022) aims accelerate inference speed and throughput at Published as a conference paper at ICLR 2024
Moreover, this modularization leads to significant imbalances in the weight/gradient distributions between modules associated with different modalities, making unified pruning difficult.
As conventional iterative pruning (Mallya & Lazebnik, 2018; Molchanov et al., 2019; Wang et al., 2020; Zhang et al., 2022b) demands extensive computations to learn pruning masks by computing the inverse Hessian in large models, layer-wise one-shot pruning (Chen & Zhao, 2018; 8; Frantar & Alistarh, 2023; Sun et al., 2023a) has recently been developed to compress them efficiently. These approaches discard uninformative model weights in single iteration without computing the expensive second- order Hessian matrix for the entire model architecture, yet often suffer from finding the optimal sparsity ratio per layer, as they resort to intra-layer weight importance without sufficient understanding of global (i.e., inter-layer) weight correlations in the model, hence leading to suboptimal pruning.
To overcome this limitation of layer-wise pruning and build an efficient approach for LVLM compres- sion, we propose Efficient Course-to-Fine Layer-Wise Wise Pruning (ECoFLaP) that obtains an adaptive sparsity ratio per layer in a single step based on a 'global importance score' (Coarse) and then removes parameters that are less critical (to the model's performance), in layer-wise manner ( (Fine). Note that our proposed method is computationally efficient by leveraging the first-order gradient to obtain global importance score without Hessian operations, allowing layer-wise pruning to leverage a holistic correlation across the model weights. To further improve the memory efficiency of the global score computation, we introduce the zeroth-order approximate gradient computed by the forward-forwart algorithm (Malladi et al., 2023; Hinton, 2022). In the end, our ECoFLaP obtains highly compressed large models in single shot, with adaptive sparsity in each layer, leveraging global weight importance approximated by memory-efficient forward-forwer operations. We note that our proposed pruning method generalizes well not only Lision-Language multimodal tasks but also to unimodal tasks, including ImageNet (Vision) and MMLU (NLP).
We extensively validate the efficacy of our approach across a variety of models and datasets, including but not limited to EVA.VIT, FlanT5, LLaMA, BLIP/BLIP-2, CLIP. and on image classification (ImageNet- Ik), multitask multiple-choice question answering (MMLU), next token prediction (Wiki- Text), visual reasoning (NLVR²), visual question answering (VQAv2, OK- VQAA GQA), image captioning (NoCaps, COCO Captions), and image-text retrieval (Flickr30k). Our proposed ECoFLaP surpasses iterative global pruning and SoTA layer-wise pruning methods, SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023a), with relative improvements up to 5% in average performance across multiple VL tasks. Additionally, our approach outperforms recent SoTA in pruning Aision-language model, UPop (Shi et al., 2023), achieving relative improvements of 1.8% and 2.6% NLVR and COCO captions. ECoFLab also generalizes well unimodal models (FlanT5, LLaMA, EVT),,T), particularly in a higher sparsity regime. is worth noting that ECoFLaa LaP with the zeroth-order gradient reduces the GPU memory usage by up to 40% compared baselines using the first-order gradient to estimate the global importance. These results not only underscore the potential of our method but also pave the way for more efficient and effective pruning techniques in the future.
We summarize our contributions as threefold:
We propose novel layer-wise pruning method for vision-language models, coined Efficient Coarse- to-Fine Layer- Layer-Wise Pruning for Vision-Language Models (ECoFLaP), that finds adaptive sparsity per layer by leveraging the global importance score approximated via peroth-order gradients.
We validate the proposed method on various backbone structures with diverse vision-language and unimodal tasks, and demonstrate the superiority of our method by consistently outperforming the
SoTA pruning baselines for large models on multiple benchmark datasets.
We provide extensive analyses and ablation studies that help to understand the challenges of unified multimodal model pruning and the strengths of our proposed methods compared to baselines, including visualizations of importance score distribution, dynamic sparsity, loss landscape, etc.
RELATED WORK
2

---

Published as a conference paper at ICLR 2024
the expense of model performance. Unstructured pruning (Sun al., 2023a; Frantar & Alistarh, 2023), on the other hand, can be advantageous in preserving performance even with high model sparsity given Al acceleration software sparse matrix computation schemes (Han et al., 2016; Mishra et al., 2021; Das & Ramamoorthy, 2022; NeuralMagic, 2022). Recently, Frantar & Alistarh (2023) suggest one-shot pruning technique, SparseGPT, for generated pre-trained transformers (GPTs) in an unstructured manner. They newly employ sparse regression solver that prunes weights each layer based on row-wise Hessian reconstruction as formulated by a closed-form solution. Wanda (Sun et al., 2023a) proposes a magnitude- based unstructured pruning approach for large language models (LLMs). promotes layer-wise weight sparsification based on the importance, computed by multiplying weights and input activations. However, these SoTA pruning methods rely on the pre-defined pruning ratio that all layers resort to the same sparsity, restricting the upper bound on model compression. In addition, their methods are tailored to language models without concern for different modalities. On the other hand, our proposed method allows adaptive pruning at each layer without heavy computation of global gradients. Further, to the best of our knowledge, we propose a first unified sparse approximate solver for vision- laiiunla multimodal models.
Transformers for vision-language multimodal learning. Vision-languall multimodal learning has shown remarkable achievement on various tasks, such as classification (Liu et al., 2018; Liang et al., 2022a), retrieval (Fei et al., 2021), )), few-shot learning (Tsimpoukelli et al., 2021; Alayrac et al., 2022), visual QA (Kim et al., 2016; Liu al., 2023), and image/video generation (Zhou et al., 2022b; Singer et al., 2022; Lee et al., 2023). Recently, transformer modules have become the new standard for both language and visual-based (Dosovitskiy al., 2020; Liu al., 2021) tasks, and the powerful representation of these transformers allows for the modularization of multimodal learning frameworks with existing pre-trained und-modal models without an expensive re- re-training phase for multimodal data (Radford et al., 2021; Zhou et al., 2022a).
3 BACKGROUND: LAYER-WISH PRUNING FOR VISION-LANGUAGE MODELS
3.1 PRELIMINARIES: A FRAMEWORK FOR VISION-LANGUAGE MODELS
Multiple vision- language multimodal models introduce different architectural designs to understand compositional multimodal semantics, which can affect the pruning strategy. In this paper, we basically build our pruning method upon two popular multimodal learning frameworks for vision- vision-language tasks, BLIP (Li et al., 2022) and BLIP-2 (Li et al., 2023c), which are modularized with the pre-trained ViT (Dosovitskiy et al., 2020) and FlanT5 (Chung et al., 2022) architectures to encode visual and language information, respectively. They have achieved remarkable zero-shot performance on various vision-language tasks. Let the visual encoder fo; W") and language model fi w in BLIP variants be composed of a set of the weights W = (W:l < < M) and w = [W < N). respectively, where M and N indicate the number of the corresponding layers. While freezing these pre-trained backbones, BLIPs introduce Query transformer (Q-Former) (- W lightweight module with set of the weights W that extracts essential information from visual representation and warps it to the input space of the language model (Li et al., 2021). Given the multimodal input pair (x", x'), the output of the model is represented as ((o",, x²]; W where o" denotes the aligned visual representation (xx W"); W
3.2 CHALLENGES IN PRUNING VISION-LANGUAGE MODELS
Let us first describe a primary strategy for Hessian- global pruning. which aims to remove the optimal subset' of parameters minimizing the loss change SL C(w+6w L(w). Here, denotes the weight vector of the model. When the 7th model parameter is removed, implying e. 6w + w; = 0, where e, indicates the 7th canonical basis vector, its importance represented as w After that, the model prunes its weights with the lowest importance. However, computing the inverse Hessian H requires significant computational budgets and is often infeasible for large models.
H
On the other hand, layer-wise one-shot pruning aims to leverage small amount of calibration data (such as 128 data points) to remove less important weights from the model in a single step
'The subset size (i.c., sparsity) often pre-defined obtained through training.
3

---

Published as a conference paper at ICLR 2024
Gradlert
Tiparatoon intwecn SEAIE.
Low Score 100r00r1P
vilce mode!
4.50
2.5
6.012
0.0010
1:30
01
500
300
300
(a) Weight Magnitudes
(b) Weight Gradients
(c) Local Score of SparseGPT
Figure 1: (a) and (b): The imbalance of the magnitude and gradient distributions between vision
and language models. (c): The skewed distribution of the layer-wise scores of SparseGPT.
based on their local significance per layer. This approach is significantly efficient in terms of computation/memooy the model pruning phase without expensive iterations of the re-training or Hessian matrix computation for the parameters of the entire model, which is particularly beneficial for large languageemultinndall models with billions of parameters. The model compression procedure of recent layer-wis, wiiee-wise pruning methods (Lazarevich et al., 2021; Yu & Xiang, 2023; Frantar & Alistarh, 2023) can be summarized in the three steps given a target sparsity of p% and a layer index = 1, i E {1...., T.J: (1) Compute the importance score for each parameter within the layer i, (2) Discard p% of the parameters of layer i based on their importance score and adjust the remaining weights (usually for Hessian -basaadd-dadddd--- methods), and (3) calculate the layer's output with only activated (i.e., not pruned) weights, and repeat the process for the next layer (i <- i + 1) until i =
However, most layer-wiss wise pruning approaches focus on removing unnecessary weights parameters from single-modal models, such as vision-only (Chen & Zhao, 2018; Lazarevich et al., 2021; Yu & Xiang, 2023) or language-only (Frantar & Alistarh, 2023; Sun et al., 2023a) architectures, which face significant limitations when extended to multimodal (e.g., vision-language) models due to the critical gap in weight/gradient distributions between different modalities, shown in Figures la and lb. Unlike global pruning methods estimating the importance of each parameter based on the objective loss, layer-wise pruning without the guidance of global weight correlations suffers from obtaining relative significance across weights in different layers/modules due to such severe distributional imbalances in the different modality modules. For example, in Wanda, inputs for different layers significantly vary in their scales (Ba et al., 2016; loffe & Szegedy, 2015), and in SparseGPT, the importance of the local loss functions for different layers is also incomparable (Please see the skewed distribution of the local score shown in Figure 1c). This problem makes layer-wise pruning approaches usually set the pruming ratio to be a fixed value for all layers, resulting in a suboptimal compression rate and performance. This is also evident in the results from Singh & Alistarh (2020) that the global WoodFisher pruner outperforms its layer-wise counterpart by a large margin, especially at high sparsity regimes.
4 ECOFLAP: EFFICIENT COARSE-TO-FINE LAYER- WAYE PRUNING
As discussed above, deploying layer-wise pruning approach for large vision-language models is challenging due to three primary issues, distribuiina/aarchteetti modalities, significantly large model size, and lack of global knowledge. To overcome such critical challenges, propose Efficient Coarse-to-Fine Layer-wise Pruning (ECoFLaP), to efficiently compute an adaptive pruning ratio for each layer with global importance scores (Coarse), and then accurately remove the parameters layer by layer with local significance (Fine).
disparity between different
4.1 THE FRAMEWORK OF COARSE-TO-FINE LAYER-WISK PRUNING
Our focus is to prune the Bile-like multimodal architectures, as discussed in Section 3.1. Note that compress the vision and language model in BLIP variants while keeping Q-Former intact because it is sufficiently lightweight, occupying only ~ 5% of the parameters in the whole framework. Given the multimodal calibration data D {xy, xf K k a conventional layer-wise pruning method aims to
4

---

Published as a conference paper at ICLR 2024
Iterative Global Pruning
Layer-wise Pruning
ECoFLaP
stops
x stop
(2) Fine step
L
(1) Coarse
Fixed sporsiy rate
step
Language
tt layer
tthh layer
model
Former
model
Spanstty
Spassty
Image
x. step
Image
X 1 step
Image
Figure 2: Illustration of our ECoFLaP compared to global and layer-wise pruning. The boxes with blue, green, and black border denote the language, vision, and Q-Former modules, respectively. The dotted red arrows show the working flow of the algorithm. The beige color indicates the pruning of the current step (layer) is conditioned on the pruning decisions made in the preceding steps (layers), which are marked in gray. ECoFLaP first performs the efficient coarse step to obtain the pruning ratio for each layer by leveraging the zeroth-orde gradient, and then removes the uncritical weights in layer-wise fashion in the fine step.
find the sparse weight W, at layer via the corresponding local objective Ci:
W; W.
W; arg max S(W; W,,,,, s.t.,
Pi:
(I)
where S.) is the score function to compute the importance of the weight, W; is denoted as the pruned weight from W,, and p.% the desired sparsity layer i. Basically, pt is the same for all layers in layer-wise pruning methods.
In our proposed Efficient Coarse-to-Fine Laser-wise Pruning (ECoFLaP) framework, estimate the optimal sparsity for each layer in the vision- language model by computing the global importance of model weights. One straightforward direction to obtain the layer sparsities to perform global pruning on the model for the target global sparsity by leveraging the global loss objective L.
W
W arg max (WW S.L.,
p-
22)
Then, we can estimate the layer sparsity via p; = W W However, this approach requires an expensive operation obtain gradients for all weights in the large model and to extract the pruned weights via a argmax, consuming enormous memory and computational resources. Therefore, we introduce an alternative numerical approach, which computes the keep ratio (1 sparsity ratio) based on importance scores linearly, to avoid expensive operations that can estimate the global importance score efficiently. First, we obtain the importance score for each weight via the global objective function, namely, S (W L). and then convert the scores to sparsity by three steps: (1) Find the total parameters that need to he selected based on p. (2) Normalize the scores, (3) Obtain the sparsity for each layer based on the number of parameters to be picked and the parameters of this layer. The example to determine the sparsity ratio for the 7th layer of the model shown below,
normalize(s:,s)
(3)
Σs S
P: (Sormalize(s;555 s) NNeleat)
where N (1 -
(4)
W.
We then inject the derived sparsity ratios back to Equation to prune the model layer-wisely. We also introduce hyperparameter Pmazz to control the maximum sparsity for each layer to avoid the layer collapse, which is discovered by previous literature (Tanaka et al., 2020) that the global pruning approach may remove all the parameters for one layer. To incorporate this hyperparameter in the Equation 3, we simply pre-pick the parameters for each layer to satisfy the max sparsity condition, subtracting the Nselect with the number of pre-picked parameters, and start the algorithm. In our experiments, we find using the same sparsity ratio for the layers in one block (such as the feed-forward layer and attention layer in transformer block) makes the results more robust, so we also extend the
5

---

Published as a conference paper at ICLR 2024
algorithm to determine group sparsity ratios by via group scores (aggregating all the layer scores in block). For the layer-wise pruming step, we utilize Wanda, which combines the input activation of the layer with the weight magnitude the local importance score, to remove uncritical parameters based on the obtained layer sparsity ratios.
4.2 ZEROTH-ORDER GLOBAL INFORMATION
(a) Zeroth-order Gradient
(b) Forward-forwart Algorithm
In the previous section, we introduce our coarse- to-fine pruning framework determine the prun- ing ratios for layer-wis, pruning. Note that our goal is to prune models with billions of param- eters, so we again focus the one-shot setting in this step without any iterative refinement. To this end, our first attempt is to use the first-order backward gradient with weight magnitude (Le- Cun et al., 1989), |W|- [V, D)1, as the global importance score, and the performance improvement demonstrates the effectiveness of our framework (Please see Table I and Figure 4). However, even though the first-order backward gradient is still tractable for most models. like less than 5B parameters, 7B LLaMA (Touvron et al., 2023) requires more than 40GB even with float16 weight type and batch size being one.
W+cc
Lg 11-b L2 2r
V.L
C2
W
cz
W
Figure 3: Illustration of (a) the zeroth-order gradi- ent in weight space, and (b) the forward-forward algorithm.
To further reduce the cost of computing gradients, we instead compute the zeroth-order approximated gradient by replacing the backpropagation with the forward-forwart algorithm (Hinton, 2022; Malladi al., 2023). We take the 2th layer of a model as an example. the zeroth-order gradient of W; can be computed by perturbing the weight with Gaussian noises twice, that is,
C(W, + ez, L(W, EZ, d)
D)1l2 = EEE-EEDDEEEE
(5)
2e
We hence use C(W. D)ll as the importance score of W., Note that we do not multiply with the weight magnitude as solely using the approximated gradients works better empirically. The computation of Equation is extremely memory-
and the exact process for the 2th layer is shown in the following: (1) First, we choose random seed and then sample noise (now
use 2x of memory of "this layer"). (2) we modify the model's 's weight with W + Ez and compute the loss change L(W + cz, d). (3) we then sample the same noise 2 as the first step (by setting the same random seed), and we modify the model's weight with W cz, which can be done by subtracting 2ez from the previous weight, and compute L(W EZ, d). (4) obtain the gradient norm by (C(W ez)- L(W - (2e) (all terms are scalars). From the above process, for an N-layer network, we only need (N+1) times of one-layer memory, which is significantly memory-efficiend than storing the whole gradients that cause 2N times of one-layer memory. This explains why our approach uses much less memory than first-order based methods, which also need to cache input activations. One may suspect we need to sample many noises to capture the loss of the landscape and to estimate the gradient accurately. However, our experiments and the previous studies (Malladi et al., 2023) show that one noise is sufficient. We describe our approach in detail in Algorithm I.
EXPERIMENTAL SETUP
We report evaluation metrics and more experimental details in Appendix C.
Architectures. We use multiple uni- and multi-modal architectures for experiments: the encoder- decoder vision of BLIP-2 (Li et al., 2023c), composed of pre-trained EVA-ViT (ViT-g/14 from EVA-CLIP) (Sun et al., 2023b) and F5anT5 (Chung et al., 2022), is used for most experiments and ablation studies. And, we also extend our approach to BLIP (Li et al., 2022) with ViT (Dosovitskiy al., 2020) and BERT (Devlin et al., 2019) backbones. In addition, evaluate approach solely on unimodal vision and NLP tasks with EVA-ViT FlanT5, and LLaMA backbones.
Evaluation Datasets. We evaluate the zero-shot ability of BLIP-2 on various datasets after pruning, such as VQAv2, OK- VQA, and GQA for visual question answering, NoCaps for image captioning, and Flickr30k for image-text retrieval. For BLIP, we evaluate the performance change of the BLIP

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of existing pruning approaches with ECoFLaP on the zero-shot performance with BLIP-2 at 0.5 sparsity. We report accuracy for visual question answering, CIDEr and SPICE for image captioning, and TR@1 (text recall) and IR@) @ (image recall) for image retrieval. We also compute the macro average score over tasks and the GPU memory usage for each method.
Method
Sparsity
Method's Mem. Usage
VQAv2
Visual Question Answering
OK-VQA
GQA
Image Captioning
NoCaps
Image retrieval
Flickr30k
Macro
Avg.
(GB)
Accuracy
CIDEr
SPICE
TR@1
IR@1
Full Model
0%
63.1
41.1
44.1
121.7
15.8
97.6
89.7
62.1
Global Magnitude Pruning
50%
8.46
0.0
0.0
0.0
0.1
0.0
Gradient- hased Pruning
50%
22.4
56.9
35.3
41.7
96.8
13.5
93.0
82.4
55,4
SparseGPT
50%
9.04
57.4
36.1
40.8
108.1
14.1
96,4
86,3
57.4
Wanda
50%
8.87
57.9
35.3
42.2
106.9
14.1
95.1
84.6
57.2
ECoFLaP
First-order Gradient
50%
22.4
59.4
36.3
42.3
109.9
14.2
95.9
86.2
58.2
Zeroth-order order Gradient
50%
8:93
58.4
35.6
42.9
110.2
14.3
95.5
85.8
58.0
with Parts
pick
0.75
8:20
6.4-
8:35
525-
6.44
58.6
Wands
Warda
8:50
Wance
CORELAP firt orter gradi
DEatiaP Wit orter gradl
Ecatiat dest onder gratl
Learth Steer grud
OCCTLLT Lwrohh order grdd
4.50
ICol lzureth order grad gradinti
Overal Sparite
Overal Searshy
Ovena Searsty
(a) BLIP-2 on VQA
(b) F5an'55 on MMLU
(c) EVA-VIT ImageNet
Figure 4: The accuracy and model sparsity trade-off for Wanda, ECoFLaP with first-order and zeroth-order gradient on both BLIP-2 (multimodal), FlanT55 (unimodal), and EVA-VIT (unimodal).
fine-tuned on NLVR and COCO captions. For unimodal models, we evaluate FlanT5 on MMLU, evaluate LLaMA 7B with WikiText, and evaluate EVA-VIT with ImageNet-Ik.
Baselines. We compare ECoFLaP to several global pruning and layer-wise pruning approaches. For Global Magnitude Pruning, we perform global pruning based on the magnitude the weight. Gradient-based Pruning is iterative global pruning approach, where of the first-order gradient and the weight magnitude as the importance score and prune the model target sparsity in 3 iterations. SparseGPT is a layer-wise flssiannbbssd method. Wanda is also a layer-wise approach utilizing the multiplication of the weight magnitude and the norm of input activation the local importance score. The comparison of the above pruning methods and their importance metric can be found in Table 8. We also compare our method with UPop. which prunes and re-trains the vision-language model simultancously in unified progressive pruning manner.
the multiplication
RESULTS
6.1 MAIN EXPERIMENTAL RESULTS
We demonstrate the zero-shot performance on various datasets using BLIP-2 pruned by our ECoFLaP and baselines at 0.5 sparsity ratio in Table 1. And we compare our approach with UPop using BLIP backbone on 2LVR and COCO captions under the fine-tuning and non-fi non-fine-tuning settings in Table 3. We also provide the comparison of our approach and Wanda at various sparsities on both multi- and uni-modal models in Figure We summarize our observations as follows:
ECoFLaP consistently outperforms Global Magnitude Pruning (GMP) and Gradient-based Pruning as shown in Table 1. GMP fails to generate meaningful results at 0.5 sparsity. In particular, we find that GMP suffers from pruning the vision module, EVA-VIT, so the degraded visual repre- sentation harms the multimodal model performance. Gradient- based Pruning performs significantly better by selecting crucial weights based on gradient information of the whole model, but it requires a larger memory than GMP due to the full pass to compute the gradients. On the other hand, com- pared Gradient-based Pruning, our ECoFLaP with zeroth-order gradients uses only 40% memory
7

---

Published as a conference paper at ICLR 2024
Table 2: Zero-shot evaluation on 11 classification tasks with CLIP using 0.4 of sparsity ratio. We ad- ditionally report results using local scores to compute layer sparsity ratios. "FGVC, OFlowers, OPets, and SCars stand for FGVCAircraft, OxfordFlowers, OxfordPets, and StanfordCars, respectively.
Methods
Caltech101
EunSAT
FGVC
Food101
ImageNet
OFlowers
OPess"
SCars"
SUN397
UCFI01
Avg
92.9
47.8
24.8
86.1
71.3
62.6
Wanda
85.1
31.2
33.5
9.0
66.0
45.6
37.2
71.3
35.0
50.8
54.7
47.2
SpureGPP
90.5
41.5
47.4
19.2
79.4
50.8
39.0
86.4
53.4
55.3
63.2
58.7
Spansiies
Local Scores
w/ Wanda
84.0
30.1
31,5
6.6
62.9
44.5
34.2
68.1
34,1
49.5
52.9
45.3
SpursoGiPT
83.2
46.1
48.0
ECaFLaP
w/ Wanda
87.5
37.9
43.8
16.0
72.8
53.5
57.0
80.3
46.9
58.2
62.4
56.0
w/ SpurseGFT
90.0
424
45.4
22,5
81.2
56.5
64.1
88.3
56.8
60.2
63.4
61.0
during pruning while improving average accuracy by 2. 22%,, 9.88%, and 3.41 %, visual question answering, image captioning, and image retrieval tasks, respectively.
ECoFLaP outperforms SoTA layer-wise pruning baselines since our ECoFLaP is able exploit the global information. Our approach with zeroth-order gradient consistently achieves better performance against Wanda on all tasks and metrics by 1.1%, 2.3%. and 1.0%, on visual question answering. image captioning, and image retrieval tasks, respectively, showing the effectiveness of our coarse- to-fine framework. Furthermore, we highlight that the gap between Wanda and ECoFLaP becomes larger when the sparsity increases to 0.6, where ECoFLaP outperforms Wanda by 9.6% on VQA (shown in Figure 4a). Our approach also surpasses SparseGPT by 1.4% of relative improvement on the average score. Note that our zeroth-order ECoFLaP uses on-par memory budgets compared to layer layer-wise pruning methods, Wanda and SParseGPT, by avoiding expensive backpropagation computation. Additional results the combination of ECoFLaP and SparseGPT are in Table 9.
ECoFLaP surpasses baselines on pruning CLIP models by a significant margin. Beyond pruning modularized VL models, ECoFLaP can also work well with joint VL models like CLIP. In Table 2, we evaluate Wanda, SparseGPTT, and our approach on CLIP by measuring the zero-shot accuracy on 11 classification tasks before and after pruning over the sparsity 0.4, and our ECoFLaP excels against baselines while effectively mitigating the degeneration of zero-shot accuracy through the model pruning. In addition, we assess the impact of using global scores for computing layer sparsity ratios by comparing it with the approach using local scores derived from layer-wise pruning methods (namely. Wanda and SparseGPT). The result shows that the strategy of using local scores leads to decrease in performance compared to leveraging a uniform sparsity ratio. This observation emphasizes the ineffectiveness of employing local scores for assessing layer importance (Please see our motivation in Figure 1).
ECoFLaP outperforms the multimodal model prun- ing baseline, UPop, in both non-fine-tuning and fine- tuning cases as shown in Table 3. Note that UPop requires re- te-triining the backbone model in the pruming phase, which is significantly expensive in memory and computation, especially for large models. On the con- trary, ECoFLaP finds an important subset of weights in the model in single shot and then fine-tunes the pruned model. We also emphasize that our approach general- izes different architectures and datasets, where BLIP uses a BERT-encoder for NLVR² (classification) and a BERT-decoder for COCO (image captioning).
Table 3: Performance comparison on BLIP at 0.5 sparsity on NLVR² and COCO cap- tions.
Method
NLVR'
test
CIDEr
COCO cap.
SPICE
Full Model
82.3
83.6
133,3
23.8
w/o fone-tuning
UPop Wanda
76.9
77.8
78.3
78.1
97.1
18.4
ECoFLaP
79.1
79.2
111.0
20,3
w/ fine- tuning
UPop
80.3
81.1
128.9
23.3
ECoFLaP
$1.8
82.5
23,8
ECoFLaP also generalizes well to unimodal tasks/architectures. In Figure 4b and Figure 4c, we visualize the accuracy of MMLU with FlanT55 and ImageNet- 1k with EVA-ViT after pruning with various sparsity ratios (from 0.1 1 to 0.6). Since Wanda applies the same pruning ratio to all layers, performance degrades with increasing sparsity (~ 40% 1), which is catastrophic for some layers containing much more informative weights than others. Alternatively, the performance degradation caused by pruning weights can be significantly mitigated by proposing adaptive layer-wise pruning with dynamic sparsity for each layer based on estimated global weight importance. Furthermore, as shown in Table 7, our approach shows the ability to generalize to larger LLM, where we attain 10.6% relative improvement over Wanda when evaluated on WikiText using perplexity with LLaMA 7B.
8

---

Published as a conference paper at ICLR 2024
Table 4: The ablation on the Table 5: The ablation on number of samples.
Table 6: The ablation on number of noises (total samples 32).
maximum sparsity.
BLIP-2
Max
Sparsity
BLIP-2
Number of Samples
BLIP-2
Number of noises
(== 0.5)
0.5
0.6
0.8
Ip = 0.5)
32
64
128
( p= 0.5)
VQA
57.9
58.4
57.1
51.6
VQA
58.2
58.4
58.3
5M.4
VQA
58.4
58.1
58.0
58.1
6.2
ABLATIONS AND VISUALIZATION
We provide ablation studies of our zeroth-order ECoFLaP on maximum sparsity for each layer pM the number of data (D), and the number of perturbed noises per sample. We report the zero-shot VQA accuracy obtained from the BLIP-T5 with 0.5 total sparsity. Note that we use the numerical method to obtain the sparsity ratios for experiments (Please see Equation 3).
Maximum Sparsity. First, we observe that the maximum sparsity is critical as it affects the results considerably, and the optimal value happens at 0.6, which is a conservative value given the target ratio is 0.5. Similar to the findings discovered by Tanaka et al. (2020), the algorithm might suffer layer collapse when the maximum sparsity is not set (PM = 1). We find it simple but effective to introduce this hyperparameter
would like to avoid using the iterative solution proposed by Tanaka et al. (2020) because we are targeting the one-shot pruning. Based on the result shown in Table 4, we simply set PM = p + 0.1 throughout our experiments.
Number of Data and Noises. In the next ablation experiment, we focus on understanding how many samples and noises we need to estimate the gradient with zeroth-order optimization. We first ablate the effect of the number of data by fixing the number of noises per sample to be one, and the result in Table 5 shows that our approach is robust to the number of samples across 16 to 128, and we thus using 32 samples for better efficiency. Next, we explore if having a more accurate gradient estimation by sampling more noises can improve the accuracy. In this ablation, we make sure to forward through the model the same number of times when using different numbers of noises, meaning that we cut the number of data to 1/n when we increase the number of noises to n. The result in Table 6 demonstrates that using more data samples is more effective compared to sampling more noises when the budget is fixed.
Visualization and Analysis. We also provide the visualization of the sparsity ratios obtained by ECoFLaP (Figure 6) and the loss landscape of BLIP-2 (Figure 5) to justify the use of reroth-order optimization. In Figure 6, we observe that our ECoFLaP variants allocate sparsity ratios in a similar distribution, in favor lower sparsity ratios for the visual model and higher sparsity ratios for the language model, and this alignment provides evidence that the zeroth-order gradient obtained by the forward-foree forward algorithm can be the accurate but cheaper alternative to the first-order gradient in importance estimation. The landscape shown in Figure 5 presents that BLIP-2 is located at smooth basin with a single local minimum. This loss landscape justifies the success of utilizing zeroth-order optimization we avoid sampling weights located in other basins that may cause inaccurate gradient estimation. We also present a qualitative analysis of the pruned model in Appendix E.
7 CONCLUSION
This paper has investigated the challenges of pruning large-scale vision-language models due to the dilemma between immense memory/computational overhead from global pruning and the suboptimal model performance of layer-wise pruning. Existing global pruning methods require an expensive backpropagation process compute the inverse Hessian to obtain global weight importance, which makes these approaches infeasible for current large LLMs and large LVLMs. On the other hand, layer-wise pruning approaches are highly efficient by pruning weights based on layer-wise local information, but often suffer from a lack of global importance, resulting in a significant performance degeneration. To address the limitations of these pruning approaches while enjoying their merits, we propose ECoFLaP that adaptively finds the optimal sparsity for each layer by efficiently estimating the global weight importance, and then accurately prunes the languaela model in layer-wise manner dependent the obtained pruning ratios. We utilize the zeroth-order optimization to obtain the gradients with forward pass only and achieve superior performance over multiple uni- and multi- modal tasks against recent layer-wist pruning methods by significant margin, while consuming only around 40% of GPU memory compared to the iterative global pruning approach. We hope our new efficient pruning can help deploy quality yet compact vision- vinion-language models.

---

Published as a conference paper at ICLR 2024
8 REPRODUCIBILITY STATEMENT
Our codes are based on the publicly available LAVIS (Li et al., 2023b), Wanda (Sun et al., 2023a), MMLU (Hendrycks al., 2021), UPop (Shi et al., 2023), and CoOp (Zhou al., 2021). The experimental setup and details can be found in Section 5 and Appendix C, respectively, We also have made our code publicly available.
ACKNOWLEDGMENTS
We thank Prateek Yadav for helpful discussions. This work was supported by ARO Award W911NF21102201 ONR Grant 1-014--3-1-23566 and NSF-AI Engage Institute DRL-21 1263. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. International Conference Computer Vision (ICCV), pp. 8947-8956, 2019. URL
https: /ttts::/api.... http:/pp..eeaaticcccoooaaa...... ott.spo.o.....o.oooo.pppsssssss.
18
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607.0644 2016. URL https: / api semant commaticscholare. ort 8236317. 4
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4598-4602, 2023,
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In European Conference Computer Vision (ECCV), 2014. URL /ettps:/aapte semant www.annncccchlarr.....
ptgg/CorpusID....o : 12726540. 18
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 33:1877-1901, 2020.
Shi Chen and Qi Zhao. Shallowing deep networks: Layer- pruning based feature representa- tions. IEEE transactions on pattern analysis and machine intelligence, 41((2):3048-33 -0566 2018. 2, 4
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv, abss1504.00025, 2015. URL https: /api......// semant iescholaccrr .rrg org/CorpusII: 2210455. 18
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction cantucttio-finetuneed language models. ArXiv, abs/2210. 1416, 2022. URL https:/ /app........ semant ieenttisschhi eschol Crgg/Corpuss::::000 8554, 3,
10

---

Published as a conference paper at ICLR 2024
Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1606-3613, 2013. URL https::////. www.iii.ccatticllaa.a..... wtt.hhoo.a...roooouuuut.... 4309276. 18
Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the variational information bottleneck. In Proceedings of the International Conference on Machine Learning (ICML), 2018. 1,
Anindya Bijoy Das and Aditya Ramamoorthy. Coded sparse matrix computation schemes that leverage partial stragglers. IEEE Transactions on Information Theory, 6866):456-4181, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierar- chical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009. URL httph::t/aaaaac wwwanticscheeeeeeee.........e.ee http://ooooooa...ooooooooaaaaaaa 18
org/ http:/oorputD::5245333
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- tion for Computational Linguistics, 2019. URL https: api. semanticscholar: org/ CorpusID:5296777
6
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /9929, 2020. 3,
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1,2
Hongliang Fei, Tan Yu, and Ping Li. Cross-lingual cross-modal pretraining for multimodal retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition (CVPR) Workshop, 2004. 18
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. ArXiv, abs/2301.00771 2023. URL https://apg semanticschoiar: org/ Corpus 025553777440 1. 2, 3,4
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398- 414, 2016. URL https: //tpi/appp/./////// semanticscholas
org/Cor Por/CorppsID 8081284 18
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-254, 2016.
Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12:2217-2201 2017. URL
https: /apii wwmanteniioo ies.annncccollar org/CorpusII :1::::::1110000 11810 819992.... 18
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representatiinns (ICLR), abs/2009.00330, 2021. URL https: //api www.nticsccoooooooeoooooooo beemaattissccoooaa..ooo//rroopps atw.aaarorrooo/oostt:/0//
6475. 10,

---

Published as a conference paper at ICLR 2024
Geoffrey E. Hinton.
The forward-forward algorithm: Some preliminary investigations.
ArXiv, abs/2212.13345, 2022. URL https::::/// /taps:///ppii http://aa...mmatscccc.o.ooo semanticscholar wttpa/ooog/oooooooo/eo 254537921. 2,6
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6693-6702, 2019. URL https: / /api. semanticscholit org/
CorpusID: 1522822652222 18
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015, PMLR. URL https: pettps://proceeedte /ptpceeeeeee.g.... ml m.lr.prss// v /tioffffllllll ht.l.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2014. URL httpt:://////....m /appi:////pp..////// www.atettiiioc. wichoola...co.........o.... org org/CorpusID: : 85177077. 18
Jin-Hwa Kim, Sang-Way Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung- Woo Ha, and Byoung-Tak Zhang, Multimodal residual learning for visual qa. Advances in neural information processing systems, 29, 2016. 3
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Jonathan Krause, Michael Stark, Deng, and Fei-Fei. object representations for fine- fine-grained categorization. 2013 IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 554 561, 2013. URL https: / appi. reeanticscchlle orwpaanttsscccoooooooooooooeeee org/CorpusID: 143422277.
18
Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101-2411 16, 2022. 2
Ivan Lazarevich, Alexander Kozlov, and Nikita Malinin. Post-training deep neural network pruming via layer-wise calibration. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS), 1989. URL https://appt /api....... semanticschol.s org/CorpusID 7785881.
Jaewoong Lee, Sangwon Jang, Jachyeong Jo, Jaehong Yoon, Yunji Kim, Jin-Hwa Kim, Jung-Woo Ha, and Sung Ju Hwang. Text-conditioned sampling framework for text-to-image generation with masked generative models. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.0377 2023a. 1
Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C.H. Hoi. LAVIS:
A one-stop library for language-vising vision intelligence. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 31- 41, Toronto, Canada, July 2023b. Association for Computational Linguistics. URL https: // //aclanthology wwwpoog..o...oggoooooo 2223 acl - demo. 3. 10
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems, 2018. 17
12

---

Published as a conference paper at ICLR 2024
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems (NeurlPS), 2021. 2021.
ascholas URL https:t/////... //api. - demanticscholaan ar org/ wwwg//oorrpeeooooooooooooooeeeee pusssssrruuuup ID: : 2.36034
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference Machine Learning (ICMI.), 2022. 3,
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language- pre-training with frozen image encoders and large language models. ArXiv, abs/2301, 12597, 2023c. URL https::pp//. / //ps:///pppp /api. semant cscholar ar org/ http//rrpusspa 2.53 53559555099 1,3,
Tao Liang, Guosheng Lin, Mingyang Wan, Tianrui Li, Guojun Ma, and Fengmao Lv. Expanding large pre-trained unimodal models with multimodal information injection for image-text multimodal classification. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2022a.
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800, 2022b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.c 08485, 2023. 1,3
27X45:2304008485,
Yu Liu, Li Liu, Yanming Guo, and Michael Lew. Learning visual and textual representations for multimodal matching and classification. Pattern Recognition, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the International Conference on Computer Vision (ICCV), 2021.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 1627, 2023.
Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aireraft. ArXiv, abs/1306.5151, 2013. URL https://api.
semant semanticscholass org/ httgg//opuslliia ID: 2118703. 18
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alexandru Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. ArXiv, abs/2305.1733. 2023. URL https: / api www.nticscooolllatt..cocccccccaa org :tt.aacchh......ooooorooaaaaa:aa :
2,
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3190-3199, 2019. URL https:tt////e //api. semanti cscholas cemanttcschoaas ar org/ wwt./oorgg.as...eo Corpus ID: 17399173. 18
JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of a bert-based question answering model. arXiv preprint arXiv: 1910vvv190.000 191009660000001 2019. 2
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ArXiv, (bs/1609.078433 2016. URL https: /tttps:/appe http:/pssssatttttooooaaaaa..ao.m CorpusID: 6299141.1 18
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh. Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.0837 08378. 2021.
13

---

Published as a conference paper at ICLR 2024
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
NauralMagic. deepsparse.
Deepsparse.
2022.
URL
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729, 2008. URL https:ttp///om /api...... or 1ttr://oopssI::1111000 930013.
18
Geon Park, Jaehong Yoon, Haiyang Zhang, Xing Zhang, Sung Ju Hwang, and Yonina Eldar. Bitat: Neural network binarization with task-dependent aggregated transformation. ECCV Workshop on Computational Aspects of Deep Learning (CADL), 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3498-3505, 2012. URL httpt::t///........ /taps://appp /app.ppp...//// semanti www.ttccccclllacccc ar. org/ Corpus 383200. 18
Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo. J. Hockenmaier, and
Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74 93, 2015. URL
OrggCorppsID::.. 6941275.. 18
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021. 2021.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21:140:1-140:67, 2019. URL https:tpp///... /api. semant seemaniicscoollr
org/ 20083800 18
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429, 2023.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P1 1238. URL https: htt////aaaaattttoggo/e/ wrg. 1a18 1238. 18
Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Upop: Unified and progressive pruning for compressing lision-language transformers. In Proceedings of the International Conference on Machine Learning (ICML), 2023. 2,
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint ar0iv:2209 14792, 2022.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximations for model compression. Neural Information Processing Systems ((NeurIPS), abs/2004. 14340, 2020. URL https: //api semant csscol ar. www./oorroooooooooo TD: 21 664169665 4
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, 121/0222, 2012. URL https: : semanti (schol ar. wrw.orr.ooooooatt//// 97134, 18
14

---

Published as a conference paper at ICLR 2024
Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. Anmual Meeting of the Association for Computational Linguistics (ACL), abs/ 1.00491, 2019. httpt: http://ap..emaaiiiieeseaee www.nticschoteee..... org/ CorpusID: 53178817788 7. 18
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023a. 2, 3, 10
Quan Sun, Yuxin Fang, Ledell Yu Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved
training techniques for clip at scale. ArXiv, abs/2303.15389 2023b. URL https: //api.
semanticscholle .rg. wwt./orrgossooooooooooo Corp/orrusss ID: 2277766677777
Hidenori Tanaka, Daniel Kunin, Daniel I.. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems (NeurIPS), abs/2006. 2020. URL /te:://aapp /tpppaap...../ semanticschool...
org/Corpus 21 2195588211111 9,9
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 133771, 2023. URL https:/ sttppa//aap...maaii.cc...o semanticscholar .rg/
IorpusID: 2tt.:::::57299000000 6
Maria Tsimpoukelli. Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07776, 2020.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint
2019. 2
Siyuan Wei, Tianzu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. Proceedings of the International Conference on Machine Learning (ICML., 2023.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485-3492, 2010. htthttps: /api...... www.nticscho...o....
org/CCrppsiii 1309931. 18
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.144 14178, 2023.
Jaehong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural networks. In Proceedings of the International Conference Machine Learning (ICML). 2017.
Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and
Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In Proceedings of the International Conference on Machine Learning (ICML), 2022a.
Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao. Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruming via bi-level optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022b.
15

---

Published as a conference paper at ICLR 2024
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-
language models. International Journal of Computer Vision, 130:2337 2348, 2021. URL https://app semant
org/ Corpus ID :277386023, 10
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 2022a. 3
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-image generation. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2022b. 3
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint orXiv:2304.10500 2023. 1
16

---

Published as a conference paper at ICLR 2024
Algorithm 1: Efficient Coarse-to-Fine Layer-wise Pruning
Input:
weights of the vision model W weights of the language model W calibration dataset D, target sparsity P. maximum sparsity per layer Pmox
Output: The pruned weights W and W
// Coazse sLep: deLermine the spazalty zatios vis global information
s < S(W", W D,C):
// get the gicbal importence score // the sparsity Equationn 3
p + geaSparityFromSenees W W -P.Ppmae P
// Fine step: perform layer-wi se pruning
W W
0.0
17 we initialize the 6 weight to be the idenzity matrix, depicting the fact that the input of the first layer is the raw input.
We +
for i = M do
W; arg max W,
prune vision weights layer by layer // prune language weights layer by layer
W W + {W;)
W f W
for 1 do
W. arg max S(W) WI1,, P.:
10
W w" (W:
12 return W W
0.60
first-order grad
0.55
sedde grad
Sparsity
0.45
0.40
0.35
0.30
Separation larguage model
100
200
300
400
500
600
Layer index
Figure The loss landscape of the BLIP-2 model.
Figure 6: Sparsity ratios obtained by ECoFLaP.
A THE LOSS LANDSCAPE
We present visualization of the loss landscape (Li et al., 2018) for BLIP-2 in Figure 5, where achieve this by perturbing the pre-trained BLIP-2 model 2500 times to approximate the parameter space and using 256 samples from CC3M compute the loss. Our visualization reveals distinctive corn-shaped loss landscape, which characterized by a single basin centered around the pre-trained weight of the BLIP-2. Moreover, the landscape surprisingly smooth with linear slope around the pre-trained weight. This smooth and corn-shaped loss landscape explains why we can estimate the gradient with zeroth-order optimization this successfully, we avoid sampling weights situated in other basins that could lead to inaccurate gradient estimation.
B RESULTS ON LLAMA
We further assess approach's 's performance using perplexity the WikiText dataset with LLaMA, a prominent open-source LLM. The results on the 7B model presented in Table 7 demonstrate that ECoFLaP achieves noteworthy 10.6% relative improvement compared Wanda when operating
17

---

Published as a conference paper at ICLR 2024
Table 7: Evaluation of Wanda and ECoFLaP on LLaMA 7B 0.6 sparsity by perplexity on WikiText.
LLaMA @ 0.6 sparsity
Perplexity
Full Model Wanda
7.26
10.68
ECoFLaP
first-order gradient
10.16
zeroth-order gradient
9.83
at 0.6 sparsity. Notably, in this experiment, find that employing larger value in Equation substantially enhances performance. This observation suggests that the zeroth-order estimation can benefit from exploring regions further from the target weight, capability not inherent in gradients obtained through backpropagation.
C EXPERIMENTAL DETAILS
Evaluation Datasets and Metrics. We evaluate the zero-shot ability of BLIP-2 on various datasets after pruning. We VQAv2 (Goyal et al., 2016), OK-VQA (Marino et 2019), and GQA (Hudson & Manning, 2019) for visual question answering, NoCaps (Agrawal et al., 2019) for image captioning, and Flickr30k (Plummer al., 2015) for image-text retrieval. For BLIP, we evaluate the performance change of the BLIP, fine-tuned on NLVR2 (Suhr et al., 2019) and COCO captions (Chen et al., 2015). For unimodal models, we evaluate FlanT5 on 57 tasks in MMLU (Hendrycks et., 2021), evaluate LLaMA 7B with WikiText (Merity et al., 2016), and evaluate EVA-Vi WiT on image classification task with ImageNet- 1k (Deng et al., 2009). We report accuracy for question-answering datasets, NLVR2, ImageNet-1K, 1k, and tasks in MMLU. Next, we use CIDEr and SPICE to evaluate image captioning tasks and use TR (top-1 text recall) and IR @ (top-1 image recall) for image retrieval tasks. Lastly, report the model's perplexity of WikiText in evaluating the LLaMA.
Calibration Datasets. Our approach utilizes a small subset (128 samples) of a dataset as the calibration data from CC3M (Sharma al., 2018), ImageNet (Deng et al., 2009) and C4 (Raffel et al., 2019) for calibrating BLIP-2, unimodal EVA-ViL VIT, and unimodal FlanT5 (and LLaMA). We use NLVR² and COCO captions as the calibration data for fine-tuning the BLIP backbone.
Dataset splits. For VQAv2, OK-VQA, and GQA, we use val, test, and test-dev split, respectively. We use the validation set for NoCaps and the test set for Flickr30k. In our BLIP experiments, report results on both val and test set for NLVR², while we use the Karpathy test split (Karpathy & Fei-Fei, 2014) for COCO captions. In our unimodal experiments, we use the publicly available test set for MMLU, and we use the validation set for ImageNet-Ik, Ik. For WikiText, we report the perplexity on the validation set.
Datasets and Metrics for CLIP experiments. We use a diverse set of 11 image classification datasets evaluate the model's ability to recognize generic and specific objects: Caltech101 (Fei-Fei et al., 2004), DTD (Cimpoi et al., 2013). EuroSAT (Helber et al., 2017). FGVCAircraft (Maji et al., 2013), Food101 (Bossard et al., 2014), ImageNet (Deng et al., 2009), OxfordFlowers (Nilsback Zisserman, 2008), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), SUN397 (Xiao et al., 2010), UCF101 (Soomro al., 2012). We report accuracy for each dataset Table 2.
Seed. The default seed option for each codebase is used for evaluation: seed 42 for all the experiments for BLIP-2, FlanT5, and EVA- VIT, seed 0 for the LLaMA experiment, and seed for the CLIP experiment.
Hyperparameters. For the coarse step, we set PM = p 0.1, [D] = 32, and the number of noises to be for ECoFLaP with zeroth-order gradient. is set to le except for LLaMA, where find le-1 works better, but we almost did not tune this hyperparameter. For ECoFLaP with first-order
gradient, use [D] 128.
Resources. All the experiments are done with one 40GB A100 or one 48GB A6000, except for ECoFLaP with first-order gradient on LLaMA we 2x 48GB A6000.
18

---

Published as a conference paper at ICLR 2024
Table 8: The comparison of different methods terms of their category and used importance measure.
Methods
Global or Layer-wise
Importance Mcasure
Global Magnitude Pruning
Global
Wiles
Gradient-basd based Pruning
Global
Wilk
V Gllij
SparseGPT
Layer-wise
WT/ XX" + XI) +
/diag
Wanda
Layer-wise
W ig
|X,||
ECoFLaP
First Stage:
Vw; C
Mixed
w/ Wanda and zeroth-order
Second Stage:
[aa
|X,||
Table Comparison of SparseGPT and ECoFLaP (with SparseGPT) on the zero-shot performance with BLIP-2 at 0.6 sparsity. We report accuracy for visual question answering, CIDEr and SPICE for image captioning, and @R@1 (text recall) and IR@ @ (image recall) for image retrieval. We also
compute the macro average score over tasks and the GPU memory usage for each method.
Method's Mem. Usage
Visual Question Answering
Image Captioning
Image retricval
Macro Avg.
Method
Sparsity
VQAv2
OK-VQA
GQA
NoCaps
Flickr30k TR@1
(GB)
Accuracy
CIDEr
SPICE
IR@1
SparseGPT 60%
9.04
48.7
27.9
35.1
101.0
13.4
95.2
85.0
51.8
ECoFLaP
SparseGPT
60%
9.05
50.4
27.9
36.3
105.3
14.0
95.3
86.1
53.1
D ECOFLAP ON SPARSEGPT
As the key contribution of our method for efficient estimation of layer-adaptive pruning rate for layer-wise pruning is orthogonal to SparseGPT, ECoFLaP can also be combined with SparseGPT. As shown in Table 9, EcoFLaP (w/ SparseGPT) consistently achieves superior performance over multiple VL tasks.
Additionally, the performance improvement seems relatively smaller compared to ECoFLaP with Wanda (i.e., original EcoFLaP). We hypothesize that this is because the reweighting mechanism in SparseGPT is somewhat correlated to dynamic sparsity ratios, where both of them are important to preserve the performance in high sparsity ratios.
E QUALITATIVE COMPARISON OF THE MODEL BEFORE AND AFTER PRUNING
We conducted an analysis comparing the predictions of the Full model (before pruning) and the model pruned by ECoFlaP (with 0.5 sparsity) the GQA datasets, and presented three illustrative examples in Table 10. We found one interesting observation: out of 12578 questions, the Full model correctly answered 1219 questions that our model did not, yet our model also accurately responded to 1072 questions where the Full model failed to respond (there are 4322 questions that both models answer correctly). This indicates that pruning does always lead to performance degradation; in some instances. it may even enhance model robustness for certain questions. During our review of incorrect predictions by either our model the Full model, we noticed that most were conceptually similar to the ground truth. This suggests that the pruned model still retains most of the capability. Furthermore, in some cases, such as with the "horses running' example, both models provided answers that might be considered correct by many humans, even though they differ from the ground truth.
19

---

Published as a conference paper at ICLR 2024
Table 10: Qualitative examples to compare the predictions generated by the Full Model and the ECoFLaP-prunce model. We pick three example muage-question pairs from GQA datasets. We observe the model after pruming does not always perform worse than the Full Model.
Images
Question
7401
20964557
20602949
Questions
What shape the microaave the stove below? Which vehicle parked on street
What is home running across?
Ground Truth
reclangular
car
ground
Fall Model's Prediction
hill
Lap-ee Prediction
roctongolas
Neyele
20


Reviews:
[{"review_id": "aKsGXI4Iu4", "reviewer": "Reviewer_4r21", "summary": "This paper presents the Efficient Coarse-to-Fine Layer-Wise Pruning (ECoFLaP) method for Large Vision-Language Models. The method utilizes zeroth-gradient optimization to determine layer-specific sparsity ratios and applies layer-wise unstructured weight pruning. Experimental results demonstrate improvements on some benchmarks.", "questions": "Please refer to my detailed comments in the weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-- Pruning is a crucial topic for efficient large models.\n\n-- The proposed method is interesting to me.\n\n-- Extensive experimental results demonstrate some improvement compared to previous methods.", "weaknesses": "Overall, the paper leans more towards an engineering-focused approach. The proposed method appears to be a straightforward combination of existing techniques without offering mathematical insights or clear motivations. The paper requires significant revisions to improve its writing quality.\n\n-- The paper claims that pruning multi-modal large models differs from other large models, but lacks mathematical or experimental support for this assertion.\n\n-- The use of layer-wise pruning due to the computational cost of calculating the inverse of the Hessian is common in other large model pruning approaches.\n\n-- The assumption of having sufficient GPU resources in the pruning scenario is not adequately discussed.\n\n-- While zeroth-gradient optimization is effective for efficient fine-tuning of large models [1], its motivation and suitability in a pruning scenario, considering challenges like slow convergence and sensitive hyper-parameters, remain unclear.\n\n-- The use of calibration datasets in experiments raises concerns about fairness and the significance of employing zeroth-order optimization to save memory if calibration datasets are used for fine-tuning.\n\nThe paper suffers from poor writing quality and contains noticeable typos:\n\n1. that finds adaptive sparsity per layer by leveraging the global importance score approximated via first-order gradients. (should be zeroth-order gradients)\n\n2. Note that our proposed method is computationally efficient by leveraging the first-order gradient to obtain a global importance score without Hessian operations (should be zeroth-order gradients)\n\n[1] Malladi, Sadhika, et al. \"Fine-Tuning Language Models with Just Forward Passes.\" arXiv preprint arXiv:2305.17333 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zShClMcNmh", "reviewer": "Reviewer_GD72", "summary": "The paper addresses the challenges of deploying Large Vision-Language Models (LVLMs) due to their high computational and energy costs. Traditional global pruning methods are costly, and recent layer-wise pruning approaches lack a global perspective. To overcome this, the paper introduces Efficient Coarse-to-Fine Layer-Wise Pruning (ECoFLaP), a two-stage approach that leverages global importance scores for determining sparsity ratios and then performs efficient layer-wise weight pruning. The computation of global importance scores is achieved using the zeroth-order approximate gradient through a forward-forward algorithm.   ECoFLaP demonstrates significant performance improvements in both multimodal and single-modal models, particularly in high-sparsity scenarios. Notably, it achieves these improvements while using only 40% of GPU memory compared to backpropagation and maintaining competitive accuracy.", "questions": "- The sparsity ratios in the experiments cover a range from 0.1 to 0.6. Is it possible to extend this range to include a wider spectrum of sparsity ratios?\n- Can this method be combined with SparseGPT?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Global pruning indeed faces challenges in sparsity allocation, particularly for multi-modal models. The paper provides a thorough and persuasive analysis of the motivation behind this issue.\n+ The method presented in the paper is straightforward yet proven to be effective, and the experiments conducted are comprehensive.\n+ The manuscript is readily understandable and offers sufficient experimental details for reproducibility.", "weaknesses": "- There is a need for a more detailed description of Algorithm 1.\n- Several metrics are used for importance scores, including weight magnitude, the multiplication of gradient and weight magnitude, gradient only, and sensitivity analysis. It would be beneficial to provide a comprehensive comparison of these metrics.\n- There are some typos in this paper: (1) In Section 4.1, \"and then convert the scores to sparsity by three steps\" should be \"and then convert the scores to sparsity by four steps\"? (2) In Section 4.2, \"like less than 5B parameters\" should be \"Like less than 5B parameters\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4wj45bxNIi", "reviewer": "Reviewer_5Mvp", "summary": "This paper aims to develop an efficient pruning methods for BLIP like multimodal models. Unlike previous layer-wise pruning methods, this paper proposed a global importance score which can be efficiently approximated using the global model gradients. Layer-wise weight pruning then was applied on the multimodal model. The proposed methods were compared with recently developed pruning methods and consistently improve upon existing methods on accuracy with the same level of sparsity.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper is easy to consume and logically smooth.\n2. The scope of this paper is well defined, which is to prune Blip-like multimodal architecture. Challenges of pruning Blip-like models were presented clearly.\n3. The key idea of using zeroth-order approximated gradient makes computing the global important score efficient, which is useful.", "weaknesses": "1. In addition to the numerical comparison, can you show some example results that compare the before and after results? \n2. While it is not easy to measure the real performance improvement, can you discuss about it with the proposed pruning method?\n3. Based on the importance score, what are important layers? What's the distribution of scores across all layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZInbnkNNVf", "reviewer": "Reviewer_q2nJ", "summary": "This article primarily addresses the task of model compression for Large Vision-\nLanguage Models (LVLMs). Traditional iterative global pruning methods involve\ncomputationally expensive operations, while layer-wise pruning approaches lack a\nglobal perspective, potentially resulting in suboptimal performance after pruning. This\npaper proposes a two-stage coarse-to-fine weight pruning approach for LVLMs, which\nutilizes the global importance score for unstructured pruning.", "questions": "See the weakness parts.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces a layer-wise pruning method for LVLMs that utilizes\nglobal importance scores. The global importance score is obtained by\napproximating the first-order gradients of the model parameters. By\nconsidering the global perspective, this approach aims to effectively identify\nand prune less important weights in each layer of the LVLM model.\n2. This paper clearly highlights the challenges dealing multimodal compression\ncompared to single-modal model compression. The modularization of\nmultimodal models makes compression more challenging, and the paper\nprovides visualizations of gradients that effectively demonstrate the imbalance\nin magnitude and gradient distributions between vision and language models.\n3. Experimental results have shown that this method is effective across various\nbackbones and modal models.", "weaknesses": "1. The method in this paper is relatively simple, and novelty is insufficient.\n2. The paper's writing could be improved as it lacks a theoretical analysis in the\nmethod introduction section to explain the effectiveness of the proposed method.\nAdditionally, there is a scarcity of formulas in the paper, and many details are\nnot adequately clarified.\n3. The experiments conducted were not comprehensive enough.\na) The validation of the models in the experiments was also insufficient. Only\nthe compression effects on the BLIPs model were tested, and the widelyused\nCLIP model, which is a mainstream multimodal model, was not\nevaluated. Furthermore, the comparison with the UPop model was only\nconducted on the NLVR2 and COCO Caption datasets, without\ncomparisons on other datasets such as Flickr30k and VQA2.0.\nb) The ablation study experiments were somewhat simplistic and did not\nsubstantiate the fundamental reasons for the effectiveness of the proposed\nmethod. It would be beneficial to enhance the content of the ablation study\nexperiments to provide a more detailed analysis and strengthen the overall\ncredibility of the article.\n4. Some minor mistakes:\nFor example, “and then convert the scores to sparsity by three steps: (1)\nCompute the total parameters that need to be selected based on p, (2) Normalize\nthe scores, (3) Compute the parameters that should be picked for each layer, (4)\nObtain the sparsity for each layer based on the number of parameters to be\npicked and the parameters of this layer. ”\nThe phrase \"three steps\" in this sentence can also be changed to \"four steps.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)