Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Authors: Yaoqin He
Junchen Fu
Kaiwen Zheng
Songpei Xu
Fuhai Chen
Jie Li
Joemon M. Jose
Xuri Ge
Venue: ICML 2025 poster (2025)
Keywords: Efficient  Pre-trained Vision-Language Model
Patch Redundancy Filter
Architecture Redundancy Filter
Primary area: deep_learning->other_representation_learning
TL;DR: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch and Layer Filtering

Abstract: In this paper, we present a novel approach, termed Double-Filter,to “slim down” the fine-tuning process of vision-language pre-trained (VLP) models via filtering redundancies in feature inputs and architectural components. We enhance the fine-tuning process using two approaches. First, we develop a new patch selection method incorporating image patch filtering through background and foreground separation, followed by a refined patch selection process. Second, we design a genetic algorithm to eliminate redundant fine-grained architecture layers, improving the efficiency and effectiveness of the model. The former makes patch selection semantics more comprehensive, improving inference efficiency while ensuring semantic representation. The latter’s fine-grained layer filter removes architectural redundancy to the extent possible and mitigates the impact on performance. Experimental results demonstrate that the proposed Double-Filter achieves superior efficiency of model fine-tuning and maintains competitive performance compared with the advanced efficient fine-tuning methods on three downstream tasks, VQA, NLVR and Retrieval. In addition, it has been proven to be effective under METER and ViLT VLP models.

Double-Filter:
Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch &Layer Filtering
Yaoqin He"
Junchen Fu 1
Kaiwen Zheng' Songpei Xu
Fuhai Chen
Jie Li
Joemon M. Jose
Xuri Ge
Abstract
In this paper, we present a novel approach, termed Double-Filter, to "slim down" the fine-tuning process of vision-language pre-trained (VLP) models via filtering redundancies feature inputs and architectural components. We enhance the fine-tune tieettunig process using two approaches. First, develop patch selection method incorpo- rating image patch filtering through background and foreground separation, followed by a refined patch selection process. Second, we design a genetic algorithm to eliminate redundant fine- grained architecture layers, improving the effi- ciency and effectiveness of the model. The for- makes patch selection semantics more com- prehensive, improving inference efficiency while ensuring semantic representation. The latter's ss fine-grained layer filter removes architectural re- dundancy to the extent possible and mitigates the impact on performance. Experimental re- sults demonstrate that the proposed Double-Filter achieves superior efficiency of model fine-tuning and maintains competitive performance compared with the advanced efficient fine-tuning methods on three downstream tasks, VQA, NLVR and Re- trieval. In addition, it has been proven to effec- tive under METER and ViLT VLP models.
Original Image
VIT [CLS]
Yolo Detection
Our Patch Filler
Prediction
Prediction
Prediction
Prediction
Prodiction
Prodiction
Trandfomes Bacc
FFN
FEN
MICA
MHCA
X6
MHCA
XIREA
D75
MHSA
MHSA
VICC
I VIICA
- BERT
MISS
MESA
VIT
BERT
BER
Image
Text
Image
Text
Image
Text
Full Tuning
(b) Fine-Tuning
Our Fine-grained
(METER)
(DAS)
Architecture Layer Filter
Figure I. VLP models exhibit two primary types of redundancy: image patches and architectural layers. For the former, segment patches into foreground and background by YOLO detector, then filter them based their importance determined by a pre-trained ViT model. For the latter, fine-grained layer filtering overcomes architectural optimization bottlenecks, offering greater flexibility.
datasets (e.g., paired image-text datasets), these models achieve remarkable performance when fully fine-tuned for tasks such as visual question answering (VQA) (Zhou et al., 2020; Hong et al., 2021), natural language for visual reason- ing (NLVR) (Suhr et al., 2019) and cross-moda retrieval (Gao et al., 2024; Long et al., 2024), etc. However, fully fine-tuning these models for downstream tasks incurs sub- stantial computational costs, prompting researchers inves- tigate more efficient fine-tuning methods, such integrat- ing adapters replacing specific components within the model's 's backbone.
1. Introduction
Vision-Language Pre-trained (VLP) models (Du et al., 2022; Chen et al., 2020; Lu et al., 2019; Dou et al., 2022b) have established themselves as essential tools within the research community. Through large-scale pre-training on extensive
Mainstream efficient fine-tuning methods, such as adapter- based and other parameter-efffcien approaches (Houlsby et al., 2019; Hu et al., 2021; Karimi Mahabadi et al., 2021; Gao al., 2024; Fu al., 2024b), reduce the number of trainable parameters by incorporating adapter modules into the layers of the pre-trained backbone. Although these methods lower the training costs by freezing the backbone network, they are still inefficient in the inference process because they do not optimize the original architecture of the VLP model. Considering the architecture redundancies in the complex Vl.P- backbone and extra parameters
Equal contribution 'Shooo Computing Science, University of Glasgow, Glasgow, United Kingdom." "Scoool of Computer and Data Science, Fuzhou University. Fuzhou, China. 'School of Infor- matics, Xiamen University, Xiamen, China. "School of Artificial Intelligence, Shandong University, Jinan, China. Correspondence to: Xuri Ge <xurr. sdu.edu.en>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
of adapters, some recent works (Wu et al., 2024b; Kim et al., 2021; Wu et al., 2024a) focused on removing such redundancy in VLP models. For example, a simple block- skipping strategy (Wu et al., 2024b) reduces redundancy in VIP models by replacing complex blocks in the original backbone with adapters that minimally impact downstream tasks. These studies prove the existence of architectural re- dundaney in existing VLP models and alleviate the problem of inefficiency in the architecture level to a certain extent.
VLP models, and implemented in our fine--rainee Archi- lecture Layer Filter (ALF). Unlike traditional methods (Wu et al., 2024b) that replace entire transformer blocks, our approach targets specific sub-cou components, such as the Multi- Head Self-Attention layer (MHSA), Multi-Head Cross At- tention layer (MHCA), and Feed-Forward Network (FFN) layer. By selectively filtering these fine-grained sub-layers and substituting them with lightweight adapters, we effec- tively reduce architecture-level redundancy a more granu- lar level. Notably, our AGA evaluates the filtering effects of complex sub-layer combinations through multi-generation crossover inheritance and regulatory mutation, guided by a specially designed fitness function.
However, these basic, coarse-grained de-redundancy meth- ods, which reduce transformer blocks in VLP models, over- look the finer architectural details, such as the sub-layers within transformer blocks. Specifically, as shown in Fig- ure 1, when transformer block is identified as redundant (as in (Wu al., 2024b)), all its components (e.g., cross- attention, self-attention, and feed-forward network layers) are removed together, overlooking opportunities for selec- tive optimization. Furthermore, these studies overlook data- level redundancy. In VLP models, both text and image data are processed; however, the main bottleneck arises from image patches, as their quantity significantly exceeds the length of text in these tasks (see Section 3.3 for details). As shown in MAE (He al., 2022b), an image can be effec- tively reconstructed using just 25% of its patches, indicating that many patches are redundant and hinder model efficiency. However, directly applying the random patch selection strat- egy from (He et al., 2022b) risks omitting valuable image information, leading to significant information loss. (Liang et al., 2022) addresses this by using the ViT (Alexey, 2020) classification token to rank and select patches with the high- est scores, however, this approach mainly captures objects and often neglects contextual information essential for com- plex multimodal tasks.
Our contributions in this paper are as follows:
We propose a novel multi-level efficient fine-tuning strategy for VLP models, called Double-Filter, by re- moving both image patch redundancy and fine- fine-grained architecture layer redundancy.
We devise a new Image Patch Filter (IPF) that filters redundant patches using ViT attentions, applied sepa- rately to the foreground and background as detected by YOLO. This approach ensures both spatial and seman- tic integrity.
We design an efficient adjustable genetic algorithm- based (AGA) fine fine-grained Architecture Layer Filter (ALF) designed to minimize architectural layer redun- dancy in VLP models for downstream fine-tuning. ALF leverages multi-generation crossover inheritance and regulatory mutation to swiftly identify filtered model variants, eliminating redundant sublayers.
Experimental results show that by filtering over 60% of im- age patches and 12 model layers, our Double-Filter achieves comparable performance on METER model fine-turning compared to state-of-the putational costs by 21.18G in terms of FLOPs. In addition, Double-filler is equally effective in the ViLT model.
To address these two issues, we propose a novel Double- Filter to address both data-level and architecture-level re- dundancies, enabling efficient fine-tuning of VLP models.
PEFT methods, reducing com-
Novel Data-level Redundancy Removal: We introduce an Image Patch Filter (IPF) to eliminate extraneous image patches that do not impact contextual semantics. Specif- ically, using an offline YOLO detector (Redmon, 2016), we distinguish between foreground and background areas, where detected salient objects represent the foreground, and the remaining areas are considered background. Low- importance patches within both foreground and background regions are then filtered based on the attention scores of the classification ([CLS) token. As illustrated in Figure 1, our IPF effectively preserves global semantics across patches, rather than focusing solely on salient objects.
2. Related Work
Vision Language Pre-trained models Vision-language pre- training (VLP) (Chen et al., 2024; Du et al., 2022) has advanced through leveraging large-scale image-text pairs for tasks such as masked language modeling, masked image modeling, and Image-Text Matching. Most VLP models such as ViL-BERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) relied on separate encoders for visual and textual features (e.g.. BERT for text, Paster-RCNN for im- ages) and used independent Transformer branches for cross- modal interaction. Later models like VisualBERT (Li et al., 2019), VL-BERT (Su et al., 2019), and UNITER simpli- fied this by integrating both modalities into a single Trans- former network. METER (Dou et al., 2022b) takes this a
Novel Architecture-level Redundancy Removal: At the architectural level, we introduce novel adjustable genetic algorithm (AGA) to precisely identify and filter redundant fine- -re-ggained sub-layers within the transformer blocks of
2

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
YOLO Detector
Broodiar
VIT
Image
Image Embeddings
Text
Text Database
Database
Image Patch Filter
E Embeddings
Prediction
Preduction
First Generation
Second Generation
Prediction
Prodection
Fitness Ranking
Transfo Faoot
Laper
Fliness (C.)
Mutation 010
MEHCA
MICCA
Variants
rairing
:
(AGA
VIE
DLHSA
MUSA
Filness (6
Tinness U.
VI2
BERL
Individual
Adagtation
:
a
Individuals
MIHSA
BERI
Image
Text
Filness (Ca)
Fitzess (G4)
Vemeylee
Image
Text
Full VLP Model
ur Fine-grained
METER
Adj ddjustablee Genetic Algorithm (AGA)
Architecture Layer Filter
Figure 2. The proposed Double-Filter framework comprises two primary components: the Image Patch Filter (IPF) and the fine-grained Architecture Layer Filter (ALF). The IPF utilizes a YOLO detector to partition an image's foreground and background areas and then selects patches based on their [CLS] weights in each area. The Fine-grained ALF introduces new Adjustable Genetic Algorithm (AGA), which formulates layer filtering as genetic algorithm optimization problem to approximate optimal fine-grat fine-grained filtering across layers.
step further by fully embracing an end-to-end transformer- based design, exploring various combinations of vision en- coders (CLIP-Vit ViT (Radford et al., 2021)) and text encoders (RoBERT ga (Liu al., 2019). DeBERTa (He al., 2020)), while also optimizing multimodal fusion mechanisms (e.g.. merged attention vs. co-attention). Despite these improve- ments, many VIP models still exhibit redundancy during the finetuning at both model and data levels, leading to in- efficiencies. The approach proposed in this paper aims to deliver a universal and efficient fine-tune tuning solution that addresses these challenges across broad spectrum of VLP models, particularly those with complex cross-modal inter- actions.
it from prior research approaches.
3. The Proposed Approach: Double-Filter
Figure provides an overview of our Double-Filter method, applied to the widely used VLP model METER (Dou et al., 2022b). The method consists of two main components: the Image Patch Filter and the Fine-Grained Architecture Layer Filter. Double-Filter is designed to facilitate com- prehensive and efficient fine-tuning at both the patch and architectural levels in VLP models, particularly those with complex cross-modal interactions.
In Section 3.1, outline the image patch selection process, detailing the wrregroud-bbackrene on the attention scores for the [CLS] token from Vision Transformers (ViT) (Alexey, 2020). Section 3.2 discusses fine-grained architectural redundancy removal using our genetic algorithm, which prunes unnecessary layers while preserving model performance.
Efficient Finetuning Previous studies have largely focused on adapter-based methods (Gao et al., 2024; Sung et al., 2022; Zhang et al., 2022; Feng et al., 2025; Zanella & Ben Ayed, 2024; Wu et al., 2023; 2024a; Fu et al., 2024a) to enable efficient fine-tuning of Vision Languuge Pre-trained (VLP) models, aiming to lower the adaptation cost down- stream tasks. However, these approaches often introduce architectural redundancy due to the addition of new trainable modules, which can limit overall efficiency. To tackle these challenges, recent skipping-basee approaches like DAS (Wu et al., 2024b) have introduced parameter and computation efficient transfer learning (PCETL). However, this approach primarily targets structural redundancy by replacing entire transformer blocks at coarse level, overlooking erucial ef- ficiency bottleneck-pater redundancies in visual data and finer- grer-grained architectural layers. In contrast, the Double Filtering framework uniquely addresses both architectural and patch-level redundancies simultaneously, distinguishing
token filtering based
3.1. Image Patch Filter
In this section, we introduce the Image Patch Filter (IPF), method designed efficiently filter image patches by discarding less relevant regions while preserving vital back- ground details that offer essential contextual cues. The innovation in IPF lies in its approach to partitioning the entire image into foreground and background portions using a YOLO detector, which enhances the informativeness and efficiency of the patch selection process.
3

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Algorithm 1 Image Patch Filter
Algorithm 2 Fiin-GGraine Architecture Layer Filter
1: Input: YOLO detector D, Image Sparsity p
1: Input: Number of transformer blocks N, target number of layers L, number of elite chromosomes K, number of generations G. population size M, training set TS, validation set V.S, fitness score function, crossover and mutation operations.
2: Output: Retained patches TF
3: Extract classification score SCLS for 1;
4: Get bounding boxes {B1, B2,...; Ba) D(T)
5: Get foreground FG = U B,, Get background
2: Output: Optimized architecture C* with reduced num- ber of layers L.
BG = I - FG
3: Initialize M chromosomes Cm = [c] (m) (m) I.
6: Compute the space occupancy ratio using the areas of A(F))
where c!m² [0, for m 1 to M, repre- sent architectural choices. Denote the population as MC (C1, C2,..., CM).
FG and BG:
A[FG)+((BG)
7: Compute the total number of patches to retain, (Téel
[(1 - p) IT'I]
4: for g = 1 to G do
5: Sample random training and validation subsets: TS C TS, VS C VS.
8: Compute the number of patches for FG and BG:
|T = [c [T and (TP = |Tgp - -T
6:
for m = to M do
9: Select the top |Tpp patches for FG based on Scis!
7: Train C. TS.
T Top- [
(TF)):
8: Compute FSmm Fitness(Cmm TS, VS).
10: Select the top |T patches for BG based on scus!
9:
end for
FS; ΣØ= FS; for each C, E MC.
THe = Top- Top-iTpp
(TBG);
10:
Compute P.
11: Obtain the retained patches: Tp -TE U UTB::
11: Select K elite chromosomes with highest fitness scores: EC = Top- (MM).
12:
Get remaining chromosomes RCC = MC EC 13: Perform selection by sampling with a roulette wheel
3.1.1. PATCH IMPORTANCE
approach (Lipowski & Lipowska, 2012), choosing
VLP models typically represent an image as a sequence of patches (Alexey, 2020; Dou et al., 2022b:a). To enable effective patch filtering, we assign an importance score to each patch by leveraging the classification token ((CLS)) of the pre-trained ViT following (Liang et al., 2022).
RC'| parent chromosomes from RC to form the set
PC based on P: PC = {C1, C2...., Cn C, e RC}.
14: Perform crossover to create child chromosomes:
CRC = Crossover(PC).
15:
for each C. in CRC do
The ViT discerns and weighs the relevance of various image patches, primarily through attention scores assigned by the (CLS) capturing their relative importance for image classifi- cation tasks. This attention mechanism can be formulated as follows:
16:
if Zj=1 oj
(8)
+ Lthn
Perform mutation: C Mutation (C,). end if
17:
18:
19:
end for
20:
Update MC combining them with the elite chromo-
somes EC: MC = =CRC U JCC.
KT
SCLS = Softmax
Iclass
21:
end for
22: Return: Optimized architecture C*, which is the chro-
mosome in MC with the highest fitness score.
where Jelass is the query vector associated with the [CLS] token. Ki is the aggregated key matrices from all tokens. d is the dimensionality of the query vector. SCIS denotes the attention weights that modulate the contributions of different tokens based on their computed relevance. In the multi-head attention layer, multiple sets of attention scores are computed for the [CLS] token, each reflecting different facets of the image's context.
Specifically, for a given image 1, its foreground and back- ground are denoted as FG and BG. The patches corre- sponding to I, FG, and BG are denoted as T", FCF and TB respectively. YOLO Detector employed to gen- erate n bounding boxes {B,, B2,.... B₂} for the image, where each bounding box represents individual object. The foreground, FG. is defined as FG Uh= B, repre- senting the union of these bounding boxes. The background, BG, comprises the remaining portions of the image, as BG - I-II.... FG.
3.1.2. FOREGROUND AND BACKGROUND
While filtering patches based on their importance appears logical, relying solely on the attention scores for [CLS] token tends to prioritize object-specific information, poten- tially overlooking essential background details necessary for complex multimodal tasks. To address this, we pro- pose categorizing images into foreground and background portions.
The objective of IPF is to retain informative patches TyE with a sparsity P E (0, 1) compared to the original patches
'In practice, some patches overlap between FG and BG; We also include these patches to FG for simplicity.

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
T4. Formally, |Ttl [p- [T']], where (T) denotes the number of patches T, and [-] represents the ceiling oper- ation. Once the target patch count is determined, a space occupancy ratio is introduced to specify the number of foreground patches TF and background patches TP. to retain. The is calculated as the ratio of the area of the foreground regions to the area of the entire image. The number of retained foreground patches is calculated as [T [c. [Tell, while the number of retained back- ground patches is defined as | iT 1 Within each region, patches are ranked based on their importance scores, SCLS, and only those with the highest scores are retained according to the defined counts. The detailed pro- cedure for IPF processing provided in Algorithm 1.
The fitness function is crucial in a genetic algorithm, evaluating each individual's quality. In AGA, the fitness function is defined as:
Fitmess Function:
Fitness(C,TS, VS) = -
L(C, + +3,
where c((-)) denotes the loss on sampled datasets for given model, and a bias term 3 is defined as > 2/ max (f<(C,, TS) + C(C,, VS) | {1,2, .M} to ensure that the Fitness score remains positive for the following selection.
Crossover Operation: ates fresh individuals through single-point or two-point crossover methods. In the single-point crossover method, random crossover point k is selected, and genes beyond this point are swapped between two parent individuals, produc- ing two offspring individuals:
The crossover operation gener-
3.2. Fine-grained Architecture-Layer Filter
To mitigate redundancy in Transformer-based models for cross-modal tasks, we propose an Adjustable Genetic Al- gorithm (AGA) for fine-graing ain-ggrined architecture filtering, ap- proximating an optimal configuration 2. For fine-grained replacement, AGA decomposes Transformer blocks into detailed components, including Multi-Head Self-Attention (MHSA),
Cahlall = [[f ..... Ck E eall
E F
Cohiaaz= lof ..... of C &xl.
where E and F represent the two parent individuals.
Network (FFN), and in some cases, Multi-Head Cross-Attention (MHCA), collectively termed "architecture layers (AL)". AGA selectively substitutes these computationally intensive ALs with lightweight adapter blocks (Houlsby et al., 2019). The novelty of AGA lies in three main aspects: (1) fine-grained architectural replace- ments; (2) novel formulation of architecture filtering task as genetic algorithm; (3) We customize the design of muta- tion operation to satisfy the requirement while diversifying each generation effectively.
Mutation Operation: To cater to the replacement of L layers within the chromosome C', we propose a mutation operation for adjusting gene values. Firstly, we calculate ΣiN -- e; which gives the current count of genes with value 1. If Z + L, we then adjust as follows: if Z > L, randomly select 7 - T. genes with value of 1 and switch them to 0; if Z < L. select L - Z genes with a value of O and change them to 1. This mutation process ensures that the updated chromosome C" fulfills the condition of 3N 4 = L,, ensuring the desired L layers are replaced.
Z
Specifically, for VLP model with N Transformer blocks, where each layer contains MHSA, MHCA, and FFN modules, each individual is encoded chromosome C = [c1, c ... C.N) Here, each gene c, E {0, }} indicates whether the corresponding layer is replaced by a lightweight adapter: a value of 1 indicates replacement while 0 signífies retention of the original structure.
Problem Formulation:
The Double-Filter strategy combines an IPF strategy for image patch selection with an AGA for Fine-grained Architecture-Layer Filter, comprehensively enhancing the efficiency of fine-tuning for VLP models.
3.3. FLOPs Analysis
The AGA aims to identify the approximate optimal reduc- tion with T. filtered layers. AGA proceeds over G gener- ations, with M individuals in each generation. In each generation, individuals are evaluated on subsets TS and VS that are randomly sampled from the training set TS and vali- dation set VS, respectively. The most promising individuals from the final generation is selected as the final architecture. Similar to traditional genetic algorithms (Sampson, 1976; Mirjalili& & Mirjalili, 2019), we define our fitness function, crossover, and mutation operations as follows.
In this section, analyze the Floating Point Operations (FLOPs) (Li et al., 2020; Hobbhahn & Sevilla, 2021) re- quired by the transformer model. Using the image tower from the fusion layer of METER (Dou et al., 2022b) as an example, we theoretically estimate the potential FLOPs savings offered by the Double-Filter Framework compared to the vanilla model.
We denote the number of transformer blocks in METER (Dou et al., 2022b), the sequence length of the image, the sequence length of the text, and the hidden dimension as N. S, S, and D, respectively. Note that, in practical VLP models, S < S, and we consider a batch size of 1 for sim- plicity. Within a transformer block, the main computational
NNote that identifying an absolute optimal structure is NP-hard, rendering exhaustive search impractical.
5

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
costs arise from the Multi-Head Self-Attention (MHSA), Multi-Head Cross-Attention (MHCA), and Feed-Forward Network (FFN). Here, we focus our analysis on the FLOPs associated with these three components.
28NSD + 4NS to approximately (Spq 16po + 4r) 45. In practical computation, we also consider incorporating the FLOPs of YOLO; however, since its computational complexity is significantly smaller com- pared to the transformer model, we omit it from the theoret- ical analysis.
FLOPs in Multi-Head Self-Attention (MHSA): Each MHSA layer consists of three projection operations (6SSD)) Query, Key, and Value, followed by the computation of at- tention scores (2S² D). application of these scores (2S D).
4. Experiment
and subsequent linear projection of the concatenated heads (2SD2),² Therefore, the total FLOPs for MHSA can be ex- pressed as:
In this section, conduct experiments to evaluate the pro- posed Double-Filer for efficient fine-tuning of VLP models.
FLOPSMHSA 6SD2 252 + 25- D 28D2
4.1. Datasets & Metrics
= 8SD2 + D
We validated our approach three visual language tasks:
FLOPs in Multi-Head Cross-Attention (MHCA): The primary distinction between MHCA and MHSA lies in the sequence length of the Query, Key, and Value. The Query has the same sequence length as the image, while the Key and Value align with the sequence length of the text. Conse- quently, the FLOPs can be formulated as:
Visual Question Answering (VQA): We conducted exper- iments on the VQA2.0 dataset (Goyal et al., 2017), which transforms traditional open-ended natural language ques- tioning into a multi-classification task with 3,129 answer options. Following the foundational configurations of the METER (Dou et al., 2022b) and ViLT (Kim et al., 2021) models, trained the models on the training and validation sets and assessed performance through online evaluations of the test set.
FLOPSMHCA (2SD² + 45D2) + - - 2SD2
= 4SD + +35D2 + 4S$D
2 2
Natural Language for Visual Reasoning (NLVR): 2Shr al., 2019) dataset designed as binary classification determine whether text accurately describes the content of two images. According to the default settings of ViLT and METER, the images combined with the text into the model, followed by the other image combined with the same text. The final prediction is derived by concatenating the outputs from these two inputs.
We omit the term 45D2 + 4SSD since 5 < S, allowing for a clearer computation in the following steps.
first input of
FLOPs in Feed-Forward Network (FFN): A standard FFN has an expansion factor of 4, meaning the hidden layer size 4D. Consequently, the total FLOPs for the FFN is D
Total FLOPs for a transformer model: Therefore, the total FLOPs for N transformer blocks are:
Cross-modal Retrieval: We evaluate the cross-modal retrieval capability of our model on the Flickr30K dataset (Plummer et al., 2015), using the standard split introduced by (Karpathy & Fei-Fei, 2015; Ge et al., 2024). For initialization, we follow the same operation as DAS (Wu et al., 2024a), leveraging the pre-trained retrieval heads of METER and ViLT for similarity estimation. To improve the robustness of training, adopt hard-negative mining strategy where each positive sample is paired with 15 randomly sampled negatives per iteration.
FLOP = N (FLOPSMHSA + FLOPSMHCA + FLOPSFEN)
2 N ((8SD² + + D) + 4SD² + 16SD²)
D
FLOPs with Double-Filten Filter: Using the Double-Filter frame- work, we retain a fraction p the original patches (sequence length), 9 in the MHSA layers, and r in the MHCA, and in FFN, where r, r 1. The total FLOPs under the Double-Filter (DF) can then be calculated as:
FLOPspr = N (g + 4p + 4r 16poSD²)
Metric: We evaluate using accuracy for NLVR and VQA, and Recall@1 for the image retrieval (IR) and text retrieval (TR).
= (8pq + 16po + NS) + 4
This analysis demonstrates that the Double-Filee Frame- work can reduce the FI.OPs of a transformer model from
4.2. Implementation Details
'The FLOPs for matrix multiplication [m,, n] (n, p] is 2mmp (Ozaki et al., 201 l).
We conducted experiments on the METER (Dou et al., 2022b) and ViLT (Kim al., 2021) models. For the ME-
6

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Table Comparison between Double-Filter and other PEFT methods for widely used VLP modals, METER (Dou et al., 2022b) and ViLT (Kim et al., 2021), on VQA and NLVR tasks, as well as cross-modal retrieval task. The best performance is bold and the second best is underlined.
METER
Updated
VQA
NLVR
Retrieval
Method
Param.
test-dev
+FLOPs
test-P
+FLOPs
IR/TR R@1
+FLOPs
Full Tuning
323.31M
77.43
0.00
83.05
0.00
82.22/94.
0.00
Classifier Only
69.93
0.000
73.23
0.00
78. 78.80/89.00 .00
0.00
Shallow Prompt
0.30M
68.51
+28.71G
65.69
+26.84G
74,20/88.66
+28.71G
Deep Prompt
1.84M
70.78
+6.53G
72.64
+5.59G
84.84 .40
+6.53G
LoRA
0.29M
74.00
0.00
78.82
0.00
79 86/92.6926
0,00
Adapter
5.34M
74.70
+1.64G
79.93
+1.38G
80.38/91
+1.64G
Scaled PA
3.59M
75.11
+1.12G
80.38
+0.66G
80,40/93.20
+1.12G
DAS
5.34M
74.80
11.16G
80.11
-5.13G
80. 60.12/91
11.16G
Double-Filter
5.34M
74.25
-21.18
80.12
-12.51G
80.05/91.22
-21.18G
VILT
Updated
VQA
NLVR
Retricval
Method
Param.
test-dev
+FLOPs
test-P
+FLOPs
IR/TR R@1
+FLOPs
Full Tuning
115.43M
71.26
0.00
76.13
0.00
64.40/83.51
0.00
Classifier Only
63.75
0.00
56.08
0.00
57.42/78.00
0.00
Shallow Prompt
0.15M
66.47
+19.53G
66,47
+19.53G
55, 55.92/74.80
+19.53G
Deep Prompt
1.84M
69.30
+5.14G
73.34
+5.14G
58. 58.64/79.55
+5.14G
LoRA
0.15M
68,44
0.00
72.77
0.00
57.44/77.70 70
0.00
Adapter
3.56M
70.85
+0.86G
75.51
+0.86G
62.68/81.40
+0.86G
Scaled PA
1.80M
70.40
+0,44G
75.13
+0.44G
61.88/79
+0,44G
DAS
3.56M
69.28
-1.03G
74.89
-1.03G
60.66/80.80
-1.03G
Double-Fitter
3.56M
68.37
-4.72G
74.50
-4.72G
61.18/79.39
-4.72G
TER model, we focus filtering the most computationally intensive component: the fusion layer for METER and, the transformer encoder for ViLT. The adapter bottleneck for both networks was set to 96 following (Wu al., 2024b). We used YOLOv8 (Reis al., 2023) as the Yolo Detecter. Notably, compared with METER, ViLT does include the fusion network. So only consider de-redundancy of its encoders. Therefore, when calculating FLOPs cost in the later experiments, we only considered YOLO's FLOPs for METER, not ViLT. The FL.OPs of YOLO are about 4.5G. For images without any detected objects, we used the me- dian of CLS scores distinguish between foreground and background. To better align the IPF method with the ViLT model, we did not skip the first block's 's self-attention layer, and other settings remained consistent with the base settings of the METER and ViLT models. The other detailed set- tings for Double-Filter are placed in Appendix Table 1. All experiments were conducted on one RTX 3090 GPU.
Table 2. Efficient comparisons of different methods in terms of inference time (items/s) on two tasks based on different VLP models with inference time.
METER
Method
Full Tuning
Adapter
DAS
Double-Filter
VQA
55.32
$4.05
60.49
70.72
NLVR
46.50
45.73
51.47
55.13
ViLT
VQA
115.50
113.62
124.63
135.24
NLVR
54.49
53.86
55.80
58.36
Liang, 2021), Deep Prompt (Jia et al., 2022), LoRA (Hu
et al., 2021), Adapter (Sung et al., 2022), Scaled PA (He et al., 2022a), and DAS (Wu al., 2024b).
Detailed comparisons shown in Table 1. Existing PEFT methods generally use fewer updated parameters compared to full finetuning. However, except for our Double-File Filter, DAS, and LoRA, other methods tend to increase compu- tational complexity (e.g. FLOPs). While LoRA does not increase computational complexity, its performance lower than the other two methods. Specifically, compared meth- ods with lower inference complexity, our Double-Filter achieves the largest reduction, with average of more than twice the reduction compared to DAS (-21.18G FLOPs for VQA, -12.51G FLOPs for NLVR on METER). More- over, the performance degradation minimal, averaging only 0.30% on the METER network and 0.55% on ViLT net- work. We also compared full tuning, Adapter, DAS, and our
4.3. Efficiency-Performance Comparison
We evaluate our proposed Double-Filter based on two widely-used VLP models, i.c. METER (Dou et al., 2022b) and ViLT (Kim et al., 2021), on three benchmarks (NLVR² (Suhr al., 2019). VQA (Goyal al., 2017) and Flickr30K (Plummer al., 2015)) of different downstream tasks. We compare it on efficiency and effective fine-tuning with other popular PEFT methods, including Shallow Prompt (Li &

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Table 3. Pfficienyy-Performaacc halance of Image Patch Filter (IPF) for METER and ViLT VQA, NLVR tasks.
Table 4. Jfficienny-Peerfommman halance of fine-grained architec-
ture layer filter for METER and ViLT VQA and NLVR tasks.
Patch
VQA
NLVR
Method Filtering
VQA
NLVR +FLOPs
Method
Filtering
test-dev +FLOPs
test-P
+FLOPs
Layers
test-dev +FLOPs
test-P
METER
METER
Baseline
75.28
1.68G
81.28
0,99G
Baseline
75.28
1.71G
81.28
1.02G
40%
75.16
-8.97G
81.15
-4,9GG
74.92
-3.84G
80.60
-2.64G
50%
75.01
-14.81G
81.16
-6,59G
12
74.82
-8.43G
80.98
-7.57G
IPF
ALF
60%
74.67
-16.52G
81.32
-8.6G
18
74.23
-22.9G
80.06
-00.33G
70%
74.16
-20.24G
80.14
10.6G
73.61
-27.91G
80.25
-14.25G
VILT
ViLT
Baseline
0
70.13
0.73G
76.26
0.73G
Baseline
70.13
0.73G
76,26
0.73G
20%
69.32
-3.06G
75.47
-3.06G
69.15
-1.04G
75.50
-1.04G
IPF
30%
68.95
-5,09G
74.68
-5.99G
ALF
67.78
-3.06G
75.15
-3,59G
40%
68.36
-7.11G
74.60
-7.11G
6
66.63
-5.08G
73.88
-5.6G
iPFlours)
ALFiours)
random patch filtering (Random) and patch filtering based on CLS attention scores (EVIT (Liang et al., 2022)). The experiments performed the NLVR task on METER model. The results show that our IPF can better maintain performance when filtering patch redundancy. And this ad- vantage becomes greater as the patch filtering rate increases. At reasonable ratio, we can achieve the best efficiency and performance balance for VLP model fine-tuning.
EVIT
A1.5
DAS
Random
Random
NLVRR Test Accuracy
80.0
2000
Patch Filtering
Architecture Filtering
4.5. Efficiency-Performanne Balance of Architecture Layer Filter
Figure 3. Ablation studies of different image patch filtering meth- ods, i.e. Image Patch Filter (IPF) vs. EViT (Liang al., 2022) vs. Random Filter, and architecture filtering methods, i.c. fine- grained architecture layer filter (ALF) vs. DAS (Wu et al., 2024b) vs. Random Filter, for METER on NLVR task.
In this section, we conducted ablation experiments on our proposed ALF method by skipping different layers, and ob- served that the METER model exhibits significant amount of redundancy. For instance, even after removing 24 layers out of total of 36 layers in the NLVR task, it still performs well. In contrast. ViLT shows noticeable performance decline after removing just 4 layers. This discrepancy likely stems from the structural differences between the models: METER employs two separate encoders for processing tex- tual and visual information and includes cross-attention modules, whereas ViLT only utilizes word embeddings and image patch embeddings, and lacks cross-attention modules.
method regarding inference speed. In Table 2, our proposed Double-Filter improves the inference speed, demonstrating superior efficiency. Our proposed Double-Filte exhibits the lowest computational complexity among all existing PEFT methods, improving inference speed. Additionally, shows competitive results compared other PEFT methods.
4.4. -eficience-Perffrmmanee Balance of Image Patch Filter
To further validate the effectiveness of our ALF method, we compare it against DAS (Wu et al., 2024b) and a Random approach (which randomly removes an equivalent number of layers) in the NLVR task on the METER model. As shown in the right of Figure 3, compared to two other methods, our ALF effectively maintains performance stability even with more layers removed, demonstrating its advantage in balancing model reduction and performance retention.
We conduct extensive ablation studies to investigate the im- pact of different variants of the proposed image patch filter (IPF) performance and efficiency. We investigate the ef- fects of different patch filtering ratios and different filtering methods in Table 3 and Figure 3. Specifically, in VQA task of the METER model (Dou et al., 2022b), as the patch filter- ing ratio increases, although the accuracy decreases slightly, the FLOPs decrease significantly in Table 3. shows that IPF greatly helps model fine-tuning efficiency. Addi- tionally in the NLVR" task, reducing different patches can sometimes improve performance, suggesting that redundant patches may have a negative impact on performance.
4.6. Visualization
To more intuitively demonstrate our proposed Double-Filter method, we provide detailed visualizations of the results of Image Patch Filter (IPF) and fine-grained Architecture Layer Filter (ALF) in Figure 4. It shows that our IPF ef- fectively preserves key information images. Compared
In the left of Figure 3. to further validate the effectiveness of our proposed IPF, we conducted ablation studies comparing
8

---

Double-Filted Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Extensive experimental evaluations on three downstream tasks, VQA, NLVR and Cross-modal Retrieval, demon- strate that our Double-Filter method achieves a better bal- ance between efficiency and performance in fine-tuning various VLP models.
Limitation. Our Double-Filter leverages patch importance weights derived from Vision Transformers (ViT) and se- lected based on a pre-trained YOLO detector. While they are widely adopted, this reliance may limit flexibility. In future work, we plan to explore alternative strategies for determining patch importance and patch selection, such as offine lightweight pre-trained models. In addition, we will use a better genetic algorithm variant to search for the op- timal subnetwork structure to solve the problems of high computational complexity and easy convergence to local optimality during training.
Original Image
Random Filter
EVIT
Our IPF
VQA (METER)
Filtered Layer
Remained Layer
NLVR (METER)
Impact Statement
Figure 4. Visualizations of the proposed Double-Filter for METER (Dou et al., 2022h) on NLYR and VQA tasks, including the results of our proposed image patch filter and fine-grained architecture layer filter.
This paper presents work whose goal is to advance the field of Machine Learning and Deep Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
with EViT (Liang al., 2022) which only focuses on salient objects, our IPF can better focus on global contextual se- manties, including important objects in the foreground and background contextual contents. Additionally, by visualiz- ing filtered layer results of fine-grained ALF based on the VLP model METER (Dou et al., 2022b) for NLVR and VQA tasks, we note varying patterns in the layers omitted by the model across different tasks. The reason may be that the simple shorter texts in VQA allow it filter more fine-gratd tine-grained layers of the text tower compared to the long text with complex semantics in the NLVR task. Overall, visualizations clearly demonstrated the effectiveness of effi- cient filtering based on the proposed Double-Filter approach at both data- and network-level.
References
Alexey, D. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
Chen, S., Gu, J., Han, Z., Ma, Y., Torr, P., and Tresp, V. Benchmarking robustness of adaptation methods on pre- trained vision-language models. Advances in Neural Information Processing Systems, 36, 2024.
Chen, Y.-C., Li, L., Yu, L., EI Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Universal image-text repre- sentation learning. In European Conference Computer Vision, pp. 104-120. Springer, 2020.
5. Conclusion
Dou, Z.-Y,, Kamath, A., Gan, Z., Zhang, P., Wang. J., Li, L., Liu, Z., Liu, C., LeCun, Y., Peng, N., et al. Coarse-to-fine vision- laiiual pre-training with fusion in the backbone. Advances in Neural Information Processing Systems, 35: 32942-32956, 2022a.
In this paper, introduce novel Double-Filter method for efficient fine-tuning of VLP models, aimed at optimizing both data input and network architecture. For data input optimization, we developed new Image Patch Filter (IPF) that employs pre-trained ViT model to identify signifi- cant patches in the foreground and background, detected by YOLO. This effectively reduces data redundancy while preserving comprehensive information. For network archi- tecture optimization, we propose a fine-grained Architecture Layer Filter (ALF) based on an adjustable genetic algorithm (AGA). AGA efficiently eliminates redundant layers in the original VLP models through multi cellii-gnneraiinn crossover inheritance and mutation, enhancing the model's 's efficiency.
Dou, Z.- Xu, Y., Gan, Z., Wang. J., Wang. S., Wang, L., Zhu, C., Zhang, P., Yuan, L.,, Peng, N., et al. An pirical study of training end-to-end vision-and-languagg transformers. In Proceedings of the IEFE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 68166-18176, 2022b.
Du, Y.. Liu, Z., Li, J., and Zhao, W. X. A survey of vision-language pre-trained models. In Raedt, L. D. (ed.),

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Proceedings of the Thirty-First International Joint Con- ference on Artificial Intelligence, IJCAI-22, pp. 5436- 5443. International Joint Conferences on Artificial Intel- ligence Organization, 7 2022. doi: 10.24963/ijcai.22222 762. URL ht httpp::/// / tt..doooogggo..ooo...ooooo.oooo. 0ttg///1.249//iccca 2022/7 0022/7622 762. Survey Track.
Hobbhahn, M. and Sevilla, J. What's the backward-forward
flop ratio for neural networks?, 2021.
Hong. W., Ji, K., Liu, J., Wang, J., Chen, J., and Chu, W. Gilbert: Generative vision-language pre-training for image-text retrieval. In Proceedings of the 44th Interna- tional ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pp. 1379-1388 2021.
Feng. Y., Geng, Y.aa., Zhu, Y., Han, Z., Yu, X., Xue, K., Luo, H., Sun, M., Zhang, G., and Song, M. Pm-moe: Mixture of experts on private model parameters for personalized federated learning. In Proceedings of the ACM on Web Conference 2025, pp. 134 146, 2025.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nip. In International Conference on Machine Learning, pp. 2799. PMLR, 2019.
Fu, J., Ge, X., Xin, X., Karatzoglou, A., Arapakis, I., Wang, J., and Jose, J. M. lisan: Efficiently adapting multimodal representation for sequential recommendation with de- coupled peft. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 687-697, 2024a.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2021.
Fu, J., Yuan, F., Song, Y., Yuan, Z., Cheng, M., Cheng, S., Zhang, J., Wang, J., and Pan, Y. Exploring adapter-based transfer learning for recommender systems: Empirical studies and practical insights. In Proceedings of the 17th ACM international conference on web search and data mining, pp. 208 217, 2024b.
Jia, M., Tang, I.., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Karimi Mahabadi, R., Henderson, J., and Ruder, S. Com- pacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022-1035, 2021.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang. R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-59) 2024.
Karpathy, A. and Fei-Fei, L. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128 22-3-37, 2015.
Ge, X., Xu, S., Chen, F., Wang. J., Wang, G., An, S., and Jose, J. M. 3shnet: Boosting Image-sentent retrieval via visual comantic-spatinn self- oif-liighighhinng Information Processing & Management, (1): 2024.
Kim, W., Son, B., and Kim, I. Vilt: Vision-and- lannuuuuee transformer without convolution or region supervision. In International Conference on Machine Learning, pp. 5583-5594 PMLR, 2021.
Goyal, Y., Khot, T., Summers- Stay,, D., Batra, D., and Parikh, D. Making the v in vqa matter: Flevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 6904-6913, 2017.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
He. J., Zhou, C., Ma, X., Korg-Kirkpatrick, T., and Neubig, G. Towards unified view of parameter-efficient transfer learning. In International Conference Learning Rep- resentations, 2022a. URL https:/ http///pppeeeeeeee////ppp. met/foruma iidiiddi d ORdd-DDDDaxxxo
Li, L. H., Yatskar, M., Yin, D., Hsich, C.-J., and Chang, K. Visualbert: simple and performant baseline for vision and language. arXiv preprint arXiv: 1908.03557. 2019.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEFE/CVF Conference on Computer Vision and Patlern recognition, pp. 16000-16009, 2022b.
Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: experiences on accelerating data par- allel training. Proceedings of the VLDB Endowment, 13 (12):3005-3018, 2020.
He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0 03654, 2020.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. In Proceedings of the 59th
10

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Annual Meeting of the Association for Computational Lin- guisties and the I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
Sampson, J. R. Adaptation in natural and artificial systems
(john h. holland), 1976.
Su, W., Zhu, X., Cao, Y., Li. B., Lu, L., Wei, F., and Dai, J. VI-bert: Pre-training of generic visual-linguistic represen- tations. International Conference on Learning Represen- tations, 2019.
pp. 4582-4597, 2021.
Liang, Y. Ge, C., Tong, Z., Song, Y., Wang. J., and Xie, P. Not all patches are what you need: Expediting vision transformers via token reorganizations. International Conference on Learning Representations, 2022.
Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H.,, and Artzi, Y. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties. Association for Computational Linguistics. 2019.
Lipowski, A. and Lipowska, D. Roulette-wheel selection via stochastic acceptance. Physica A: Statistical Mechanics and its Applications, 391(6):2193 2196, 2012.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 2019.
Sung, Y.-L., Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision-and-language tasks, Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 5227-5237, 2022.
Long, Z., Ge, X., McCreadie, R., and Jose, J. M. Cfir: Fast and effective long-text to image retrieval for large corpora. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2188-218800 2198, 2024.
Tan, H. and Bansal, M. Lxmert: Learning cross-modality en- coder representations from transformers. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). Association for Computational Linguistics. 2019.
Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining aask-agnostic visiolinguistic representations for vision- and-language tasks. Advances Neural Information Processing Systems, 32, 2019.
Wu, Q., Huang, S., Zhou, Y., Dai, P., Shu, A., Jiang, G., and Ji, R. Approximated prompt tuning for vision-language pre-trained models. arXiv preprint arXiv:2306.15700 2023.
Mirjalili, S. and Mirjalili, S. Genetic algorithm. Evolution- ary algorithms and neural networks: theory and applica- tions, pp. 43-55, 23-55,2011 2019.
Ozaki, K., Ogita, T., and Oishi, S. Tight and efficient en- closure of matrix multiplication by using optimized blas. Numerical Linear Algebra with Applications, 18(2):237- 248, 2011.
Wu, Q. Ye. W. Zhou, Y., Sun, X., and Ji, R. Not all attention is needed: Parameter and computation efficient transfer learning for multi-modal large language models. arXiv preprint arXiv:2403. 15226, 2024a.
Plummer, B. A., Wang, L.., Cervantes, C. M., Caicedo, J. C.,, Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to- cesion-to-phraee correspondences for richer image-to-sentence models. Proceedings of the IEEE international conference on computer vision, pp. 2641- 2649, 2015.
Wu, Q., Yu, W., Zhou, Y., Huang, S., Sun, X., and Ji, R. Parameter and computation efficient transfer learning for Dision-language pre-trained models. Advances in Neural Information Processing Systems, 36, 2024b.
Zanella, M. and Ben Ayed, I. Low-rank few-shot adapta- tion of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1593-1603, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, H. Tip-adapter: Training-free adaption of clip for few-shot classification. In European Conference on Computer Vision, pp. 493-510. Springer, 2022.
Redmon, J. You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Zhou, I.,, Palangi, H., Zhang, L.,, Hu, H., Corso, J., and Gao, J. Unified vision-language pre-training for image cap- tioning and vqa. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 34, pp. 13041-13049, 2020.
Reis, D., Kupec, J., Hong, J., and Daoudi, A. Real- time flying object detection with yolov8. arXiv preprint arXiv:2305.0mm 00977, 2023.

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
A. Appendix
ALF configurations requirements of each task. For the VQA task, employed population size of 100, 20 iterations, and two-point crossover method. For the NLVR task, we used population size of 50, 10 iterations, and a single-poing crossover method. Loss calculations in the fitness function were based on 100 batches of training data (equivalent to epochs), and losses were recorded over 10 validation batches.
adjusted based on the specific
A.I. Hyperparameters and Details
In this section, we present the hyperparameters and imple- mentation details of the proposed Double-Filter method, which consists of main components: (1) Image Patch Filter (IPF) and (2) fine-grained Architecture Layer Filter (ALF).
After determining the optimal network structure, applied the filtered network structure, followed by 10 epochs of training in combination with the IPF approach. The first epoch served as a warm-up. and after completing all the epochs, the model's performance was evaluated on the test set. For the VQA task, evaluations were conducted via an online platform*.
A. IMAGE PATCH FILTERR (IPF):
For the METER model (Dou et al., 2022b), we assessed patch importance using the attention weights of the classifi- cation ([CLS]) token from the last layer of the pre-trained Vision Transformer (ViT) (Alexey, 2020). In the case of ViLT (Kim et al., 2021), all image patches were processed through the first self-attention layer, and the attention map of the first [CLS] token was extracted to evaluate patch im- portance. It is noteworthy that ViLT contains two [CLS] tokens: a text [CLS] token and a visual [CLS] token, the latter of which is used to separate modalities.
During training, only the parameters of the additional mod- ules, classifiers, pooler layers, modal-type embeddings, and adapters were updated; the rest of the network remained frozen. We adopted hyperparameters consistent with those in the original papers, and FLOPs calculations included YOLO detection ensure comprehensive and accurate re- sults.
A.1.2. ARCHITECTURE LAYER FILTER (ALF):
We focused on filtering the most computationally intensive components of each model. For METER (Dou et al., 2022b), this is the two-end fusion layer, and for ViLT (Kim et al., 2021), it is the transformer encoder.
All experiments were conducted on RTX 3090 GPU. During evaluation, inference time was measured with
Table Experimental Parameters
Common Parameters
Parameter
Value
Training epoch
10
Warm-up epoch
1
Learning rate
1 x 10-5
Adapter hidden dimension
96
Image Patch Filter (IPF)
Parameter
METER
ViLT
CLS tokens
Last layer of VIT
First layer of ViLT
Architecture Layer Filter (ALF)
Parameter
VQA Experiment
NLVR Experiment
Population size
100
50
Number of iterations
20
10
Crossover method
Two-point 100 batches over 2 epochs
Single-point
Training data per generation
Validation data per generation
10 batches
12

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
batch size of 64 for METER and 128 for ViLT. Additional details the Double-Filter method and hyperparameter settings are provided in Table 5.
found that only by combining the context of important ob- jects in the image and the background environment where they are located, the model can answer the questions more accurately. Compared to EVIT (Liang et al., 2022), which focuses only on salient objects, our IPF can hetter focus on global contextual semanties, including important objects in the foreground and background contextual content, while ef- fectively preserving key information in the image. This can effectively solve the corresponding text questions, especially those that require attention to background knowledge.
A.2. More Visualization
To more fully demonstrate proposed Image Patch Filter (IPF) of the proposed novel Double-Filter for VLP models, we provide more detailed visualizations of the IPF results in Figure and Figure 6 from the NLVR task and VQA task. respectively. Through specific text questions, we
'Since we use online evaluation to evaluate the effectiveness of the VQA task, cannot get explicit answer. Therefore, the training examples for visualization of our IPF, which do not affect effectiveness proof IPF because has no trainable parameters.
A.3. Memory Usage
We conducted comparative analysis of memory usage dur- ing inference across different fine-tuning methods, including our own model (Table 6). All evaluations were performed
Original Image
YOLO Detection
ViT [CLS]
Random Filter
EVIT
Our IPF
Question: A baby elephant is standing in the grass next to adult elephant without tusks. Answer: True.
Question: A bird files right above the water in the image on the right. Answer: True.
Question: convertible parking space overlooking beach. Answer: True.
Question: A black and white dog is standing in the grass looking at the camera. Answer: True.
Figure 5. Visualizations of the comparisons of different patch filters, including the Random Filter, EVIT (Liang et al., 2022) and our proposed Image Patch Filter (IPF) in Double-Filter for METER (Dou al., 2022b) on NLVR² test set for NLVR task.
13

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
with batch size of I to ensure consistency.
METER
Full tuning
Adapter
DAS
Double-Filter
VQA
3068M
3090M
2950M
2906M
NLVR
3030M
3050M
2916M
2884M
Table Comparison of memory usage tasks with different tuning methods.
14

---

Double-Filter: Efficient Fine-tuning of Pre-trained Vision-Language Models via Patch&Layer Filtering
Original Image
YOLO Detection
VIT [CLS]
Random Filter
EVIT
Our IPF
ALLIOO
Question: What is in the background? Answer: "trees", "sky".
Question: What is on the wall behind the girl? Answer: "photo", "picture", "doll", "painting".
Question: Are there power lines the image? Answer: "Yes".
Question: What's directly behind the bear? Answer: "tree", "trees", "rocks".
Figure Visualizations the comparisons of different patch filters, including the Random Filter, EViT (Liang al., 2022) and our proposed Image Patch Filter (IPF) in Double-Filted for METER (Dou al., 2022b) VQA task. Note that, the VQA online test set does not provide standard answers, so we use the training examples to demonstrate the effectiveness of our IPF. Since IPF does not contain training parameters, it does affect the effectiveness verification.
15


Reviews:
[{"review_id": "aiJyfUQfCn", "reviewer": "Reviewer_Qv8A", "summary": "The paper proposes Double-Filter, an efficient fine-tuning framework for vision-language pre-trained (VLP) models. It combines two redundancy reduction techniques: (1) an Image Patch Filter (IPF) that leverages YOLO for foreground/background separation and a ViT-based [CLS] attention mechanism to retain informative image patches, and (2) a fine-grained Architecture Layer Filter (ALF) using a genetic algorithm to prune redundant transformer sub-layers. The method is evaluated on visual question answering (VQA) and natural language visual reasoning (NLVR) tasks with METER and ViLT models, showing reduced computational costs (FLOPs) with minimal performance degradation compared to baseline parameter-efficient methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging feedback and suggestions. However, we argue that some concerns were addressed in our paper, so we clarify them again here. We address the main weaknesses(W*) and questions(Q*) below:\n\n>_W1: “YOLO's computational cost is not accounted for in efficiency metrics ,....”_ \n\n**Clarification:**  We would like to clarify that our paper explicitly states that our experiments take YOLO’s FLOPs into account when calculating the model FLOPs,  **in the corresponding captions of Table 1 (Line 278) and Table 3(Line 332), e.g. “_The FLOPs of Double-Filter contain the YOLO detector._”.**\nWe compare complete Double-Filter with others in Table 1 and show the FLOPs reduction statistics via the diversity IPF in Table 3.\nWe hope this clarification resolves the reviewer’s concern.\n\n>_W2: “The experimental design lacks diversity, focusing solely with fixed sparsity ratios (e.g., 60% patch filtering).”_\n\nFirst, we clarify that Table 3 and Figure 3 present ablation studies on various sparsity ratios, which is inconsistent with the reviewer's claim of a fixed ratio. \n\nSecond, exhaustively testing all ratio combinations would demand substantial computational resources. To balance performance and efficiency, we evaluate four representative ratios—40%, 50%, 60%, and 70%. This approach ensures a comprehensive yet feasible analysis and follows a widely adopted experimental methodology.\n\n>_W3: “YOLO’s inference time likely outweighs the savings from patch pruning.”_\n\nAs shown in Table 2, YOLO’s inference time is included in Double-Filter. Despite this, it still improves overall inference speed. We will clarify this in the final version.\n\n>_W4: “The Appendix provides incomplete details...”_\n\nThe main parameters of the genetic algorithm were given. We guess the concerns lie in YOLO. We clarify that our pipeline does not rely on additional settings of YOLO, and directly uses the pre-trained YOLOv8 (Reisetal.,2023). Therefore, it does not affect our reproducibility. We will add more details in the final version.\n\n>_W5: “The lack of benchmarking against LayerDrop-style dynamic inference ...”_\n\nLayerDrop-style methods typically required modifications during the pre-training stage to enable adaptive inference, whereas our focus lies in fine-tuning pre-trained VLP models without altering the upstream pre-training process. This makes the comparison less aligned in terms of scope—note that DAS (Wu et al., 2023) also does not compare with such methods. Moreover, unlike dynamic methods that often require loading the entire model and selectively dropping layers, our method only loads a searched subnetwork, reducing memory and computational overhead.\n\n>_Q1: “How does YOLO’s computational cost compare to the FLOP savings from patch/layer pruning?”_\n\nTaking METER as an example, as shown in Table 1, our Double-Filter can reduce computational costs by 21.18G FLOPs. The computational costs consider the YOLO’s FLOPs (about 4.5G FLOPs), so there are about 26G FLOPs savings from IPF and ALF. Additionally, Table 3 also shows the detailed computational costs including YOLO’s FLOPs when applying different filter ratios for IPF. The FLOPs saved by image patch filtering far exceeds the introduction of YOLO.\n\n>_Q2: “Can the pipeline function without YOLO for fair comparison to prior work?”_\n\nYOLO effectively distinguishes foreground and background without complex design, enhancing versatility. In future work, we will further explore how to replace YOLO, but this will undoubtedly require a tailored design, and the versatility may be reduced. Additionally, as we clarified in W1, our DF included YOLO's cost when comparing with others, so they are fair comparisons.\n\n>_Q3: “Why were retrieval tasks excluded, given their importance for VLP benchmarking?”_\n\nSorry for the confusion. Due to resource limitations, we conducted VQA and NLVR experiments on an NVIDIA RTX 3090Ti. However, retrieval require numerous negative samples, making it difficult to maintain the same batch size as in prior works (e.g., DAS used NVIDIA Tesla A100 GPU).\n\nTo address the concerns, we rented a larger GPU for image-text retrieval task on Flickr30K. \n\nMETER    |  IR/TRR@1 | Add FLOPs    | ViLT    |  IR/TRR@1 | Add FLOPs\n-------------------|------------------|------------------|-------------------|-----------------|-----------------\nClassifierOnly | 78.80/89.00  | 0.00      | - | 57.42/78.00  | 0.00\nShallow Prompt| 74.20/88.60| +28.71G.   | -| 55.92/74.80| +19.53G\nDeep Prompt| 78.84/89.40  | +6.53G    | -| 58.64/79.50  | +5.14G\nLoRA      | 79.86/92.60   |0.00.      | -| 57.44/77.70   |0.00\nAdapter    | 80.38/91.90  |+1.64G.    | -| 62.68/81.40  |+0.86G\nScaled PA   |  80.40/93.20 | +1.12G.  | -| 61.88/79.00 | +0.44G\nDAS       | 80.12/91.80.  |-11.16G  | - | 60.66/80.80  | -1.03G\n**Double-Filter** | 80.05/91.22  | -21.18G  | - | 61.18/79.39 | -4.72G\n\nOur model for retrieval task still maintains a high efficiency while ensuring the performance with the same setting as Table 1.", "claims_and_evidence": "The paper claims Double-Filter reduces FLOPs by 21.18G (METER/VQA) and 12.51G (METER/NLVR) while maintaining accuracy within 0.5% of full fine-tuning. However, the inclusion of YOLO's computational cost is not accounted for in efficiency metrics , potentially overstating FLOP reductions. The experimental FLOP analysis excludes the YOLO component (Figure 2, Section 3.3), which may add significant overhead during inference, undermining the claimed efficiency.", "methods_and_evaluation_criteria": "The proposed methods (Double-Filter) and evaluation criteria (VQA/NLVR benchmarks with METER/ViLT) align with the objective of improving fine-tuning efficiency for VLP models. However, the evaluation scope is narrow, excluding retrieval tasks critical for VLP benchmarking. YOLO’s overhead in the IPF step is also not accounted for, which could skew FLOP reduction claims.", "theoretical_claims": "The only theoretical component is the FLOP analysis. The derivation of FLOPs for transformer components (MHSA, MHCA, FFN) is mathematically sound, using standard matrix multiplication cost formulas. However, the analysis oversimplifies assumptions (e.g., ignoring tokenization/embedding layers, assuming perfect parallelism). YOLO’s FLOPs are excluded, making the theoretical cost reductions optimistic.\n\nOther claims (e.g., redundancy removal effectiveness) are empirical and lack theoretical backing.", "experimental_designs_or_analyses": "1. Evaluation Scope:\nThe experiments focus on two tasks (VQA 2.0 and NLVR2) with METER and ViLT as baselines. While these are standard benchmarks, the exclusion of retrieval tasks (e.g., COCO evaluation) limits the validity of claims about broad efficiency. VLP models are often evaluated across a more diverse set of tasks, so the absence of retrieval results weakens the argument for general utility.\n\n2. FLOP Calculation:\nThe FLOP analysis excludes YOLO’s computational cost, which could be significant. The claimed 21.18G FLOP reduction for METER/VQA may be offset by YOLO’s overhead, especially during inference. The analysis assumes constant FLOPs for tasks with different input sizes (e.g., VQA’s image + text vs. NLVR’s dual images + text), which may not reflect real-world scenarios.", "relation_to_broader_scientific_literature": "It combines two key techniques: (1) an Image Patch Filter (IPF)[1] that uses YOLO for foreground/background separation and ViT [CLS] tokens to retain informative patches, and (2) a fine-grained Architecture Layer Filter (ALF) employing a genetic algorithm to prune redundant transformer sub-layers.\n\n[1] Liang, Youwei, et al. \"Not all patches are what you need: Expediting vision transformers via token reorganizations.\" arXiv preprint arXiv:2202.07800 (2022).", "essential_references_not_discussed": "Not found.", "other_strengths_and_weaknesses": "Strengths:\n\nThis paper is well written and organized.\n\nAdditional Weaknesses:\n\nThe experimental design lacks diversity, focusing solely on METER/ViLT with fixed sparisty ratios (e.g., 60% patch filtering). Scalability to larger VLP models (e.g., OFA, BEIT-3) or datasets (COCO, Conceptual Captions) is untested.\n\nThe supplementary material (Appendix) provides incomplete details on genetic algorithm parameters or YOLO hyperparameters, limiting clarity on implementation.\n\nIncorporating YOLO adds complexity and potential latency to the pipeline, contradicting the efficiency goals. YOLO’s inference time likely outweighs the savings from patch pruning, especially in real-time applications.\n\nThe Architecture Layer Filter (ALF) draws parallels to LayerDrop (Fan et al., 2019) and DAS (Wu et al., 2024b), which prune entire transformer blocks. ALF’s fine-grained pruning (targeting MHSA/MHCA/FFN sub-layers) is more granular, akin to methods like Dynamic Head Pruning (Michel et al., 2019). The use of a genetic algorithm for layer selection is novel but conceptually related to AutoML frameworks (e.g., NAS), though this paper doesn’t position ALF as a neural architecture search. The lack of benchmarking against LayerDrop-style dynamic inference suggests a gap in comparing to adaptive methods.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How does YOLO’s computational cost compare to the FLOP savings from patch/layer pruning? Can the pipeline function without YOLO for fair comparison to prior work?\n\nWhy were retrieval tasks excluded, given their importance for VLP benchmarking?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZLeCL4uddj", "reviewer": "Reviewer_g8nL", "summary": "The paper introduces Double-Filter, an approach for efficient fine-tuning of Vision-Language Pre-trained (VLP) models by addressing redundancies at both the data and architectural levels. The authors propose two main components: (1) an Image Patch Filter (IPF) that selectively filters image patches by distinguishing between foreground and background regions, and (2) a Fine-grained Architecture Layer Filter (ALF) that employs an adjustable genetic algorithm (AGA) to identify and remove redundant sub-layers within transformer blocks. The authors claim that Double-Filter significantly reduces computational costs (e.g., 21.18G FLOPs reduction on METER) while maintaining competitive performance. The paper provides experimental results on METER and ViLT models, demonstrating the effectiveness of the proposed method in balancing efficiency and performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review, acknowledging our contributions, and providing instructive suggestions. We respond to each weakness (_W*_) or question (_Q*_) below.  \n\n>_W1: “... explore the impact on more complex multimodal tasks beyond VQA and NLVR, ...”_\n\nThanks for the instructive suggestion. We conducted VQA and NLVR experiments on one NVIDIA RTX 3090Ti, but the retrieval task requires many negative samples, making it difficult to maintain the same batch size compared to existing methods.\n\nTo address the concerns, we rented a larger GPU and conducted generalization experiments on Flickr30K for image-text retrieval. The response Table reports our Double-Filter with same setting as Table 1 in the paper compared to the mainstream methods.\n\nMETER    |  IR/TRR@1 | Add FLOPs    | ViLT    |  IR/TRR@1 | Add FLOPs\n-------------------|------------------|------------------|-------------------|------------------|------------------\nClassifierOnly | 78.80/89.00  | 0.00      | - | 57.42/78.00  | 0.00\nShallow Prompt| 74.20/88.60| +28.71G.   | -| 55.92/74.80| +19.53G\nDeep Prompt| 78.84/89.40  | +6.53G    | -| 58.64/79.50  | +5.14G\nLoRA      | 79.86/92.60   |0.00.      | -| 57.44/77.70   |0.00\nAdapter    | 80.38/91.90  |+1.64G.    | -| 62.68/81.40  |+0.86G\nScaled PA   |  80.40/93.20 | +1.12G.  | -| 61.88/79.00 | +0.44G\nDAS       | 80.12/91.80.  |-11.16G  | - | 60.66/80.80  | -1.03G\n**Double-Filter** | 80.05/91.22  | -21.18G  | - | 61.18/79.39 | -4.72G\n\nOur Double-Filter achieves the lowest computational complexity among PEFT methods and maintains competitive performance in image-text retrieval.\n\n>_W2: “The authors do not discuss the computational cost of running the GA, which ...”_\n\nThanks for the detailed suggestion. As mentioned in **Appendix A.1.2 (Lines 551-557), our loss calculations in the fitness function were based on 100 batches of training data (equivalent to 2 epochs), and losses were recorded over 10 validation batches**. The search cost of GA is equivalent to adding 2 epochs to the training phase, but DAS takes 3 epochs and requires full validation set validation. So the GA cost is still lower than the comparison method.\n\n>_W3: “A more detailed discussion of the limitations of the proposed method.”_\n\nThanks for the detailed suggestion. We will further discuss detailed limitations. On the one hand, our method relies on pre-trained target detectors, but it still adds additional inference time; on the other hand, we use genetic algorithms to search for the best subnet, which has a long search time in the training stage and is prone to fall into the limitation of local optima. We will add the discussion in the final version.\n \n>_Q1: “Have the authors tested Double-Filter on other VLP models.”_\n\nThanks for the instructive comment. We choose METER and ViLT as baselines because:\n\n(1) METER and ViLT are the primary baselines for our comparison methods. To ensure a fair evaluation, we followed the standard practice of using the same VLP models as our baselines.\n\n(2) METER and ViLT represent two mainstream VLP architectures, and others generally follow these architectures. METER employs complex cross-modal fusion, while ViLT maintains independent unimodal encoding. Our results show that Double-Filter achieves notable efficiency improvements, especially under METER, suggesting its potential for more complex VLP models.\n\n>_Q2: “Have the authors considered the impact of Double-Filter on inference time and memory usage.”_ \n\nThanks. In fact, **as shown in Table 2 (Line 332), we have compared the inference speed of Double-Filter with other methods**. Our results demonstrate that Double-Filter enables processing more samples per unit time, indicating a clear efficiency improvement. The following table provides additional memory consumption (inferencing with batch size of 1) to further illustrate the lower memory of Double-Filter.  \n\nMETER     | Full tuning  |  Adapter    | DAS       | **Double-Filter**\n------------------|------------------|------------------|-------------------|-------------------\nVQA       |   3068M  | 3090M     |  2950M    | **2906M**\nNLVR      |   3030M   | 3050M    | 2916M      | **2884M**\n\n\n>_Q3: “Have the authors explored other object detection methods.”_\n\nThank you for highlighting this vital direction.  We employed YOLOv8 because the YOLOs are the most mainstream object detectors, which can better balance effectiveness and efficiency of detection. We further test EfficientDet[1], which need more computing resources. Due to the limited rebuttal time, we only compared the IPF filtering overlap among detectors, and found that over 96% of filtered patches remained consistent, so we believe that this has little impact on the final patch retention. Future work will explore alternative methods, including lightweight offline pre-training.\n\n[1] Tan, Mingxing, Ruoming Pang, and Quoc V. Le. \"EfficientDet:Scalable and efficient object detection.\" CVPR. 2020.", "claims_and_evidence": "The claims made in the paper are generally supported by experimental evidence, but there are some areas where the evidence could be stronger or more comprehensive. The authors provide extensive ablation studies and comparisons with state-of-the-art PEFT methods, demonstrating that Double-Filter achieves significant reductions in computational costs (FLOPs) while maintaining competitive accuracy on VQA and NLVR tasks. For example, the paper reports that Double-Filter reduces FLOPs by 21.18G on the METER model for the VQA task, with only a minimal drop in accuracy (0.27% on average). However, the paper lacks a thorough discussion of the limitations of the proposed method. For instance, while the results are impressive on METER and ViLT, it is unclear how Double-Filter would perform on other VLP models, such as CLIP or ALIGN, which have different architectures and training objectives. Additionally, the paper does not explore the impact of patch filtering or layer removal on more complex multimodal tasks beyond VQA and NLVR, which limits the generalizability of the claims.", "methods_and_evaluation_criteria": "The proposed methods are partially suited for the problem of efficient fine-tuning of VLP models,  there are some areas where the approach could be improved. The Image Patch Filter (IPF) is a logical approach to reducing redundancy in image patches, which are known to be a major bottleneck in VLP models. By using a YOLO detector to separate foreground and background regions and then applying ViT attention scores to rank patch importance, the authors ensure that the most semantically relevant patches are retained. This approach is particularly effective for tasks like VQA and NLVR, where both foreground objects and background context are important.\n\nHowever, the reliance on YOLO for foreground-background separation could be a limitation. YOLO is a computationally expensive object detection model, and its use adds overhead to the overall pipeline. The authors do not explore alternative object detection methods, which could potentially offer a better trade-off between accuracy and computational cost. Additionally, the patch filtering process is based on ViT attention scores, which may not always capture the full semantic importance of patches, especially in complex scenes with multiple objects or ambiguous backgrounds.\n\nThe Fine-grained Architecture Layer Filter (ALF) is a novel contribution, but the use of a genetic algorithm (GA) for layer removal introduces some challenges. While the GA is effective in exploring the space of possible layer configurations, it is computationally expensive and requires multiple generations of evaluation to converge. The authors do not discuss the computational cost of running the GA, which could be significant, especially for larger models or more complex tasks. Furthermore, the GA-based approach may not scale well to models with a large number of layers, as the search space grows exponentially with the number of layers.", "theoretical_claims": "The paper does not present any theoretical proofs, so there are no theoretical claims to evaluate. The focus is primarily on empirical results and algorithmic contributions. However, the authors do provide a detailed analysis of the FLOPs reduction achieved by Double-Filter. The paper could benefit from a more rigorous theoretical analysis of the proposed methods. For example, the authors could provide a theoretical justification for why certain layers are more redundant than others, or why the proposed patch filtering strategy is optimal for preserving semantic information.", "experimental_designs_or_analyses": "The experimental design is sound generally but there are some areas where the analysis could be improved. The authors conduct extensive experiments on two widely used VLP models (METER and ViLT) and compare their method against several state-of-the-art PEFT methods, including Shallow Prompt, Deep Prompt, LoRA, Adapter, and DAS. The ablation studies on patch filtering and layer removal are particularly insightful, demonstrating the impact of different filtering ratios and layer removal strategies on model performance and efficiency.\n\nHowever, the experiments are limited to two downstream tasks (VQA and NLVR), which may not fully capture the generalizability of the proposed method. The authors do not explore the performance of Double-Filter on other tasks, such as image captioning or text-to-image generation, which are also common applications of VLP models. Additionally, the experiments are conducted on relatively small datasets (e.g., VQA 2.0), and it is unclear how the method would perform on larger or more diverse datasets.\n\nAnother limitation is the lack of analysis on the impact of Double-Filter on inference time and memory usage. While the paper focuses on FLOPs reduction, these metrics are critical for real-world applications, especially in resource-constrained environments.", "relation_to_broader_scientific_literature": "The paper is situated within the broader literature on efficient fine-tuning of VLP models. The authors discuss related work on adapter-based methods, block-skipping strategies, and other PEFT approaches, highlighting how their method differs by addressing both data-level and architecture-level redundancies. For example, the authors compare their approach to DAS, which also aims to reduce architectural redundancy but does so at a coarser level by replacing entire transformer blocks.", "essential_references_not_discussed": "The paper covers most of the essential references related to efficient fine-tuning of VLP models. The paper could reference more recent advancements in genetic algorithms for neural architecture search, which might provide further context for the proposed AGA-based layer filtering approach.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper presents an effective approach to efficient fine-tuning of VLP models by addressing both data-level and architecture-level redundancies.\n\n- The ablation studies and visualizations provide valuable insights into the impact of patch filtering and layer removal on model performance and efficiency.\n\nWeaknesses:\n\n- The paper could benefit from a more detailed discussion of the limitations of the proposed method, particularly in scenarios where patch filtering or layer removal might negatively impact performance.\n\n- The authors could explore additional methods for determining patch importance, beyond the ViT-based approach, to further improve the robustness of the IPF.\n\n- The reliance on YOLO for foreground-background separation adds computational overhead, and alternative object detection methods should be explored.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The paper mentions that the proposed method is effective under METER and ViLT models. Have the authors tested Double-Filter on other VLP models.\n2. The paper focuses on reducing FLOPs and maintaining accuracy. Have the authors considered the impact of Double-Filter on inference time and memory usage.\n3. The paper uses a YOLO detector for foreground-background separation. Have the authors explored other object detection methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6nqdNC33nl", "reviewer": "Reviewer_pyf1", "summary": "The paper introduces Double-Filter for refining the fine-tuning process of VLP models. It employs two key strategies to reduce redundancy: A new patch selection method that uses background-foreground separation to improve image feature selection; Then, a genetic algorithm designed to remove redundant architectural layers, thereby enhancing both the efficiency and effectiveness of the model. Together, these strategies aim to streamline the fine-tuning process while maintaining the model’s performance. Experimental results demonstrate that the proposed approach achieves competitive performance, with only marginal degradation compared to existing parameter-efficient fine-tuning methods, while significantly reducing computational complexity through the filtering of over 60% of image patches and 12 model layers.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable and constructive comments. We appreciate your recognition of our work's innovative designs, evaluation criteria, and insightful analysis. Below, we address your concerns and questions individually:\n\n>_Q1: “In IPF section, the paper mentions sparsity ρ ∈ (0, 1). Is this sparsity a hyperparameter, and if so, how is it set or optimized during the training process? Could the authors provide more details on how this parameter is determined?”_\n\nThanks for your detailed suggestion. The sparsity parameter (ρ) in the image patch filter is actually a hyperparameter that controls the filtering ratio of the image patches. The larger the sparsity parameter, the greater the filtering degree. In fact, in Table 3 of the paper, we have conducted ablation experiments by setting the sparsity parameter (ρ) differently to achieve a balance between efficiency and performance. We will further clarify this in the final version to avoid confusion.\n\n>_Q2: “The authors claim that YOLOv8’s cost is much lower than other modules, but no clear evidence is provided. Please estimate the forward pass cost of this module,...”_\n\nThank you for the helpful suggestion. In our original analysis, we omitted the YOLOv8 cost because it is negligible compared to the overall model computation. However, for completeness, we now explicitly to calculate it. Specifically, YOLOv8 used in our IPF module requires approximately 4.5 G FLOPs per forward pass on one RTX 3090Ti. For the ViT-based encoder, taking the METER model as an example, it requires a total of about 100G FLOPs cost in per forward pass, of which each transformer Block in the interaction network requires approximately 5G FLOPs. Therefore, YOLOs accounts for less than 5% of the total, which is approximately equal to the cost of one Block.\n\nWe will revise Section 3.3 and Table 1 to explicitly include this small cost to provide a complete picture of all forward computations. \n\nIn addition, as shown in Table 1 and Table 3, even considering the cost of YOLO, our Double-Filter methods significantly reduces the computational cost when fine-tuning the VLP model, especially for models with complex interactions such as the METER model.\n\n>_Q3. “Why choose METER and ViLT instead of other VLP models as the base models for experiments? The author needs to give a clearer explanation and reason.”_\n\nThanks for the insightful suggestion. Our choice of METER and ViLT as baselines was based on the following considerations:\n\n(1) METER and ViLT are the primary baselines for our comparison methods. To ensure a fair and meaningful evaluation, we followed the standard practice of using the same VLP models as our comparison baselines.\n\n(2) METER and ViLT represent two distinct VLP architectures. METER employs complex cross-modal fusion mechanisms, while ViLT maintains independent unimodal encoding. Our experimental results demonstrate that our Double-Filter achieves more significant efficiency improvements under the complex METER framework. This suggests that Double-Filter has the potential to be even more effective when applied to more complex VLP models.\n\nGiven these considerations, we choose METER and ViLT. Nonetheless, we acknowledge the importance of further validation on additional VLP models and will explore this in future work.\n\n\n>_Q4. “In the Fitness Function, why must ( \\beta ) be greater than half of the maximum summed losses? How does this choice impact the optimization process?”_\n\nSorry for the confusion. In this paper, the role of $\\beta$ is precisely to transform fitness values into the positive domain. In genetic algorithms, roulette wheel selection necessitates that the fitness scores generated by the fitness function be positive, as its core mechanism depends on transforming fitness values into non-negative and normalized probability distributions. If fitness values are negative, it not only leads to calculated selection probabilities being negative, which contradicts the fundamental principles of probability, but may also trigger division-by-zero errors or render probability calculations invalid when the sum of fitness values is zero or negative. Consequently, by employing a translation method to shift all fitness values into the positive domain, the relative ranking of individuals is maintained, thereby ensuring the algorithm's effective and reliable operation.  We will clearly these discussion in the new revision.\n\n>_Q5. “On page 6, in the left column, there are two formulas that appear to be excessively long.”_\n\nThank you for the detailed suggestion! We'll simplify or break these formulas down into clearer segments to enhance readability.", "claims_and_evidence": "Experimental results demonstrate that the proposed approach achieves competitive performance, with only marginal degradation compared to existing parameter-efficient fine-tuning methods, while significantly reducing computational complexity through the filtering of over 60% of image patches and 12 model layers.\nThe claims are clear with enough evidence.", "methods_and_evaluation_criteria": "The proposed efficient training and inferencing Double Filter method makes it possible to deploy multimodal LLMs on low-resource devices by removing redundancy from both data and models, and also provides an effective reference for efficient training.", "theoretical_claims": "The theoretical method proposed in this paper is correct, and the detailed algorithms are given. In addition, this paper verifies the key efficiency design derivation.", "experimental_designs_or_analyses": "Experimental results demonstrate that the proposed approach achieves competitive performance, with only marginal degradation compared to existing parameter-efficient fine-tuning methods, while significantly reducing computational complexity through the filtering of over 60% of image patches and 12 model layers.\n\nThe ablation experiments and visualization results are sufficient to demonstrate the effectiveness of each key component proposed.", "relation_to_broader_scientific_literature": "The research on VLP efficiency is very extensive. This paper introduces new thinking to this field through double filtering of data and model and based on the idea of ​​genetic algorithm, which is beneficial.", "essential_references_not_discussed": "Enough.", "other_strengths_and_weaknesses": "Strength：\n1. The paper is well-written and easy to follow. And the motivation is reasonable\n\n2. I appreciate that the authors provide a detailed proof process for the claimed efficiency while describing the model in detail (giving two detailed algorithmics). This paper provided enough details in ensuring the correctness of its theoretical proofs.\n\n3. Double-Filter achieved significant reductions in FLOPs while maintaining competitive performance. Demonstrates versatility across multiple VLP models (METER and ViLT) and tasks (VQA and NLVR2).\n\n\nWeakness：\n1. The computational cost of YOLOv8 is included in Section 3.3 (FLOPs analysis). The authors claim that its cost is much lower than other modules, but no clear evidence is provided.\nPlease estimate the forward pass cost of this module, as it is part of the forward process of the proposed method. The same problem exists in Table 1, please consider all forward computations.\n\n2. In the Image Patch Filter (IPF) section, the paper mentions sparsity ρ ∈ (0, 1). Is this sparsity a hyperparameter, and if so, how is it set or optimized during the training process? Could the authors provide more details on how this parameter is determined?\n\n3. In the Fitness Function, why must \\( \\beta \\) be greater than half of the maximum summed losses? How does this choice impact the optimization process?\n\n4. Why choose METER and ViLT instead of other VLP models as the base models for experiments? The author needs to give a clearer explanation and reason.\n\n5. On page 6, in the left column, there are two formulas that appear to be excessively long.", "other_comments_or_suggestions": "See Weaknesses.", "questions_for_authors": "1. In the Image Patch Filter (IPF) section, the paper mentions sparsity ρ ∈ (0, 1). Is this sparsity a hyperparameter, and if so, how is it set or optimized during the training process? Could the authors provide more details on how this parameter is determined?\n\n2. Please estimate the forward pass cost of this module, as it is part of the forward process of the proposed method. The same problem exists in Table 1, please consider all forward computations.\n\n3. Why choose METER and ViLT instead of other VLP models as the base models for experiments? The author needs to give a clearer explanation and reason.\n\n4. In the Fitness Function, why must \\( \\beta \\) be greater than half of the maximum summed losses? How does this choice impact the optimization process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9wHHoFhF6J", "reviewer": "Reviewer_H3jf", "summary": "The paper introduces Double-Filter, a novel approach for efficient fine-tuning of vision-language pre-trained (VLP) models. The method addresses redundancy at both data and model levels. At the data level, an Image Patch Filter (IPF) leveraging a YOLO detector and ViT attention scores to retain only the most informative image patches is proposed, while at the model level, a fine-grained Architecture Layer Filter (ALF) using an adjustable genetic algorithm (AGA) to selectively replace redundant sub-layers within transformer blocks is designed. Experiments on VQA and NLVR tasks using the METER and ViLT models demonstrate that Double-Filter can significantly reduce FLOPs with only marginal performance degradation, thereby offering a compelling efficiency-performance trade-off.\n\n## update after rebuttal\nThank you to the authors for their rebuttal, which has addressed most of my concerns. Hence, I maintain my recommendation of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer very much for the kind words, for the interest in our research activities, and for the very insightful comments.\n\n>_W1: “Potential weaknesses include an increased system complexity due to the integration of a YOLO detector and genetic algorithm, and limited exploration of the method’s generalizability beyond VQA and NLVR.”_\n\nThanks for the comment.\nFor the system complexity of the integration of a YOLO detector and genetic algorithm, as shown in Table 1 and Table 2, after introducing YOLO-based IPF and AGA-based ALF, our Double-Filter has significantly reduced the calculation of FLPOs, and the inference time has been greatly improved. Although the model introduces more algorithms, the inference efficiency of the VLP model has been significantly improved.  \n\nFor the exploration of the method’s generalizability to more tasks, we further experimented on more complex image and text retrieval tasks by renting a larger GPU. Due to time limits, the response Table reports our Double-Filter with same setting as Table 1 in the paper compared to the mainstream methods.\n\nMETER    |  IR/TRR@1 | Add FLOPs    | ViLT    |  IR/TRR@1 | Add FLOPs\n-------------------|------------------|------------------|-------------------|------------------|------------------\nClassifierOnly | 78.80/89.00  | 0.00      | - | 57.42/78.00  | 0.00\nShallow Prompt| 74.20/88.60| +28.71G.   | -| 55.92/74.80| +19.53G\nDeep Prompt| 78.84/89.40  | +6.53G    | -| 58.64/79.50  | +5.14G\nLoRA      | 79.86/92.60   |0.00.      | -| 57.44/77.70   |0.00\nAdapter    | 80.38/91.90  |+1.64G.    | -| 62.68/81.40  |+0.86G\nScaled PA   |  80.40/93.20 | +1.12G.  | -| 61.88/79.00 | +0.44G\nDAS       | 80.12/91.80.  |-11.16G  | - | 60.66/80.80  | -1.03G\n**Double-Filter** | 80.05/91.22  | -21.18G  | - | 61.18/79.39 | -4.72G\n\nOn the cross-modal retrieval task, our proposed Double-Filter also shows the lowest computational complexity among existing PEFT methods, and maintains competitive performance. This further demonstrates the task generalization ability of our proposed Double-Filter method.\n\n\n>_Q1: “Could you elaborate on your choice of a genetic algorithm for the fine-grained Architecture Layer Filter?”_\n\nThanks. **As description in Section 3.2, we propose an Adjustable Genetic Algorithm (AGA) for fine-grained architecture filtering, approximating an optimal configuration.** Based on the traditional genetic algorithm, it introduced adaptive adjustment for custom number of filtering layers, so as to better perform adaptive training and deployment capabilities on different devices. Each model configuration is encoded as a chromosome, with genetic operations (fitness evaluation, crossover, and mutation) guiding optimization across generations. The fitness function balances loss on sampled datasets, ensuring optimal layer reduction. Crossover swaps genes between parents, while mutation adjusts gene values to maintain a fixed number of replaced layers. \n\nAdditionally, as mentioned in **Appendix A.1.2 (Lines 551-557), our loss calculations in the fitness function were based on 100 batches of training data (equivalent to 2 epochs), and losses were recorded over 10 validation batches**. The search cost of GA is equivalent to adding 2 epochs to the training phase, but DAS requires 3 epochs and requires full validation set validation. So the GA cost is still more efficient than the comparison method.\n\nMoreover, **please refer to our response to Q2, which further discusses optimization quality and stability.**\n\n>_Q2: “Given that GAs are heuristic and do not always guarantee an optimal result and requires hyperparameter tuning, did you consider alternative optimization methods that might offer stronger guarantees on the quality or stability of the optimization outcome?”_\n\nThanks for the insightful comment. **As we said in Footnote2 on Page 5, identifying an absolute optimal structure is NP-hard, rendering exhaustive search impractical. So as mentioned in Section 3.2 (line 252), \"The AGA aims to identify the approximate optimal reduction with L filtered layers.\"**\n\nComparing with other searching algorithms, genetic algorithm uses population search to avoid falling into the local optimal solution, and has a greater probability to find the global optimal solution. And another reason we choose GA-based model is that genetic algorithms can be well parallelized and improve computational efficiency, because of the independent evaluation among individuals.\n\nTo offer stronger guarantees on the quality or stability of the optimization outcome, the \"Elite Strategy\" is adopted in step 11 of Algorithm 2 to ensure that the optimal solution of the current generation is not lost during each generation update, thereby ensuring the quality of the search. We design the mutation operation to satisfy the constraint of the genetic algorithm while diversifying each generation effectively.\n\n\n**We hope that the above responses could address the reviewers' concerns and questions.**", "claims_and_evidence": "The paper makes several key claims: (1) The proposed dual filtering strategy effectively reduces both data-level and architecture-level redundancies, which in turn cuts down computational costs during fine-tuning. This is supported by theoretical FLOPs analysis and extensive ablation studies. (2) The Image Patch Filter preserves global semantic integrity by separately considering foreground and background regions, rather than focusing solely on salient objects. (3) The fine-grained Architecture Layer Filter, implemented via an adjustable genetic algorithm, enables selective pruning of sub-layers while retaining model performance. Experimental results comparing Double-Filter with other parameter-efficient fine-tuning (PEFT) methods (e.g., LoRA, Adapter, DAS) provide quantitative evidence for these claims, the visualization of IPF outputs supports the claim (2)", "methods_and_evaluation_criteria": "The methodology is built on two complementary components. The IPF first segments an image using a YOLO detector into foreground and background, then ranks patches based on [CLS] attention scores from a pre-trained ViT model. The ALF formulates layer pruning as an optimization problem solved by a genetic algorithm that iteratively adjusts a binary chromosome representation of the transformer’s sub-layers. Evaluation is conducted on two downstream tasks (VQA and NLVR) using common metrics such as accuracy and FLOPs reduction, alongside detailed ablation studies that examine varying patch filtering ratios and layer removal extents.", "theoretical_claims": "The paper provides theoretical analysis regarding the reduction of FLOPs, demonstrating how the Double-Filter framework scales down the computational complexity of transformer blocks from the vanilla model. It also argues that selectively replacing sub-components (rather than entire blocks) offers a finer control over efficiency.However, the current experimental setup does not include a dedicated ablation study that directly compares selective sub-component replacement with whole block removal. A focused comparison—using image patch filtering (IPF) while contrasting ALF with block-level filtering methods would provide more definitive evidence for the benefits of fine-grained architectural control.", "experimental_designs_or_analyses": "Experiments are carried out on two widely adopted VLP models—METER and ViLT—across the VQA and NLVR benchmarks. The results indicate that Double-Filter achieves a substantial reduction in FLOPs (e.g., over 21G reduction for VQA on METER) while incurring only a minimal drop in accuracy. Detailed ablation studies further explore the impact of varying the patch filtering ratio and the number of layers pruned, validating the effectiveness of both the IPF and ALF components. The inclusion of inference speed comparisons also proves the effectiveness of proposed method in boosting VLP finetune.", "relation_to_broader_scientific_literature": "The work builds on recent advances in efficient fine-tuning methods for VLP models, extending beyond traditional adapter-based approaches and block-skipping strategies (e.g., DAS(Wu\net al., 2024b)). The work also builds on a patch filtering techique that assigns an importance score to\neach patch by leveraging the classification token ([CLS]) of the pre-trained ViT following (Liang et al., 2022).", "essential_references_not_discussed": "The paper comprehensively discussed the related works.", "other_strengths_and_weaknesses": "Strengths of the paper include its dual-level approach to reducing redundancy, a sound theoretical grounding through FLOPs analysis, and extensive experimental validation across multiple benchmarks. The modular design of the proposed filters offers flexibility and could be adapted to various VLP models. On the other hand, potential weaknesses include an increased system complexity due to the integration of a YOLO detector and genetic algorithm, and limited exploration of the method’s generalizability to tasks beyond VQA and NLVR.", "other_comments_or_suggestions": "It would be beneficial for the authors to expand the evaluation to include a broader set of tasks or datasets could further validate the robustness of the approach. A discussion on hyperparameter sensitivity for the genetic algorithm would also help in understanding the stability of the ALF component.", "questions_for_authors": "Could you elaborate on your choice of a genetic algorithm for the fine-grained Architecture Layer Filter? Given that GAs are heuristic and do not always guarantee an optimal result and requires hyperparameter tuning, did you consider alternative optimization methods that might offer stronger guarantees on the quality or stability of the optimization outcome?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKoHQCPddL", "reviewer": "Reviewer_YEqs", "summary": "This paper presents Double-Filter, a method aimed at optimizing the fine-tuning process of vision-language pre-trained (VLP) models. The method reduces redundancy through two strategies: first, a novel patch selection approach that enhances feature representation via background-foreground separation; second, a genetic algorithm to eliminate redundant architectural layers, improving model efficiency.\nExperiments on benchmark tasks, including VQA and NLVR2, with METER and ViLT models, show that Double-Filter achieves substantial reductions in computational complexity (e.g., up to 21.18 GFLOPs in VQA with METER) while maintaining competitive accuracy, with minimal performance degradation (0.27% on METER and 0.65% on ViLT). Visualizations further confirm that IPF effectively preserves global contextual semantics, and ALF ensures task-specific layer optimization.\n\n## update after rebuttal\nAuthors' rebuttal have solved my concerns. I think previous score is high enough and I will keep my rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very detailed and instructive feedback. We are very happy that you recognize our motivation and work and give positive support. Regarding your suggestions and concerns, we will respond one by one below, hoping to answer your questions.\n\n>_Q1. “YOLO's object detection may be inaccurate, leading to incorrect foreground-background segmentation and potentially affecting the patch selection process.”_\n\nThanks for your comment. \nIndeed, the situation you mentioned is possible. But this is one of the motivations behind our decision to remove redundancy in the foreground and background separately and retain a certain proportion of patches respectively. On the one hand, our attention mechanism is based on the global attention distribution, so even if there are missed or wrong detections in the foreground, our IPF method can retain these important entity areas in the background (we assume they are important). Therefore, in this case, it can maintain the same global salient object attention ability as EViT(Liangetal.,2022). On the other hand, since we distinguish the foreground and background, we can pay more attention to the background than EViT, which only focuses on the salient area, even if there is no clear semantics, it is very important for the logical integrity of the image context expression.\n\n>_Q2. “The paper lacks a detailed analysis of YOLO's parameter count and FLOPs, which is important for assessing its computational cost in the proposed method.”_\n\nThank you for the helpful suggestion. Specifically, YOLOv8 used in our IPF module requires approximately 4.5G FLOPs per forward pass on one RTX 3090Ti. Taking the METER model as an example, it requires a total of about 100G FLOPs cost in per forward pass, of which each transformer Block in the interaction network requires approximately 5G FLOPs. Therefore, YOLOs accounts for less than 5% of the total, which is approximately equal to the cost of one Block. We will revise Section 3.3 and Table 1 in the new paper to explicitly include this small cost to provide a complete picture of all forward computations. \n\n>_Q3. “Why does the paper not consider addressing redundancy in the text modality?”_\n\nThank you for your insightful question and we appreciate the opportunity to clarify this aspect of our work.\nOur study focuses primarily on image redundancy because images account for the majority of computational overhead in Vision-Language Pretraining (VLP) models. **As discussed in Section 3.3 (Lines 259–264) of our paper**, textual redundancy is considerably lower than that of image patches, making image redundancy the more critical bottleneck to address.\nTasks commonly used in VLP, such as image captioning, VQA, and cross-modal retrieval,etc., typically involve short text sequences. For example, in the METER and ViLT models we adopt, the maximum text length is only 40–50 tokens, whereas ViLT-B/32 processes over 240 image patches—meaning text accounts for less than 20% of the total input length. Similar observations are also reported in [1, 2].\nMoreover, since the computational cost (FLOPs) of transformer-based models scales linearly with sequence length, the relatively short length of text implies that the FLOPs consumed by the text modality are inherently much lower. Therefore, we concentrate on optimizing the visual modality, where redundancy is more significant and computational savings are more impactful.\n\nReferences:\n\n[1] Yang, Senqiao, et al. \"Visionzip: Longer is better but not necessary in vision language models.\" arXiv preprint arXiv:2412.04467 (2024).\n\n[2] Kim, Wonjae, Bokyung Son, and Ildoo Kim. \"ViLT: Vision-and-language transformer without convolution or region supervision.\" International Conference on Machine Learning, PMLR, 2021.\n\n>_Q4．A minor but important point: I noticed that the paper inconsistently uses \"FLOPS\" and \"FLOPs\" to refer to floating-point operations._\n\nThank you for pointing this out. We will standardize the terminology throughout the paper, consistently using \"FLOPs\" to ensure clarity and accuracy. And we will carefully proofread the full text to ensure there are no typos.\n\n\n**We hope that the above responses could address the reviewers' concerns and questions.**", "claims_and_evidence": "The claims made in the submission are clear and the paper provided enough evidence. The proposed method achieved significant reductions in FLOPs while maintaining competitive performance.", "methods_and_evaluation_criteria": "The proposed method combined Image Patch Filter (IPF) for data input optimization and Architecture Layer Filter (ALF) for network architecture optimization, providing a comprehensive solution for efficient fine-tuning. The motivation for fine-grained model redundancy removal is reasonable.\nExperiment results show that the proposed method can reduce a large portion of input tokens while maintaining the performance.", "theoretical_claims": "This paper provided enough details in ensuring the correctness of its theoretical proofs. The author not only provides a clear description of each component of the model design but also presents well-defined assumptions, verifying the key efficiency design derivations and proof steps.", "experimental_designs_or_analyses": "1. It achieved significant reductions in FLOPs while maintaining competitive performance. \n\n2. The ablation experiment is well designed and sufficient, clearly demonstrating the effectiveness of each key component proposed.\n\n3. Clear visualization results are given to prove the effectiveness of the method.", "relation_to_broader_scientific_literature": "The motivation for fine-grained model redundancy removal is reasonable. It maintain a coherent and complete structure suitable for submission to ICML2025.", "essential_references_not_discussed": "References are enough in this paper.", "other_strengths_and_weaknesses": "Strength：\n1. This paper appears to maintain a coherent and complete structure suitable for submission to ICML2025.\n\n2. Achieves significant reductions in FLOPs while maintaining competitive performance. The topic of model architecture pruning is interesting and will be important for model deployment (e.g. on edge devices).\n\n3. Unlike traditional methods that focus on coarse-grained block-level pruning, the ALF component employs a fine-grained filtering strategy. By targeting specific sub-layers (e.g., Multi-Head Self-Attention, Feed-Forward Network) within Transformer blocks, it achieves a more precise balance between model compression and performance retention.\n\nWeakness：\n1. The Image Patch Filter (IPF) relies on YOLO and ViT to determine patch importance. However, in complex scenarios or cases with occlusions, YOLO's object detection may be inaccurate, leading to incorrect foreground-background segmentation and potentially affecting the patch selection process.\n\n2. The paper lacks a detailed analysis of YOLO's parameter count and FLOPs, which is important for assessing its computational cost in the proposed method.\n\n3. Why does the paper not consider addressing redundancy in the text modality？\n\n4. A minor but important point: I noticed that the paper inconsistently uses \"FLOPS\" and \"FLOPs\" to refer to floating-point operations. It would be helpful to standardize the terminology throughout the paper for clarity and accuracy.", "other_comments_or_suggestions": "A minor but important point: I noticed that the paper inconsistently uses \"FLOPS\" and \"FLOPs\" to refer to floating-point operations. It would be helpful to standardize the terminology throughout the paper for clarity and accuracy.", "questions_for_authors": "1. The Image Patch Filter (IPF) relies on YOLO and ViT to determine patch importance. However, in complex scenarios or cases with occlusions, YOLO's object detection may be inaccurate, leading to incorrect foreground-background segmentation and potentially affecting the patch selection process.\n\n2. The paper lacks a detailed analysis of YOLO's parameter count and FLOPs, which is important for assessing its computational cost in the proposed method.\n\n3. Why does the paper not consider addressing redundancy in the text modality？", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a double filter approach to compress the fine-tuning process of vision-language pre-trained (VLP) models via filtering redundancies in feature inputs and architectural components. A patch selection and redundant layer elimination are developed for performance preservation and model acceleration. It received positive reviews mostly. The reviewers are generally acknowledge the contribution of the proposed method. One reviewer [Qv8A] mentions the experimental issue regarding the comparison diversity, which is solved via the author responses. Overall, the AC has checked all the files, and feels the current work is ready for reporting at present. The authors shall incorporate reviewers' comments in the camera-ready revisions.
Decision: Accept (poster)