Compressing and Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
Authors: Qingyi Si
Yuanxin Liu
Zheng Lin
Peng Fu
Yanan Cao
Weiping Wang
Venue: EMNLP 2023 Main (2023)
Keywords: visual question answering
out-of-distribution
robustness
debiasing

Abstract: Despite the excellent performance of vision-language pre-trained models (VLPs) on conventional VQA task, they still suffer from two problems: First, VLPs tend to rely on language biases in datasets and fail to generalize to out-of-distribution (OOD) data. Second, they are inefficient in terms of memory footprint and computation. Although promising progress has been made in both problems, most existing works tackle them independently. To facilitate the application of VLP to VQA tasks, it is imperative to jointly study VLP compression and OOD robustness, which, however, has not yet been explored. This paper investigates whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. To this end, we systematically study the design of a training and compression pipeline to search the subnetworks, as well as the assignment of sparsity to different modality-specific modules. Our experiments involve 2 VLPs, 2 compression methods, 4 training methods, 2 datasets and a range of sparsity levels. Our results show that there indeed exist sparse and robust subnetworks, which are competitive with the debiased full VLP and clearly outperform the debiasing SoTAs with fewer parameters on OOD datasets VQA-CP v2 and VQA-VS. The codes can be found at https://github.com/PhoebusSi/Compress-Robust-VQA.

Compressing and Debiasing Vision-Language Pre-Traines Models for Visual Question Answering
Qingyi Si Yuanxin Liu Zheng Lin
Peng Fu', Yanan Caol.2, Weiping Wang'
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China NNationl Key Laboratory for Multimedia Information Processing. School of Computer Science, Peking University
(siqingyi, (siqingyi,Iinzhenn,, linzheng, fupeng, caoyanan, wangweipingg@iiee.... cn, liuyuanxinéstu. pku. edu. cn
Abstract
Oura-10%
Ours 30%
(mmet)mm)))
Despite the excellent performance of vision- language pre-trained models (VLPs) on con- ventional VQA task, they still suffer from two problems: First, VLPs tend to rely on language biases in datasets and fail generalize to out- of-distribution (OOD) data. Second, they are inefficient in terms memory footprint and computation. Although promising progress has been made in both problems, most existing works tackle them independently. To facilitate the application of VLP to VQA tasks, it is im- perative to jointly study VLP compression and OOD robustness, which, however, has not yet been explored. This paper investigates whether a VLP can compressed and debiased simulta- neously by searching sparse and robust subnet- works. To this end, we systematically study the design a training and compression pipeline search the subnetworks, well as the assign- ment of sparsity to different modality-specifff modules. Our experiments involve VLPs, compression methods, 4 training methods, datasets and a range of sparsity levels. Our re- sults show that there indeed exist sparse and ro- bust subnetworks, which are competitive with the debiased full VLP and clearly outperform the debiasing SoTAs with fewer parameters OOD datasets VQA-CP v2 and VQA-VS.!
ODD Pariomance (Ace 12
0ure-50%%
(mmorppp)))
Oum 00% to
LPF-8>. OF-VQAsum
CF-VO #LPFsum UNH
VODE RUBi
DLR
Parameters (Millins)
Figure 1: Comparison of accuracy and model sizes with debiasing SoTAs on VQA-CP v2. The green and cyan lines represent our "Ixmert(Ipf) + mask train(Imh)" and "Ixmer((lmh) mask train(Imh)", respectively, with modality-specifs sparsity.
Although large-scale vision-language pre- trained models (VLPs) achieve further improve- ments in the in-distribution (ID) VQA benchmark (Goyal et al., 2017), they also fail to address the dataset-bias problem (Agrawal al., 2018), e.g., Ixmert (Tan and Bansal, 2019) suffers a 23.26% drop between ID and OOD accuracy. At the same time, the improvement brought by VLPs is partly due to their large model size, which increases the computational cost of deploying VQA models. To facilitate the application of VLPs to VQA tasks, the two problems should be addressed simultaneously. However, existing researches mostly focus on each of them separately.
1 Introduction
Visual Question Answering (VQA) (Antol et al., 2015) is an important task at the intersection of CV and NLP. In the last decade, deep neural networks have made promising progress in VQA. However, recent studies (Agrawal al., 2016; Manjunatha et al., 2019) have found that VQA models are prone to dataset biases. As a result, they always suffer from sharp performance drops when faced with out- of ff-distributinn (OOD) test datasets, whose answer distributions are different from the training set.
The dataset-bias problem VQA is well studied by numerous debiasing methods based on conven- tional small-scale models( Anderson et al., 2018; Cadene et al., 2019). Their main solution (Cadene et al., 2019; Clark et al., 2019; Liang et al., 2021b; Mahabadi and Henderson, 2019) regularize the loss according the bias degree of training sam- ples. In terms of the increased computational cost, a line of recent efforts have been made to compress pre-trained language models (PLMs) in the NLP
"Equal contribution. Corresponding author: Zheng Lin. 'The codes can he found at https: h/ti::///ttthtll thub com/ www.oussi/ooopsssseeesee

---

field (Chen et al., 2020b: Li et al., 2020a,b; Liang al., 2021a; Liu et al., 2021, 2022; Prasanna et al., 2020) and VLPs for visual-linguistic tasks (Fang et al., 2021; Gan et al., 2022). They show that large-scale PLMs and VLPs can be compressed into lightweight models without degrading perfor- mance. Refer to App. A for more related work.
guage encoder (Lene), object relationship encoder (Rene) and cross-modality encoder (Cenc).
We adopt unstructured pruning to obtain a com- pressed version (i.e., a subnetwork) of the origi- nal VLPs. Specifically, given a VLP f(0) with parameters A. we apply a binary pruning mask m E {0, 13.10 to the model parameters, which gives rise (m00), where + is the element-wise product. The parameters to be pruned are:
This paper jointly studies the compression and debiasing problems of VLP for the VQA task. To this end, we combine the existing debiasing and pruning methods to establish a training and com- pression pipeline, and conduct extensive experi- ments with the pre-trained Ixmert, which is the most popular VLP in VQA, under different OOD settings. We show that there exist sparse Ixmert sub- networks that are more robust than the full model, which suggests that the goal of OOD robustness and computational efficiency can be achieved si- multaneously.
0g -/ (Weab, W fis Wptrl O U Bese U 0 Xene
where Wemb. fiis-fo and Wplr are the weights of embedding layer, vision fe layer and pool layer, Rene U are the parameters of Trans- former layers. More details of Ixmert can be found in App. B.1. Another model visualBERT (Li et al., 2019), which is also used in our experiments, will be introduced in App. B.2.
LLene U R
We also present a comprehensive study on the design of the training and compression pipeline. as well as the assignment of sparsity to different model modules, to identify subnetworks with bet- ter OOD generalization. Our findings highlight the importance of 1) Employing two-stage training and compression pipeline and integrating the debi- asing objective throughout the entire process. 2) If there are two debiasing methods working well with the full model, training the full model with the relatively woor-pefforming one and compressing it with the better one. 3) Assigning modality-specifie sparsity to different modules of VLP.
2.2 Pruning Methods
We consider two representative pruning methods. i.e., magnitude-based pruning (Han et al., 2015) and mask training (Louizos et al., 2018; Ramanujan et al., 2020; Sanh et al., 2020; Sehwag al., 2020).
Magnitude-based Pruning approximates the im- portance of model parameters based on their abso- lute values and eliminates the less important ones. We adopt the basic version of maggitude-basee pruning, i.e., one-shot magnitude pruning (OMP). OMP can optionally be combined with further fine- tuning of the pruned subnetwork to recover the performance drop.
Our main contributions are follows: (1) We present the first (to our knowledge) systematic study on sparsity and OOD robustness for VLPs. (2) Our empirical studies on the training and com- pression pipeline and sparsity assignment can serve as valuable guideline for the future design of VLP subnetwork searching methods. (3) We obtain sub- networks that outperform existing debiasing So- TAs in terms of the trade-off between accuracy and model size on OOD datasets VQA-CP v2 and VQA-VS (see Fig. 1, Tab. and Tab. 2).
Mask Training directly optimizes the binary pruning mask m towards the given objectives. Specifically, each weight matrix W E Rdxxdo is as- sociated with two mask matrices, namely a binary mask m E 1) 1]dxxd and a real-valued mask m E xdo, In the forward propagation, m is computed from m through binarization:
if i 2 20
mij
(2)
else
2 Method
where & is the threshold. Then, the original weight matrix W is replaced with a pruned one m W.
2.1 VLP Architecture and Subnetworks
This section takes Ixmert as an example to intro- duce how we extract subnetworks. Lxmert contains an embedding layer, visual fe layer, a pooler layer, a VQA-specific classifier and a stack of Trans- former layers, which involve three encoders: lan-
When it comes to backward propagation, we follow (Liu et al., 2022; Mallya al., 2018; Radiya-Dixit and Wang, 2020; Zhao al., 2020) and use the straight- thhhoughhhh estimator (Bengio et al., 2013) to estimate the gradients of i using the gradients of

---

m, and then update in in i B is the learning rate.
70m
DC where
pp².
To prevent Ps from being ignored, LMH also adds an entropy penalty item R in the final loss:
We initialize in according to the magnitudes of the pre-trained weights Ixmert. This strategy is shown to be more effective than random initial- ization for pre-trained language models (Liu et al., 2022; Radiya-Dixit and Wang. 2020) and we also validate this in our experiments with Ixmert (see App. C.2). Specifically, mn is initialized as:
Limb =t t- (og(((pss)) + (1 f) log 6(Paa)) + (6)
RUBi adopts training strategy similar LMH to regularize the main model's ' probability, and uses standard cross-entropy as the training loss:
Pdccb softmar(Pmm 5(p))
if W i pruned by OMP else
Lrbbi
1 N
Σ
N
(de()) [ax]
(7)
m
(3)
a X
LPF measures the bias degree as aμ Pb [ax] to regularize the loss of the main model:
where a 2 1 is a hyper-parameter. At initialization, we set the threshold Φ 0.01 (any other value with the same order of magnitude should also be fine). To ensure that the subnetwork satisfies the given sparsity, & is re-computed every fm training steps.
Lipt N 20 (og-=x)log(s00 (on[ [ax]
(8)
where the 7/ is a tunable hype-paramette
2.3 Debiasing Methods
The deabising methods in VQA usually contain a main model and biased model. The biased model, which learns the language bias, is used to measure the training samples' bias degree and adjust the training loss for the main model. We experiment with SoTA debiasing methods. i.e., LMH (Clark al., 2019), RUBi (Cadene al., 2019) and LPF (Liang al., 2021b), of which LMH is widely studied for the OOD scenario of VQA (Chen et al., 2020a; Liang et al., 2020; Si et al., 2021) and NLU (Jia and Liang, 2017; McCoy et al., 2019; Schuster et al., 2019; Zhang et al., 2019). For comparison, we also describe the binary cross-entropy here. Binary Cross-Entropy (BCE) computes the cross-entropy between the predicted distribution Pm. (from main model) and the soft target score of each ground-truth as:
2.4 Problem Formulation
Given the pre-trained Ixmert f(Opt). our goal is to find subnetwork f (m @ fft) that satisfies target sparsity level s and maximizes the OOD performance:
maxm.0p fr. (Eoon (f (m @ By))), s.t.
laale T0p.l = (9)
where EOOD denotes OOD evaluation, Illo is the Lo norm and p0p.. is the total number of parameters in pr. This goal is achieved by searching the optimal m and A,t in model training and compression.
Eq. 9 only specifies the overall sparsity. In this work, we also explore finer- fmer-grained control over sparsity, which allocates different sparsity to differ- ent modules of Ixmert, given that the overall spar- sity is satisfied. Concretely, we consider three mod- ules from different modalities, i.e., the language module, the visual module and the cross-modality module. The constraint in the optimization prob- lem is then rewritten as
Lice = t- (oo((0(660.)) + (1 log(1 8(pm.)) (4)
where å denotes the sigmoid function.
Learned-Mixin +H (LMH) adds a biased model to learn biases during training, as follows:
|maaan lo OLmml
mmaalb (1 2x), 10x..
||mx
stt.
(1 s2),
[0vil]
(1 xx),
Paes sof softmar(log((P)) (h))log(p:))
10
1vvil [0
1 @x..el
(5)
10
+
ax
|0
g(h) softplus(w h)
(10)
where @ Lan O OLen
Uf W emb }. Ovis O RR (W visiice and x X.g. are model parameters of
where Рı and Pm the predicted distribution of
biased model and main model, respectively. g(h) determines how much trust the learned biases. based on Ixmert's last hidden representation h. Fol- lowing (Clark et al., 2019), we directly use the answers' frequency under each question type as
"We use the same pa in our implementation LMH. RUBi LPF, More details LMH can be found in App. B.3 'For simplicity, the pooler layer's parameters(0.5MM are not included in eq. 10. We directly set it to the target sparsity

---

62.5
60
57.3
Overall Acc
Overall Acc
Gap
loment DVPP bos
52.5
lomerttion
full binwnjso))
50.0
45.0
45
0.2
0.6
0.0 02
0.4
0.5
0.5
Sparsity
Sparsity
Figure 2: Results of subnetworks from the BCE fine-tuned Ixmert (left) and from the LMH fine-tuned Ixmert (right) on VQA-CP v2. "Ixmen(bce/lhh) denotes full model fine-tuning Stagel, "mask train(bce/Imh)" and "OMP" denote pruning Stage2. "bce/Imh ft" denotes further fine-tuning in Stage3. "Gap" denotes the improvement of mask train( bce/Imh) over full Ixmer((bce/Imb)), The shadowed areas denote standard deviations. These abbreviations are used throughout this paper. Detailed performance on three question types is shown App. C.1
the language module, visual module and cross- modality encoder, respectively. Len- mvis and mx are the binary masks for the three modules, respectively. 81,, sn and 8x are the target sparsity levels for the three modules, respectively.
3.1 Datasets, Model and Implementation
We conduct experiments on the OOD benchmarks VQA-CP v2 (Agrawal et al., 2018) and VQA-VS (Si et al., 2022b) that evaluate the robustness VQA systems, with the accuracy-based evaluation metric (Antol et al., 2015). A more detailed discus- sion of the difference between the two datasets is shown in Sec. 3.5. We thoroughly study the above three questions VQA-CP-v2, which is widely used in the literature on debiasing VQA systems (refer to Sec. 3.2, 3.3 and 3.4 ). Then, based on the findings, we further explore the more challenging VQA-VS VS (Si et al., 2022b) (refer to Sec. 3.5) ). For VLP, we adopt the lxmerr-base-uncased model (Tan and Bansal, 2019) released by huggingface (Wolf et al., 2020). All the results are averaged over 4 random seeds. More information of the model and implementation details are shown in App. B.4.
not otherwise specified, we set the sparsity of every weight matrix to target sparsity. For exam- ple, if = 70% and there is no modality-specific constraint, then all weight matrices are at 70% (uniform sparsity). If matrices in 0 Lan are at 50% sparsity, while 8æ and xx could be different mmodality-speciffe sparsity).
50%, then all weight
2.5 Training and Compression Pipeline
We define two notations: Fc()(O)) denotes training s(0) using loss L E {Loces Limh). (0)) denotes pruning f()) using method E (OMP, mask train) and loss & (if applicable), which outputs pruming mask m. A typical train- ing and compression pipeline involves three stages: Stagel: Full Model Fine-tuning. The pre- trained Ixmert ((0p.: is fine-tuned using loss L.
3.2 Effect of Compression on OOD Accuracy Subnetworks from BCE Fine-tuned Ixmert. We compress the BCE fine-tuned Ixmert using OMP and mask training and introduce either Lice or Limb in the pruning (for mask training) or fur- ther fine-tuning process (for OMP).
which produces f(0 (0))
Stage2: Model Compression. The fine-tuned Ixmert f(0)t) (t) is compressed and we get the sub- network f (m @ fc), where m = (O))() Stage3: Further Fine-tuning (optional). The subnetwork fmm @ ffr)) is further fine-tuned using loss L', and gets f(m f)))) =F (f( ()))..
The results are shown in the upper row of Fig. 2. We can derive several observations: 1) When no debiasing methods are used, the subnetworks of "mask train(bce)" and "OMP + bee ft" improve over the full Ixmert by 1.35% ~ 2.79%, even at up to 70% sparsity. This implies that Ixmert is overparameterized and pruming may remove parameters related to the bias features. 2) "mask train(Imh)" and "OMP+ + Imh ft" achieve further per- formance boost, exceeding full Ixmert by large margin (1 (11.05% ~ 14.02%). Since mask train- ing does not change the value of parameters, the
3 Experiments
In this section, mainly investigate three ques- tions: (1) How does compression affect Ixmert's OOD generalization ability? (2) How to design the training and pruning pipeline to achieve good pparsity-performnn trade-off? (3) How to assign sparsity to different modality -ppecfc modules?

---

01.5
63.5
STRSETT talnieen arerign ruck 13
02
Sparsity
Sparaiy
Figure Results Ixmert subnetwarks obtained from different training and com- pressing pipelines on VQA-CP v2. "ft" means further fine-tuning the subnet- works in Stage3.
Figure 5: Results of Ixmert subnet- works that adopt different debiasing methods in Stagel and Stage2 on VQA-CP
Figure 3: Results of Ixmert subnet- works fine-tuned with different de- biasing methods on VQA-CP v2.
results of "mask train (Imh)" implicate that the bi- ased "full Ixmert(bce)" already contains sparse and robust subnetworks (across 10% ~ 90% sparsity). 3) "mask train" outperforms "OMP" in general, which suggests that directly optimizing the subnet- work structure is more effective than debiasing compressed subnetwork by further fine-tuning.
has been shown to generally outperform OMP with further fine-tuning, Our main observations can be described from three perspectives:
First, it is recommended introduce the de- biasing loss across Stagel, Stage2 and (if appli- cable) Stage3. The reason is three-fold: 1) As shown by Fig. 4, the subnetworks at 10%, 30% and 70% sparsity levels have better performance when starting from Ixmert(Imh), as compared with the Ixmert(bce). At 90% sparsity, "Ixmert(Imh) + mask train(Imh)" underperforms "Ixmert(bce) mask train(Imh)" (see App. C.3 for reasons), but the Accuracy gap is small. Therefore, adopting Lumh in Stagel a better choice than Lives espe- cially when the subnetworks are not at extremely high sparsity. 2) As we discussed in the previous section, introducing Limh in the mask training pro- (Stage2) substantially outperforms Live for both Ixmert(Imh) and Ixmert(bce). 3) When both Stagel and Stage2 adopt the BCE loss, further fine- tuning the subnetworks with LMH loss in Stage3 can significantly boost the performance, as shown by the results of "Ixmert(bce) + mask train(bce)" w/o ft and w/ Imh ft in Fig.
Subnetworks from Ixmert Fine-tuned with De- biasing Methods. From the lower row of Fig. 2. we can find that: 1) For the full Ixmert, the OOD performance is obviously promoted with the LMH debiasing method. 2) Unlike Ixmert(bce) subnetworks, Ixmert(Imh) subnetworks do not ex- hibit significant improvement over the full model. However, the "mask train(Imh)" and "OMP Imh ft" subnetworks, which preserve the Ixmers(Imh)'s performance at up to 50% sparsity, can serve as an efficient alternative to the LMH fine-tuned full Ixmert. 3) "mask train(bce)" and "OMP + bee ft" clearly underperform their Imh counterparts, which suggests that it is important to use the debiasing method in pruning and subnetwork further fine- tuning even when the full model is already trained with the debiasing method.
Second, Stage3 unnecessary it adopts the same training objective as Stage2. Com- paring the blue and red (or cyan) bars in Fig. 4, we can see that further fine-tuning with the same training objective generally degrades the performance of "Ixmer((Imh) mask train(Imh)", "Ixmert(bce) mask train(Imh)" and "Ixmert(bce) mask train(bce)).. This phenomenon suggests that Stage3 can climinated to save computation cost.
Fig. 3 compares the subnetworks fine-tuned with LMH, LPF and RUBi. We find that: The subnet- works found using LMH consistently outperform those found by LPF and RUBi across different spar- sity levels. Therefore, to save computing resources, mainly use the best performing LMH in the following experiments and analysis.
3.3 Training and Compression Pipeline
In this section, we study the proper design of the training and compression pipeline, under the basic Framework deseribed in Sec. 2.5. Here we focus on the mask training compression method, as
Third, it is recommended to use different de- biasing methods in the two stages and leave the better one to Stage2. As shown in Fig. 5, al- though LPF and RUBi are less effective in debi-

---

62
12.
Overall (AcO
Overall (Ac(
Ixmertlace trainibce)
62.9
Ixmertjace) trainlimh)
36-
IxmerjJmh) mask train
lxmertibce)
lxmortōmm)
52.5
53
50-
0.2
0.4
6.4
0.6
0.6
0.0
0.5
Language Module Sparsity
Visual Module Sparsity
Cross m adality Module Sparsity
Figure 6: Results of subnetworks obtained by pruning the language (left), visual (middle) and cross-modality (right) modules. When pruning one module, the other two modules remain unpruned.
35
by Fig. 6, the compression of different modality- specific modules indeed exhibits different effects. When the full model is Ixmert(bce) (the orange and cyan lines), compressing the language cross- modality module has a positive effect on the OOD performance, and the accuracy generally improves as sparsity increases from 10% to 90%. By contrast, compressing the visual module results in inferior re- sults than compressing the other two modules, even if the number of remaining parameters is larger (note that the visual module has smaller number of parameters than the other two modules). These results suggest that, for the biased Ixmert(bce), the language and cross-modality modules capture more training set bias than the visual module, which sup- ports the above hypothesis.
Overall ACC
42
Sparsity
Figure Results visualBERT subnetworks that adopt different debiasing methods in Stagel and Stage2 VQA-CP v2.
asing the full model than LMH, "Ipf+lmh"* and "rubi+Imh" are superior "Imh+lmh". In con- trast, when reversing the dehiasing methods used in the two stages, "Imh+rubi" and "Imh+Ipf" exhibit worse performance, suggesting that the better debi- asing method should be used in Stage2. Addition- ally, "Ipf+Imh" is superior to "rubi+Imh", which indicates that using better debiasing objective in Stagel is helpful when we have multiple choices different from the Stage2 objective. We also exper- iment with another VLP model, visualBERT (Li et al., 2019), and find that "Ipf+Imh" still performs the best as in Fig.
In terms of "Ixmert(Imb) + mask train(Imh)" (the red line), although compression does not lead to performance improvement like compressing Ixmert(bce), the results also demonstrate that the language and cross-modality modules are more compressible than the visual module.
Searching for Appropriate Modality-specine Sparsity. Motivated by the above findings, we search for appropriate modality-speciffe sparsity by performing mask training with variety of sparsity configurations (see App. C.4) for the three mod- ules while keeping the overall sparsity the same.
3.4 Modality-speeiffi Sparsity
As we can see in Fig. 8, at 50% and 70% overall sparsity, the configuration that achieves the best re- sult assigns slightly higher sparsity to the language and cross-modality modules and significantly lower sparsity to the visual module, compared with uni- form sparsity. This phenomenon is in accordance with the findings in Fig. 6, implicating that com- pressing the three modules uniformly suboptimal (at 50% ~ 70% sparsity) and the language and cross cross-mouee modslioy modules should be compressed to
Pruning Each Single Modality-specific Module. Since Ixmert uses different modules to encode the multi-modal data, it is intuitive to hypothesize that different modules of Ixmert may capture the lan- guage bias to different extents. To validate this hypothesis, we compress the language, visual and cross cross-modality modules, respectively. As presented
"Ipf+Imh" denotes "Ixmert(lf)) + mask train(Imh)"

---

Sparsht 50%
Sparshy 70%
Sparstt 90%
Iemortilmh)-mae trainc/mh)
0.2
60% 50%)
50% 38%
Figure 8: Results of subnetworks pruned by different sparsity configurations on VQA-CP using "Ixmert(Imh) mask train(Imh)". Red and blue lines denote the coordinates of the data point with uniform sparsity across three modules and the data point performing the best (the specific configuration is shown below each plot) respectively. The overall sparsities are shown in the titles.
Figure 9: Comparison of differ- ent spansity assignments on VQA- CP v2. "Gap" the gap between "uniform sparsity" and "modality- specific sparsity".
lomert(/ce)) mask train(bce)
lameettbbce) mask train(hce)
10
[GDD>men
Modde
22
34
Figure 11:
Comparison of "Ixmert((mh) mask train(Imh)" subnetworks with uniform and modality- specific sparsity on VQA-VS. Results on specific OOD test sets can be found in App. D.2
(xmertiimh) mask train/lm))
kmmet(/mml mask train(inh)
12
16
(10 AGC3
LOO00
Langugg Modda
Coos- Mods-ly Media
modality-specifhe sparsity achieves the best results across the three overall sparsity levels from 50% to 90%, demonstrating its superiority, Besides, the results also suggest that, although simply allowing different matrices to have different sparsity is more flexible than uniform sparsity, it is not conducive to the final performance.
Figure 10: Results of subnetworks pruned using BCE (upper) and LMH (lower) VQA-VS. We report ID and OOD-mean accuracy here. Results on specific OOD test sets are deferred App. D.1 Different lines denote subnetworks obtained by pruning all, language, visual and cross- modallty modules respectively.
3.5 Exploration on VQA-VS
VQA-CP v2 widely used in the literature of debi- asing VQA systems. However, it only considers the guestion-type--asel bias. To account for other po- tential biases, VQA-VS constructs several types of OOD test sets according to different shortcuts (e.g.. keyword and key object). As result, VQA-VS is more challenging and allows sults on different biases. In this section, we search sparse and robust Ixmert subnetworks in VQA-VS based on the major findings obtained from VQA- CP v2.
a larger extent than the visual module. At 90% sparsity, the sparsity configuration's 's comfort zone is in the proximity of the uniform point. Further increasing the sparsity of the language and cross- modality modules result in performance decline or only minor improvements. This because 90% sparsity already approaches the compression upper bound, even for the language and cross-modality modules.
analyze the re-
Fig. 9 shows a more direct comparison between the uniform and modality-specine sparsity. We also introduce another baseline "matrix-spccific sparsity", which ranks all the model parameters, in- stead of the parameters in each weight matrix. This also results in different sparsity levels for different weight matrices, while there is no explicit control over the modality-speciion sparsity. We can see that
The Effect of Compression. Fig. 10 shows the results of full Ixmert and subnetworks on VQA-VS. We can see that: 1) When using the BCE objective, we can identify sparse "bce+hee" subnetworks that are comparable with full Ixmert (bee). 2) Different from VQA-CP v2, full Ixmert (Imh) only slightly

---

Prams
A11
VX%
Sao
Milhials
Rackbare
Pmnms.
10
OXiT-mann
Cross Eniropy
S S-MIL
-00
62.03
62.65
CPVQA Six st 2011
VGO0 and Mand 2070
LIF GLARS ci S0cts)
5 MRT.
S-M0L
S-M0LL
-0M
-004
93:11
53:55
99.61
55.00
25.00
17.06
21.30
-6.77
43.51
-2.99
Chros Eatiopy
KUBI (Cadene 20199
S-MRL
600
151
65:0
59,09
25,80
38.73
LPF (Liingg al. 2021b)
UpDu
354
54.72
43.31
GRL (G and and Selatoon 2014)
Eybn
158
43.33
59-76
14.75
<0.76
LMI (Clark al. 2019)
Upiee
35M
56.89
45.85
RUB: [Calose al. 2015)
0ybn
1581
48:33
17.08
30.61
BCC
full Ixmert
2023
7224
51.92
VGGDE, (Kx and basae 2010
DLR Dsgg o di, 20000
025m
Upbn
1351
1583
48.0
45.25
70.99
18:72
28.57
LPF
RUII (Cadeee 2019)
2021b)
full Ixsse1
full homert
202M
202M
69.19
68.48
50.83
50.07
0jb0
31.12
LMH (Clayk al.,
Ixnet
202M
70.22
(Nis 2021)
07.28
LJW Jisse al, Santhn
0ytn
1951
11.54
30.61
73.78
96.57
70.89
LXBII-MMMB
300ta)
1yDn
14
41.77
-9.7
(Ours)
H6M
71.38
BSC
CGE
LPF (Cark at 2013)
Cal batert
2033
3014
45:01
03.68
48.34
97.57
3018
51.06
82.88
55.57
(Ours)
(Ours)
lsment
143M
71.85
71.85
(Clrk 2019)
63:55
Imbelmh (Ours)
Imnent
5543
Imb+lmh (Ours)
Isment
69.34
(ylslah (Dan)
Ipdsttch (Duno)
2019)
Call IPLUG
975 Texust
91% Ieruet
20034
HCM
6151
6160
5145
66.0
$4.70
79.99
61.38
61711
96.38
6..6
Imhhllhh (Ours)
Imb+lmh (Ours)
Imb+lmh (Ours)
Iunert
Iunert
Itnent
16M
143M
18M
70.66
71.56
71.97
54.34
$4.75
54.31
(Our)) (Dkm))
--95 I
187M
43.6
Table 2: Comparison with debiasing SoTAs on VQA- VS. The subnetworks are pruned using modality- specific sparsity. "bce+bee" and "Imh+Imh" are defined in the same way as Tab.
Table I: Comparison with debiasing SoTAs on VQA-CP v2. "Ipf+lmh" denotes "Ixmert(Ipf)+ + mask train(Imh)" subnetworks with modality-speciff sparsity. "10% Ixmert" denotes keeping 10% parameters Ixmert. The subnetworks from mPLUG are pruned using uniform sparsity.
asing SoTAs with 1 1.55% and 5. 5.79%, respectively, with fewer or similar amounts of parameters, estab- lishing new state-of-theearts. 2) Our methods (30% Ixmert and 50% Ixmert) outperform the debiased full Ixmert, even with much fewer parameters. 3) Full Ixmert(Ipf) and full Ixmert(Imh) are good at different question types, which can partly explain why combining them in different stages produces more robust subnetworks.
outperforms full Ixmert (bce) in the OOD setting of VQA-VS, and underperforms in the ID setting. 3) The "Imh+Imh"S subnetworks improve over full Ixmert (Imh) on both ID and OOD test sets, across a wide range of sparsity levels, suggesting that Ixmert can also be simultancously compressed and debiased on VQA-VS.
We also add experiments on a more recent VLP mPLUG (Li et al., 2022). We adopt the base version of mPLUG, fine-tune it on the VQA-CP v2 training set and then conduct pruning using mask training. Since mPLUG formulas VQA as text generation task, we adopt the LPF debias- ing method. Note that LMH and RUBi cannot be directly applied to debias text generation models, because they are designed for classification loss over a fixed number of classes. As shown in the bottom rows of Tab. 1, the mPLUG trained with standard cross-entropy (CE) loss can be simulta- neously compressed (to 50%) and debiased (+5.48 Acc). The mPLUG trained with LPF debiasing loss can also be compressed to 50% with slight accuracy decline. These results demonstrate that the findings and techniques present in our work can be generalized to more advanced VLPs.
The Effect of poddlltty-speciffe Sparsity. Fig. 10 also shows that compressing different modality- specific modules has different effect on VQA-VS, as in VQA-CP v2. The language module is the most compressible while compressing the visual module results in the sharpest performance decline. To compare modality-specifii spansity and uniform sparsity, we directly inherit the sparsity configura- tion selected in Sec. 3.4 on VQA-CP v2. Fig. 11 shows that modality specific sparsity consistently outperform uniform sparsity, except for 90% spar- sity in the ID setting.
3.6 Comparison with Debiasing SoTAs
In this section, we will compare the best training and compression solutions identified in the pre- vious sections with the current SoTA debiasing methods.
Results on VQA-VS are presented in Tab. 2. We can observe that: 1) Our methods "bce+bee" 10% Ixmert and "Imh+Imh" 30% Ixmert outper- form all the non-VLP debiasing methods in both ID and OOD settings, with similar or fewer pa- rameters. 2) Except for LMH, other debiasing methods underperform BCE in OOD-mean. LMH improves the OOD accuracy at the cost of ID ac- curacy decline. 3) The "Imb+Imh" subnetworks
Tab. 1 shows the results on VQA-CP v2. We find that: 1) The accuracy of our methods (10% Ixmert and 30% Ixmert) beats the previous non- VLP debi-
"Since most debiasing methods (c.g.. LPF and RUBi) fail on VQA-VS (see Tab.2), we only use LMH in VQA-VS. How- ever, combining LMH and other effective debiasing methods in different stages may further outperform "Imh+lmh", as found in VQA-CP v2. We leave it for future work.

---

(even with 50% remaining parameters) obviously improve the ID performance of Ixmert (Imh) and re- tain comparable OOD performance. 4) Compared with "bce+bce", the OOD advantage of "Imb+lh"" outweighs its ID disadvantage at 50% to 90% pa- rameters. With fewer remaining parameters, the overall performance of "bce+bee" is superior.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages
4 Conclusion
To facilitate the application of VLP-based VQA systems, this paper presents the first joint study on the compression and debiasing problems of VLP for the VQA task. Through extensive ex- periments with three VLPs (i.e., Ixmert, visual- BERT and mPLUG), we analyze the impact of compression on the OOD generalization ability. We present comprchensive study on the design of the training and compression pipeline for a good sparstty-erfformance trade-off, and provide some valuable findings about the assignment of sparsity to different coodality-ppeciii modules. The com- pressed Ixmert subnetworks in this paper outper- form the SoTA debiasing methods with fewer or similar model parameter counts.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computa- tion. CoRR, abs/ 100./333.33..
Remi Cadene, Corentin Dancette. Matthieu Cord, Devi Parikh, et al. 2019. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32:841-852.
Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shil- iang Pu, and Yueting Zhuang. 2020a. Counterfactual samples synthesizing for robust visual question an- swering. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10800-10809.
Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020b. 'he lottery ticket hypothesis for pre- trained BERT networks. In NeurIPS, pages 15834- 15846.
Limitations
Although we have empirically verified that the adoption of modality-specifie sparsity is beneficial for the search for more robust subnetworks, our work still does not provide a solution on how to determine the optimal sparsity assignment effec- tively and efficiently. We invite follow-up studies to further address it in future work.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arxiv:1504.00325.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint mwxiv:1909.0300
Acknowledgement
Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. 2022. Mukea: Multimodal knowledge extraction and accumulation for mnowledge-baser visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition (CVPR), pages 5089-5098.
This work was supported by National Natural Sci- ence Foundation of China (No. 61976207) and National Social Science Foundation of China (No. 21AZD145).
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang. Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. 2022. An empirical study of training end-to-end vision-and- aaiio-aad-language transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18166-1817 1176.
References
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question an- swering models. arXiv preprint arXiv: 07356.
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018 Don't 't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 4971-
Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, and Ahmed Hassan Awadallah. 2021. What do compressed large language models forget? robustness challenges in model compression. CoRR, abs/21 10.08419.

---

Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. 2021. Com- pressing visual-lingustii model via knowledge dis- tillation. In ICCV, pages 1408-1418. IEEE
Xinzhe Han, Shuhui Wang. Chi Su, Qingming Huang. and Qi Tian. 2021. Greedy gradient ensemble for ro- bust visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1584-1593.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR. OpenReview.net.
Geoffrey E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural Com- pur. 14(8):1771-1800
Yonggan Fu, Qixuan Yu, Yang Zhang, Shang Wu, Xu Ouyang, David D. Cox, and Yingyan Lin. 2021. Drawing robust scratch tickets: Subnetworks with in- born robustness are found within randomly initialized networks. In NeurIPS, pages 1 --059-13072
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.072:
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang. and Qun Liu. 2020. Tinybert: Distilling BERT for natural language under- standing. In EMNLP (Findings), pages 4163-4174.
Trevor Gale, Erich Elsen, and Sara Hooker. 2019. The state of sparsity in deep neural networks. CoRR, abs/1902.09574.
Chenchen Jing, Yuwei Wu, Xiaoxun Zhang, Yunde Jia, and Qi Wu. 2020. Overcoming language priors in vqa via decomposed linguistic representations. Proceedings of the AAAI Conference on Artificial Intelligence, pages 11181-11188.
Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, Jingjing Liu, Lijuan Wang, and Zicheng Liu. 2022. Playing lottery tickets with vision and language. In AAAI, pages AAAI Press.
Gouthaman Kv and Anurag Mittal. 2020. Reduc- ing language biases in visual question answering with visually-grounded question encoder. In Euro- pean Conference on Computer Vision, pages 18-34. Springer.
Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. Mutant: training paradigm for out-of-distribution generalization in visual question answering, arXiv preprint arXiv:2009.0m 08566.
Zhenzhong Lan, Mingda Chen. Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sorieut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR. Open- Review.net.
Mitchell A. Gordon, Kevin Duh. and Nicholas An- drews. 2020. Compressing BERT: studying the ef- fects of weight pruning on transfer learning. In RepLANLP@ACL pages 143-155.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang. Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guo- hai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. 2022. mplug: Effective and efficient vision-language learning by cross-modal skip-connections.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6904 6913.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty. Caiming Xiong. and Steven Chu Hong Hoi. 1021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems. 34:9694-9705.
Gabriel Grand and Yonatan Belinkov. 2019, Adver- sarial regularization for visual question answering: Strengths, shortcomings, and side effects. NAACI HLT 2019, page I.
Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. 2019. Model compres- sion with adversarial robustness: A unified optimiza- tion framework. In NeurlPS, pages 1283-1294.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint 10X.v:190..03557
Yangyang Guo, Liqiang Nie, Zhiyong Cheng, Qi Tian, and Min Zhang. 2021. Loss re-scaling vqa: re- visiting the language prior problem from a class- imbalance view. IEEE Transactions Image Pro- cessing, 31:227-238.
Wei Li, Can Gao, Guocheng Niu. Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2020a. Unimo: Towards unified-modal understand- ing and generation via cross-modal contrastive learn- ing. arXiv preprint 2rXiv:2012.15409
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in Neural In- formation Processing Systems 28, pages 1135-1145 1143. Curran Associates, Inc.
Zhuohan Li. Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. 2020b. Train large. then compress: Rethinking model size for efficient training and inference of transformers. CoRR, abs/2002. 11794.

---

Chen Liang, Simiao Zuo. Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. 2021a. Super tickets in pre-trained language models: From model compression im- proving generalization. In ACI/JJCNLP pages 6524- 6538. Association for Computational Linguisties.
Paul Michel, Omer Levy, and Graham Neubig, 2019. Are sixteen heads really better than one? In NeurlPS. pages 14014-14024.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian- Mlan-Sheng Hua, and Ji Ji-Rong Wen. 2021. Counter- factual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Putten Recognition, pages 12700- 12710.
Zujie Liang. Haifeng Hu, and Jiaying Zhu. 2021b. Lpf: A language-prior feedback objective function for de- biased visual question answering. arXiv preprint arXiv:2105.14380
Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT plays the lottery, all tickets are winning. In EMNLP, pages 2228-3229.
Zujie Liang Weitao Jiang. Haifeng Hu, and Jiaying Zhu. 2020. Learning to contrast the counterfactual samples for robust visual question answering. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 2285-3292
Evani Radiya-Dixit and Xin Wang. 2020. How fine can fine-tuning be? learning efficient language models. In AISTATS, volume 108 of Proceedings of Machine Learning Research, pages 2435-2443. PMLR.
Yuanxin Liu, Zheng Lin, and Fengcheng Yuan. 2021. ROSITA: refined BERT compression with integrated techniques. In AAAI, pages 8715 8722. AAAI Press.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Ste- fan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. arXiv preprint 180::180000364.
Yuanxin Liu, Fandong Meng, Zheng Lin, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. 2022. Learning to win lottery tickets in BERT transfer via task-agnostic mask training. CoRR, abs/2204.112118
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kem- bhavi, Ali Farhadi, and Mohammad Rastegari. 2020. What's hidden in a randomly weighted neural net- work? In CVPR, pages 1 11890-11899, Computer Vision Foundation IEEE.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.0510..
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91-99.
Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through I_0 regularization. In ICLR (Poster). OpenReview.net.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019, Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, 191000000100
Rabeeh Karimi Mahabadi and James Henderson. 2019. Simple but effective techniques reduce biases. arXiv preprint 1909.06399. 9.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. 2018. Piggyback: Adapting a single network to mul- tiple tasks by learning to mask weights. In ECCV, volume 11 1208 of Lecture Notes in Computer Science, pages 72 88. Springer.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: Adaptive sparsity by fine- tuning. In NeurIPS, pages 20378-20389.
Tal Schuster. Darsh J. Shah, Yun Jie Serene Yeo. Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In EMNLP/IJCNLP, pages 3417-3423. Association for Computational Linguistics.
Varun Manjunatha, Nirat Saini, and Larry S Davis. 2019. Explicit bias discovery in visual question answering models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9562-9571.
Vikash Schwag, Shiqi Wang, Prateck Mittal, and Suman Jana. 2020. HYDRA: pruning adversarially robust neural networks. In NewrlPS.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mouaggii 2019. Ok-vqa: A visual ques- tion answering benchmark requiring external knowl- edge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204.
Qingyi Si, Zheng Lin, Mingyu Zheng, Peng Fu, and Weiping Wang. 2021. Check it again: Progressive vi- sual question answering via visual entailment. arXiv preprint arXiv: 2 146004
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- ties in natural language inference. In ACL, pages Association for Computational Linguis-
Qingyi Si. Yuanxin Liu, Fandong Meng. Zheng Lin, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. 2022a. Towards robust visual question answering: Making the most of biased samples via contrastive learning. ArXiv, abs/2210. 10.04663.
tics.

---

Qingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin. Yuanxin Liu, Peng Fu, Yanan Cao. Weiping Wang, and Jie Zhou. 2022b. Language prior is not the only shortcut: A benchmark for shortcut learning in vqa. arXiv preprint arXiv:2210.0455
Shaokai Ye, Xue Lin, Kaidi Xu, Sijia Liu, Hao Cheng, Jan Kan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, and Yanzhi Wang. 2019. Adversarial robustness vs. model compression, or both? In ICCV, pages 111-120. IEEE
Qingyi Si, Yuchen Mo. Zheng Lin, Huishan Ji, and Weiping Wang. 2023. Combo of thinking and ob- serving for kntwidee vqa. arXiv preprint arXiv:2305.6440
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng. Luowei Zhou, and Pengchuan Zhang- 2021. Florence: A new foundation model for com- puter vision. arXiv preprint arXiv:2111.1132.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019, Patient knowledge distillation for BERT model com- pression. In EMNLP/IJCNLP, pages 4322-4331.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: quantized Sbit BERT. In EMC2@ @ NeurlPS, pages 36-39, IEEE.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprint 1rXiv:1908.000
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. Vinvl: Revisiting visual representa- tions in vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NIPS, pages 1998-6008.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying ar- chitectures, tasks, and modalities through a simple sequence-to-sequenee learning framework. ICML.
Wei Zhang. Lu Hou, Yichun Yin. Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Ternarybert: Distillation-awarr ultra-low bit BERT. In EMNLP, pages 509-521. Association for Computational Lin- guistics.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. 2021a. Vlmo: Unified vision- language pre-training with mixture-of of- modality- arXiv preprint arXiv:2111.02355.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling. In NAACL-HLT, pages 1298-1308. Association for Computational Linguistics.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- ai Tsvetkov, and Yuan Cao. 2021b. Simvim: Simple visual language model pretraining with weak super- vision. arXiv preprint arXiv:2108: 10904.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hin- rich Schütze. 2020. Masking as an efficient alterna- tive to finetuning for pretrained language models. In EMNLP, pages 2226-2241.
Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. 2021. Debiased visual question answering from feature and sample perspectives. Ad- vances in Neural Information Processing Systems, 34:3784-3796.
A
More Related Work
A.1 Overcoming Dataset Bias in VQA
Most VQA systems heavily rely on the information of the question to predict answers no matter the content of the given image. That is they learned the language biases in datasets. They are not robust and always perform poor in the OOD setting where the language biases they learned in training set are invalid for test set. To promote the development of models that overcome such problem, VQA-CP v2 (Goyal et al., 2017) is proposed and has become the standard OOD benchmark in VQA. Currently, the widely used debiasing methods can be roughly grouped into non-data-augmennt 2019; Liang et al., 2021b; Mahabadi and Hender- son, 2019) and augaeetgmenttnionn methods (Chen et al., 2020a; Gokhale et al., 2020). The former ap- plies a biased model (trained with question only) to
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online.
(Clark et al.,
Canwen Xu, Wangchunshu Zhou, Tao Ge. Ke Xu, Ju- lian J. McAuley, and Furu Wei. 2021. Beyond pre- served accuracy: Evaluating loyalty and robustness of BERT compression. In EMNLP (1), pages 10653- 10659. Association for Computational Linguistics.

---

regularize the model training and thus prevent learn- ing from question. The latter generates samples to balance the training data and directly erase the bi- ases in the training set. However, the augmented data also increase the training cost, and overcoming the language-biar problem remaining the original dataset biases unchanged still remains major chal- lenge (Liang et al., 2021b; Niu et al., 2021). Thus, we only focus on non-data-augmennttion methods, such as LMH (Clark et al., 2019), RUBi (Cadene et al., 2019) and LPF (Liang et al., 2021b). Very recently, VQA- V (Si et al., 2022b) is proposed to explore the varying types of dataset biases. We also use this dataset to study how the training and compression pipeline affect different dataset biases.
hypothesis (Frankle and Carbin, 2019), many re- cent studies show that BERT can be pruned to a sparse subnetwork after (Gale et al., 2019) and be- fore fine-tuning min-ttnnng (Chen et al., 2020b; Liang et al., 2021a; Liu et al., 2022; Prasanna et al., 2020), with- out performance degrading. On this basis, we ex- tend the pruning paradigm to the fine-tuned Ixmert for OOD scenario in VQA, which incorporates the debiasing methods when fine-tuning and pruning. In the NLP and CV fields, some recent efforts have also been made to study model compression and robustness to adversarial attacks (Fu et al., 2021; Gui et al., 2019; Sehwag et al., 2020; Xu et al., 2021; Ye et al., 2019) and spurious correlations (Du et al., 2021; Xu et al., 2021) (which is more common than the worst-case adversarial attack). Dataset- bias problem is typical symptom of spu- rious correlations and poses challenge VQA models. We are the first to thoroughly investigate the sparsity and OOD robustness for VLPs VQA.
A.2 Vision-Language Pre-trained Models
Recently, VLPs (Dou et al., 2022: Li et al., 2021, 2020a; Wang et al., 2021a,b; Zhang et al., 2021; et al., 2023; Li et al., 2022) based on the Transformer backbone (Vaswani et 2017) have achieved encouraging success. Specially, OFA (Wang et al., 2022) and Florence (Yuan et al., 2021) establish the SoTA on the in-distribution VQA v2.
B More Details of Model and Implementation
B.1 Ixmert Architecture and Subnetworks
To learn better cross- pross-modality representations and
vision-language alignment, they are trained with large-scale pre-training data and generally have huge model capacity. Among them, Ixmert (Tan and Bansal, 2019) is the most widely used VLP as the backbone model in VQA field (e.g., some data- augmentation debiasing methods (Gokhale et al., 2020: Si et al., 2021; Wen et al., 2021) and the open-domain VQA (Marino et al., 2019) method MuKEA (Ding et al., 2022)). In this paper, therefore mainly use Ixmert as the backbone model and extend several debiasing methods it for in- depth research on compressing and debiasing. For completeness, we also conduct experiments on the popular VLP visualBERT (Li et al., 2019).
For Ixmert, the embedding layer and visual fc layer map language-modality input (token sequences obtained by WordPiece tokenizer) and vision- modality input (36 object fcatures obtained by Faster R-CNN (Ren et al., 2015)) into the same- dimension space. The pooler layer connects the Transformer top layer and the classifier. The Trans- former layers involve three encoders 7: language encoder (Lenc), object relationship encoder (Renc) and cross -roos-tydality encoder (Cene), and are usually composed of attention modules and feed-forward networks (FFN).
The attention modules have four kinds of weight matrices, i.e., the query, key and value matrices W W....V C Rdmada Xdmatel smoddl and the output matrix Wo Rddanse Ddeense FFN contains two linear lay- ers Win in E x Wout out ddrreddmakt
A.3 Model Compression and Robustness
Model compression techniques for Transformer- based pre-trained models are well developed (mainly around BERT), including pruning (Gale et al., 2019; Gordon et al., 2020; Michel et al., 2019), knowledge distillation (Jiao et al., 2020; Sanh et al., 2019; Sun et al., 2019), parameter shar- ing (Lan et al., 2020) and quantization (Zafrir et al.. 2019; Zhang et al., 2020). Inspired by lottery ticket
We adopt unstructured pruning to obtain a com- pressed version (i.e., a subnetwork) of the origi- nal VLPs. Specifically, given a VLP f()) with
2 'aah Transformer layer of the language encoder and ob- ject relationship encoder has a multi-head self attentit mod- ule and a feed-forward network (FFN), Each Transformer layer of the cross-modality encoder has a language self-attention module, visual self-attention module and multi-head cross- attention module. Only the language self-attention and visual self attentttnnion modules are followed by FFN, All the weight matrices of Transformer layers are summarized in eq. 12.
"Both VQA-VS and VQA-CP v2 datasets are licensed under Commons Artribution 4.0 International License.

---

15
30
Aci. Gap
VIN ACC
Num AcE
%
12.5
30.0
brembol nast cokenol brenttal 200
15
Sparsity
Sparsity
Sparsity
Sparsity
20
90-
Ace G3D
Other ACC
VIN
Imotl mit mest totestics call nosk coarene:
50
lorat mil Gee Inst
6.
0.2
2-5
01.6 Sparsity
0.0 008
0.2
10.0
Sparsity
Sparsity
Sparsity
Figure 12: Results of subnetworks from the BCE fine-tuned Ixmert (upper) and from the LMH fine-tuned Ixmert (lower) on VQA-CP v2. "Ixmert(cce/lmh) denotes full model fine- lnnettnnig in Stagel, "mask train( bee/Imh)" and "OMP" denote pruning in Stage2. "bce/lmh ft" denotes further fine-tuning Stage3. "Gap" denotes the improvement of mask train(bce/Imb) over full lxmer(bbee/lmh).
parameters 0, apply binary pruning mask E (0, 1)101 to the model parameters, which gives rise to f'm @ 0), where @ is the element- wise product. For Ixmert, we focus on the em- bedding layer, visual fe layer, pooler layer and Transformer layers of which the parameters are pre-trained, while the classifier is excluded. The language encoder, visual encoder, mross-modality encoder have T, and X Transformer layers re- spectively. The parameters be prumed are:
layer, a stack of Transformer layers. Differently, visualBERT's Transformer layers only involve a single encoder (Venc). The parameters of visual- BERT to be pruned are:
Ppr = [ (W cmb, Wplr U @V....
(13)
where W and W plr the weights of embed- ding layer and pool layer, 0 Venc are the parameters of Transformer layers:
Vinc (We Wke Wi. Wo. War: W 1
0p- = (Wont abbi Wiisee Wptel U frnn. U U Xana
ouz; a=1
(11)
(14)
where W Wemb. Wvis-fe and W pir are the weights of embedding layer, vision fc layer and pool layer, 0 Lene U Rene 0 are the parameters of Trans- former layers:
where V 12.
B.3 LMH details
LMH takes a step further based on Produce of Ex- perts (PoE) (Hinton, 2002), which simply combines the predicted distributions of the main model and the biasd model as follows:
0 Line (W W Wi: Wa, W Wi on )
Wongdom W GRR W Wky Wog W W !
0
X
Woxx WT Kox Whox Woox
WT
Qeo
WiO Wi Woo W Best We
outer? 1x
Pacs (m) + log(ps))
(15)
W
We WER Wo We W
(12)
where Pa is the predicted distribution of biased model, and indicates the bias degree of the sample. In this way, when sample is heavily biased, that is, is large, the main model will not output a large Pm for it during training. Following (Clark al., 2019), we directly use the answers' frequency under each question type as Pb-
where CX, CL and CR the language self- attention, visual self-attention and cross-attention modules respectively.
B.2 visualBERT Architecture and Subnetworks
Similar to Ixmert, visualBERT is composed of an embedding layer, a visual projection layer, a pooler
To selectively adjust the main model's behavior, LMH adds a learn function g to explicitly deter-

---

mine how much trust the learned biases:
Pdeb so (lgg + ((h))og(Pb)) (16)
60
g(h) so fopluss((w
where h is the cross-modality representation from the last hidden layer of Ixmert, w is trainable. To prevent Ps being ignored, LMH also adds an en- tropy penalty item R, and the final loss is computed as:
50-
brawtttoo) -aMerrani
knartiin1 bursten
braamttin ransint iin most mivigin my broeniedl + madi cou-aneea
bmentt mnd
0.0
0.2
0.4 Sparsity
0.6
0.8
Cinh 2 (og()((ans)) (1-99) -0)-logl -lg(11 5(PaaMM +
Figure 13: Results of subnetworks obtained by mask training with different initialization strategies of ın VQA-CP v2. "rand-init" means initializing in randomly.
(17)
B.4 Model and Implementation Details
Lxmert has about 202M parameters, and 197.7M parameters are involved in the pruning process (4.5M parameters are left to the classifier). The three modules from different modalities, namely the language module, the visual module and the cross-modality module, contain IM, 35.3M and 78.8M parameters respectively. We train the mod- els for 20 epochs with batch size of 128 on two Tesla-V100-326 or 256 on A 100088GB.. The AdamW (Loshchilov and Hutter, 2017) optimizer is adopted with a learning rate of 5e-5. Our codes are based on the huggingface transformers library (Wolf et al., 2020). We adopt visualBERT of its coco-pre version which is pre-trained with COCO (Chen et al., 2015) dataset.
in training set. Therefore, "Other" questions re- quest more reasoning than debiasing. 3yThere a sharp decline all the subnetworks' performance on "Other" questions from 70% ~ 90% sparsity. We conjecture that this is because reducing the model's capacity too drastically hurt the reasoning ability which is necessary to answer the "Other" questions correctly.
Subnetworks from LMH Fine-tuned Ixmert. The right three plots of Fig. 12 (lower) shows the performance of LMH fine-tuned Ixmert sub- networks on different types of questions. For the "Num" questions, when compressing LMH fine- tuned Ixmert (the grey and maroon lines), the per- formance of subnetworks no longer rises with spar- sity growth. This demonstrates that language biases for the "Num" questions exist in much smaller proportion of the parameters of debiased Ixmert than that of biased Ixmert. For "Other" questions, "Ixmert(bce) mask train(Imh)" is consistently perior to "Ixmert(Imh) mask train(Imh)", which demonstrates that further debiasing the debiased full Ixmert in the pruning process sacrifices the reasoning ability.
C More Experiments on VQA-CP v2
C.1 Performance of Subnetworks on Three Types of Questions
Subnetworks from BCE Fine-tuned Ixmert. For the three types of questions, shown in the right three plots of Fig. 12 (upper), we find that: 1) The performance "Num" questions is sensitive the varying sparsity levels while that on "Y/N" questions is relatively stable in general except at 90% sparsity. Specially, with the increase of spar- sity, the performance on "Num" questions of "mask train(Imh)" and "OMP Imh ft" counterintuitively greatly promote. This shows that language biases for the "Num" questions exist in large propor- tion of the parameters of biased Ixmert. 2) For the "Other" questions, debiasing methods have little gain on the performance of subnetworks. For exam- ple, the performance of "mask train(Imh)" similar with that of "mask train(bce)". This indicates that the language biases for "Other" questions minor
C.2 The Effect of Different Initialization Strategies of m for Mask Training
We conduct experiments with different subnet- works to validate the effectiveness of initializing m according to the magnitudes of Ixmert's pre- trained weights. From Fig. 13, it can be seen that "Ixmert(bce) + mask train' "Ixmert(bce) mask train( "Ixmert(Imh) mask train(bce)" (dashed lines) consistently outperform "Ixmert(bce) + rand-ini init mask train(bce)". "Ixmert(bce) rand- init mask train(Imh)", "Ixmert(Imh) + rand-init

---

First, we traverse [10%, 30%, 50%, 70%, 90%] (i.c., step size of 20%) assign mppelity- sparsity for any two modules, and compute the modality-specifii sparsity for the remaining one according to eq. 10 in the main paper. From the ex- perimental results of these sparsity configurations, we can determine the approximate range where the pruned subnetworks perform better.
Overall Acc
59
58
Second, use the same method traverse the reduced range with smaller step size of 5%. In this way, we can determine the most comfortable zone for the modality--peecifi sparsity.
0.2
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Figure 14: Results of subnetworks obtained by pruning with debiasing method LMH on VQA-CP v2.
Similarly, when the overall target sparsity is 90%, we directly traverse 80% ~ 98% with a step size of 2% to search the most comfortable zone of the modality-specifft sparsity.
mask train(bee)) (full lines) at all sparsity levels. As the sparsity increases, the gaps widen. This shows the initialization strategy we adopt more effective than random initialization.
D More Experiments on VQA-VS
D.1 Performance on varying OOD test sets of VQA-VS
C.3 A Close Look at The Performance of Subnetworks at 90% Sparsity
The Effect of Compression without Debiasing
From Fig. 14, we derive two abnormal observa- tions at the extremely high sparsity, i.e., 90%: 1) Pruning with "OMP + Imh ft" (pink and grey lines) is better than pruning with "mask train(Imh)" (cyan and brown lines). 2) Starting from "Ixmert(bce)" (pink and cyan lines) is better than starting from "Ixmert(Imh)" (grey and brown lines). The two observations at 90% sparsity are contrary other sparsity. For the first observation, we conjecture that this is because mask training (which involves binarization and gradient estimation) is more dif- ficult to optimize at 90% compared with further fine-tuning of the OMP subnetworks. The second observation can be explained by that: Further de- biasing the debiased full Ixmert in the pruning pro- cess slightly sacrifices the performance on "Other" questions, which require more reasoning ability than debiasing ability (as shown in the rightmost two plots of Fig. 12). Therefore, at the extremely high sparsity, when the benefits of debiasing on "Y/N" and "Num" questions are small, the perfor- mance penalty on "Other" questions results in drop in "Overall" accuracy. Nevertheless, the gaps between "Ixmert(Imh) mask train(Imh)" and the other two pipelines are small 90% sparsity.
For simplicity, we categorize the nine OOD test sets into 3 categories of different modalities, i.c.. language-based (OOD-lang), visual-based (OOD- vis) and cross-modality (OOD-crsM) ones. We report the average accuracy of each category, as well as the IID accuracy and the average accuracy of all OOD test sets 1ODD-mea) in Fig. 15.
The upper part of Fig. 15 shows the perfor- mance of subnetworks compressed without debi- asing method, it can be seen that: 1) All subnet- works obtained by pruning all three modules un- derperform "full model(hce)" in ID test set. This is because the ID performance relies on memory ability, which is positively related to the parameter quantity. 2) The subnetworks obtained by pruning the language module consistently outperform the full model on OOD-mean, OOD-lang and OOD- crsM test sets, which are related to the language bias. This indicates that the language module of Ixmert is slightly overparameterized. 3) In contrast, pruning other modules causes negative impact on OOD performance. Especially, pruming visual modules also results in sharp OOD-vis accuracy drop, indicating that the visual module of Ixmert is not suitable for compression.
C.4 Sparsity Configurations for the Three -odality-speciffi Modules
The Effect of Compression with Debiasing The lower part of Fig. 15 shows the VQA-VS perfor-
For the overall target sparsity of 50% and 70%, we adopt the following procedure to search the comfortable zone for the modality-spedie sparsity:
We exclude the configurations where the computed spar- sity for the remaining module is greater than or smaller than 0.

---

(smwr/occ) traniboe
Domee tralrabc))
Iamertttcc)) mark trak)
Scmerttbaal tranngoe))
10m (omeettbce) trainioo)
OOD OOS der
COCL
Lipgage Mides
kmeertimn)) mack traintmn))
lamettinhi mass traindmii
immertimii mask tranirr))
kmertimhhi mask trlniin
smott madk TO toinimno
ICCO ICCOUSKen
OUT OOMEIN
Figure 15: Results of subnetworks pruned using BCE (upper) and LMH (lower) on VQA-VS, Each column measures accuracy on ID test set, all, language-based, visual-based and cross-modality OOD test sets respectively, Different lines denote subnetworks obtained by pruning all, language, visual and cross- cooss-modality modules respectively.
(a) ID
(b) 00D-mean
(c) 00D-lang
(d) OOD-vis
(o) OOD-crsM
Figure 16: Comparison of subnetworks "Ixmert(Imh) + mask train(Imh)" with uniform sparsity and modality speciffc sparsity on VQA-VS.
mance of "Ixmert(Imh) + mask train(Imh)"", which performs the best VQA-CP v2. We can ob- serve that: 1) Pruning any modules can improve ID performance over the debiased full model ("full mode)(lm))").. This is because debiasing methods improve OOD performance at the cost of ID per- formance, while our pipeline alleviates such ID performance decline by compressing some harm- ful parameters. 2) Similarly, pruming any Ixmert modules with small sparsity (e.g., 0.2 and 0.4) also improves the OOD-mean performance. This demonstrates the existence of sparse and robust Ixmert subnetworks on VQA-VS. 3) Especially, subnetworks obtained by compressing the language module consistently perform better than subnet- works obtained by pruning other modules and the debiased full model (except on OOD-vis), since the dataset biases tend to be learned by the language module. 4) However, pruning any module fails to improve the OOD-vis accuracy, the debiasing method LMH is designed for the language bias.
D.2 The Effect of Modality-specific Sparsity varying OOD test sets of VQA-VS
We directly use the modality-specffic sparsity se- lected by the experiments Sec. 3.4 in the main paper on VQA-CP v2. Fig. 16 shows that the sub- networks with modality-specific sparsity always outperform those with uniform sparsity except for 90% sparsity on ID test set, which validates that different modules should be compressed with dif- ferent sparsity. Besides, when the overall sparsity too large or too small, the benefits of the assign- ment of modality-speecinn sparsity will decrease ac- cordingly. Note that the phenomenon OOD-vis is different from other OOD test sets as the spar- sity increases, since the debiasing methods LMH is designed for the language biases.
"Note that most debiasing methods fail on VQA-VS (Si ct al., 2022h), such as IPF and RUBi. We thus do not discuss them in this section.


Reviews:
[{"review_id": "VQsr2OTKOO", "reviewer": "Reviewer_KVsc", "summary": "", "questions": "see weakness.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "+ **Q1**: Experiments on more recent VLPs. \n>   We can understand this concern. However, we would like to clarify the reason of using LXMERT and VisualBERT in our experiments: Most reasearch on the VQA debiasing still use plain models (e.g., UpDn and S-MRL) [1]. **Among the few VLPs used in this field, LXMERT and visualBERT are the most popular ones**[2,3].  As we are the first joint study on VQA debiasing and model compressing, we choose these two representative VLPs to conduct thorough experiments. We believe our findings can generalize to varying transformer-based VLPs. \n\n+ **Q2**: Technical Contribution.\n>  We fully understand the concern of \"technical novelty\". However, _we call for an inclusive mindset towards the contribution and novelty of our work, which is different from the regular papers that focus on individual techniques_. We clarify our contribution from three perspectives:  \n>   **Firstly, we present the first joint study on debiasing and compression** for the VQA task. This topic is important as it can facilitate the deployment of SoTA VLPs in resource-constrained and security-sensitive real-world scenarios. It has been overlooked in existing literature, and our research fills this gap.  \n>   **Secondly, from the empirical perspective**, our extensive experiments on the training and compression pipeline and sparsity assignment can serve as a valuable guideline for future research on this topic.  \n>    **Thirdly, from the technical perspective**, we derive a specific compression and debiasing pipeline that establishes a SoTA sparsity-robustness trade-off. In contrast, neither existing debiasing techniques nor compression techniques alone can accomplish the objective of our topic. \n\n+ **Q3**: Formatting Issues.  \n>  Thank you for the feedback. The questions and formatting issues will be properly addressed in the next version of our paper.\n\n+ **Q4**: Connection between mask training and DiffPruning. \n>   Mask training and DiffPruning are similar in the training mechanism: they both optimize the mask vectors (referred to as Diff vector in DiffPruning) applied to the pre-trained parameters. However, they are fundamentally different in their objectives. DiffPruning aims at **reducing the newly added parameters** when transferring pre-trained models to a downstream task, while the model size is unchanged (the Diff vector is added to the frozen pre-trained parameters). In comparison, mask training aims at **obtaining a smaller model for inference** (a portion of parameters are set to zero based on the mask). \n\n+ **Q5**: Has the paper compared the performance of previous methods in one domain with methods in another domain?\n> Yes, we have compared the methods in the debiasing domain (e.g., lxmert(lmh), lxmert(lpf), GCE, etc. in Fig.1) and methods the model compression domain (e.g., lxmert(bce)+mask train(bce) and lxmert(bce)+OMP(bce) in Fig.2). The debiased lxmert, while achieving strong OOD performance, is large in size and inefficient as the original lxmert. The compressed lxmert subnetworks (without debiasing), while more efficient, are only slightly better than original lxmert  in the OOD scenario. By combining the compression and debiasing techniques, we obtain lxmert subnetworks that are both small in size and achieve strong OOD performance, outperforming individual debiasing and compression methods in terms of the sparsity-robustness trade-off. We believe that this is a substantial contribution compared to previous methods. \n\n\n\n[1] https://paperswithcode.com/sota/visual-question-answering-on-vqa-cp\n\n[2] https://github.com/cdancette/vqa-cp-leaderboard\n\n[3] https://aclanthology.org/2022.findings-emnlp.495.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of the paper is clear, and the approach is well-articulated.\n\n2. The method proposed in this paper achieves state-of-the-art (SOTA) results on some datasets.\n\n3. This paper investigates the relationship between sparsity and out-of-distribution (OOD) generalization performance.", "reasons_to_reject": "1. The backbone models used in this paper, namely VisuaBERT and LXMERT, are relatively outdated. How does the method proposed in this paper perform on newer VLP (Vision-Language Pretraining) models?\n\n2. This paper employs existing methods such as Magnitude-based Pruning in pruning, Mask Training, and LMH, RUBi, LPF in de-biasing, combining them into a single pipeline and applying them to the task proposed in this paper. I think the novelty of this paper is incremental in nature, and the authors need to further clarify their contributions.\n\n3. There are some formatting issues and typographical errors in the paper, such as in Figure 2, where the last line of the legend, \"full lxmbert,\" is missing a closing parenthesis.\n\n4. The idea of using mask training and DiffPruning [1] seems very similar. Have the authors noticed the connection between the two?\n\n5. The authors claim that \"this paper presents the first joint study on the compression and debiasing problems of VLP for the VQA task.\" Has the paper compared the performance of previous methods in one domain with methods in another domain to demonstrate that the method proposed in this paper indeed makes a substantial contribution compared to previous methods? The authors need to reflect on this point.\n\n[1] Demi Guo, Alexander M. Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. In Annual Meeting of the Association for Computational Linguistics.", "questions_to_authors": "see weakness.", "paper_topic_and_main_contributions": "In this paper, the author  investigates whether a VLP can be compressed and debiased simultaneously by searching sparse and robust subnetworks. \nThey present a comprehensive study on the design of the training and compression pipeline for a good sparsity-performance trade-off, and provide some valuable findings about the assignment of sparsity to different modality-specific modules. The compressed lxmert subnetworks in this paper outperform the SoTA debiasing methods with fewer or similar model parameter counts.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sNuYkNCNLr", "reviewer": "Reviewer_HrWM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "+ **Q1**: Restrict the experiments to the LXMERT model? \n> **We did not only conduct experiments on LXMERT model, but also validated our method on visualBERT**.   We would like to clarify the reason of using LXMERT and VisualBERT in our experiments: Most reasearch on the VQA debiasing still use plain models (e.g., UpDn and S-MRL) [1,2,3]. Among the few VLPs used in this field, LXMERT and visualBERT are the most popular ones.  As we are the first joint study on VQA debiasing and model compressing, we choose these two representative VLPs to conduct thorough experiments. We believe our findings can generalize to varying transformer-based VLPs.  \n> [1] https://paperswithcode.com/sota/visual-question-answering-on-vqa-cp   \n> [2] https://github.com/cdancette/vqa-cp-leaderboard    \n> [3] https://aclanthology.org/2022.findings-emnlp.495.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The concept of simultaneously compressing and debiasing the VLP model presents an intriguing approach.\n2. The proposed methodology demonstrates its efficacy when applied to the LXMERT model.", "reasons_to_reject": "The authors restrict their experiments to the LXMERT model. It raises the question of whether similar performance enhancements would be observed with other VLP models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This research delves into compression techniques for the VLP model, specifically within the domain of Visual Question Answering (VQA). Using the LXMERT model as a baseline, the authors concurrently examine compression and debiasing issues. They explore various pruning strategies, including magnitude-based pruning and mask training, along with multiple debiasing approaches.\nExperimental findings indicate that the compressed LXMERT subnetworks surpass state-of-the-art (SOTA) debiasing methods, achieving this performance with fewer or comparable model parameters.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c7tjjod6oA", "reviewer": "Reviewer_AWxQ", "summary": "", "questions": "What is the definition of a_k in the paper?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "+ **Q1**: Technical Contribution.\n>  We fully understand the concern of \"technical novelty\". However, we call for an inclusive mindset towards the contribution and novelty of our work, which is different from the regular papers that focus on individual techniques. We clarify our contribution from three perspectives:  \n> **Firstly, we present the first joint study on debiasing and compression for the VQA task.** This topic is important as it can facilitate the deployment of SoTA VLPs in resource-constrained and security-sensitive real-world scenarios. It has been overlooked in existing literature, and our research fills this gap.  \n>    **Secondly, from the empirical perspective,** our extensive experiments on the training and compression pipeline and sparsity assignment can serve as a valuable guideline for future research on this topic.  \n>    **Thirdly, from the technical perspective,** we derive a specific compression and debiasing pipeline that establishes a SoTA sparsity-robustness trade-off. In contrast, neither existing debiasing techniques nor compression techniques alone can accomplish the objective of our topic. \n\n+ **Q2**: Fairness of comparison. \n> We respectfully disagree. It is worth noting that the comparison of entire systems is common when benchmarking the SoTA methods, and the previous VQA debiasing SoTAs already employ different backbone models. We compare our debiased subnetworks with previous debiasing methods at a \"system-level\" to demonstrate a progress made in this field. Moreover, even when using the same backbone model, the subnetworks surpass the debiased lxmert (lmh) and lxmert (lpf) in terms of OOD performance, while also requiring significantly fewer model parameters. \n\n+ **Q3**: What is the definition of $a_k$ in the paper?\n> Thank you for pointing out the missed details. $a_k$ denotes the ground-truth answer for the $k_{th}$ sample.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The comprehensive set of experiments in this paper effectively demonstrates that the sparsity introduced by compression can contribute to improved performance on out-of-distribution (OOD) tasks.\n\n2. The proposed subnetwork can achieve promising results on two OOD datasets.", "reasons_to_reject": "The technique contribution is limited because the used compression methods and debiasing methods are directly adopted from previous work.\n\nFurthermore, it is not fair to compare the proposed subnetworks with other debiasing methods directly because the compression methods change the model itself, and the training strategies are searched as well, which limits the contribution of the proposed subnetworks.", "questions_to_authors": "What is the definition of a_k in the paper?", "paper_topic_and_main_contributions": "This paper presents an empirical study outlining a procedure to address the twofold issue of a model's failure to generalize to out-of-distribution data and model compression challenges. The core contribution lies in the author's exploration of the integration of debiasing techniques, training strategies, and compression methods. Their insights could inspire on how compression impacts generalization, performance, and different modules within the model. The proposed subnetworks can achieve promising performance n the debiasing task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SwMGBKIR8y", "reviewer": "Reviewer_rJ51", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "+**Q1**: The concern of Equation9 & The data used in model development.\n> This is a misunderstanding. We would like to clarify three points:  \n> (1) Equation 9 aims to describe our ultimate objective, rather than implying that \"we utilize OOD data during the model development stage\". We will revise this paragraph to avoid ambiguity.   \n> (2) All of our experiments followed the same settings with all previous baselines on VQA-CP and VQA-VS to ensure comparative fairness.   \n(3) We fully agree with your viewpoint that \"models are typically developed without access to OOD data\",  and have been implementing it as much as possible in this paper. However, due to the lack of a in-distribution validation set for VQA-CP ([1,2] mentioned this issue), we can only follow all previous debiasing methods's settings and directly use the test set to select checkpoints, because  VQA-CP is the most popular OOD benchmark in VQA, and almost all debiasing methods for VQA evaluates their models on this benchmark. Note that even in this setting, finding robust subnetworks is non-trivial, because the full model and subnetworks without debiasing training are not robust.   \nHowever, to avoid the impact caused by this issue, we further experiment on the VQA-VS dataset. VQA-VS ensures that no OOD data is available during training and has addressed the limitations of VQA-CP from multiple aspects, setting a more reasonable standard for the OOD testing procedure. Our method can still find compressed and robust models on VQA-VS. \n\n[1] On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law.   \n[2] Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper addresses two important challenges with vision-language pre-trained models (VLPs) - reliance on biases that hurts generalization, and inefficiency in terms of computation/memory.\n2. The systematic analysis of training pipeline, compression techniques, and sparsity assignment provides new insights into effectively compressing and debiasing VLPs.\n3. The authors propose joint compression-debiasing approach that obtained compressed VLP subnetworks that match full debiased models and outperform prior debiasing techniques. It provides a promising new technique for model compression and debiasing that could transfer to other NLP domains.\n\n", "reasons_to_reject": "My major concern is that in equation (9), when trying to obtain the compressed model, the authors conducted out-of-distribution (OOD) evaluation. This means the authors utilized OOD data during the model development stage. This setting diverges from existing OOD generalization literature, where models are typically developed without access to OOD data. Since the authors have incorporated OOD data into model development, it is unsurprising they can simultaneously find compressed and robust models. The standard approach is to develop models only on in-distribution data, then evaluate robustness separately on held-out OOD data. Using OOD data during development confounds the results and limits conclusions about true OOD generalization abilities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents the first joint study on compressing and debiasing vision-language pre-trained models (VLPs) for visual question answering (VQA). Although VLPs achieve good performance on standard VQA, they rely on dataset biases which hurts out-of-distribution (OOD) generalization, and are inefficient in terms of memory and computation. This paper investigates whether a VLP can be simultaneously compressed to reduce costs and debiased to improve OOD accuracy by searching for sparse and robust subnetworks. Through systematic analysis of training/compression pipeline designs and sparsity assignment to different modality modules, experiments with 2 VLPs, 2 compression methods, 4 training methods, 2 datasets and varying sparsity levels show it's possible to obtain sparse VLP subnetworks that match debiased full VLPs and outperform debiasing state-of-the-art on OOD datasets with fewer parameters. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I appreciate authors' full engagement and attempt to answer concerns and questions about the manuscript. The manuscript motivates the problem and provide insight into tradeoff between sparsity and performance and achieve performance parity with significant parameter reduction. While relation between smoothing, compression and bias reduction have been studied in traditional ML techniques, Authors have identified an opportunity to debias and compress at the same time for VLM, which in my view is novel and exciting and get facilitate further research in this area.
Decision: Accept-Main