Learning to Jointly Share and Prune Weights for Grounding Based Vision and Language Models
Authors: Shangqian Gao
Burak Uzkent
Yilin Shen
Heng Huang
Hongxia Jin
Venue: ICLR 2023 poster (2023)

Abstract: Transformers have seen growing interest in processing different modalities,  including language and image data. As a result, we can process vision and language data using transformers that are architecturally similar. Leveraging this feature of transformers, we propose weight sharing across two transformer backbones and within the same transformer backbone and pruning across two backbones in a unified framework. More specifically, we investigate weight sharing and pruning for two components of the transformers: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) layers. To jointly perform weight sharing and pruning, we propose to use a regularization term to align model weights and the desired structure during the multimodal pre-training step. The structure vectors of sharing and pruning are generated by using a hypernetwork, which can capture complex interactions between pruning and sharing across layers and modalities. We train the hypernetwork and model weights iteratively so that the learned structure evolves along with model weights. After minimizing the proposed objective in the pre-training step, we perform weight sharing and pruning and fine-tune the compressed model on downstream tasks. Finally, we perform experiments on vision and language tasks, including Referring Expression Comprehension (REC), Visual Question Answering (VQA), and Object Detection using the state-of-the-art grounding based models: MDETR and GLIP. Our experiments show that we can compress these models by $35-40\%$ by sharing and pruning MSA and FFN weights without almost any loss in accuracy.

Published as a conference paper at ICLR 2023
LEARNING TO JOINTLY SHARE AND PRUNE WEIGHTS FOR GROUNDING BASED VISION AND LANGUAGE MODELS
Shangqian Gao',, Burak Uzkent", Yilin Shen', Heng Huang', Hongxia Jin DDpartment of Electrical and Computer Engineering, University of Pittsburgh Samsung Research America {shg84, heng. beng.huang]er [apang]@pitt.... edu, {b. .ouukent, yilin. shen, hongxia. jin} & samsung. com
ABSTRACT
Transformers have been successful in processing different data modalities, such as language and image data, which could use transformers with similar architectures to achieve good performance. Leveraging this observation, we propose weight sharing across two transformer backbones and within the same transformer back- bone and pruning across two backbones in a unified framework. More specifically. we investigate weight sharing and pruning for two components of the transform- ers: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) lay- ers. To jointly perform weight sharing and pruning. we propose to use regulariza- tion term to align model weights and the desired structure during the multimodal pre-training step. The structure vectors of sharing and pruning are generated by using hypernetwork, which capture complex interactions between pruning and sharing across layers and modalities. We train the hypernetwork and model weights iteratively so that the learned structure evolves along with model weights. After minimizing the proposed objective in pre-training step. we perform weight sharing and pruning and fine-tune the compressed model on downstream tasks. Finally, we perform experiments on vision and language tasks, including Refer- ring Expression Comprehension (REC), Visual Question Answering (VQA), and Object Detection using the state-of-the of-the-art art grounding based models: MDETR and GLIP. Our experiments show that we can compress these models by 40% by sharing and pruning MSA and FFN weights without almost any loss in accuracy.
INTRODUCTION
The dominant architecture in natural language processing (NLP) is Transformer (Vaswani et al. 2017 Besides NLP, recent advance in computer vision shows that transformer based model, like ViT Dosovitskiy et al. 202T) or DeiT (Touvron et al. 2020), can achieve similar or even better performance than convolutional neural networks (CNNs) on various tasks. As a result, it allows us to use architecturally similar models on cross-modal tasks with vision and language data. This setting naturally provides foundations to structurally share weights across different modalities. The advantage of weight sharing is that it encourages weight reuse and thus reduces the number of parameters while maintaining the model capacity to some extent. On the other hand, existing weight sharing techniques have some limitations. Most of them (Lee et al. [021] You et al. [022] Lan et al.) 2019 Reid et al.] (02TT) use manually designed sharing rules to share a whole layer or block, largely restricting the flexibility of weight sharing. This reduced flexibility can lead drastic performance drops. To maximally utilize model parameters, we propose to unify cross-modal sharing, layer- wise sharing, and pruning, all in single unified framework. Unlike previous works, the minimal structure of these operations is a weight vector instead of a whole layer or block, which drastically increases the flexibility of sharing and pruning. Also, to avoid using manually designed strategies, the positions of sharing and pruning are learned in an end-to-end differentiable manner.
Was a research intern in Samsung Research America.

---

Published as a conference paper at ICLR 2023
To pursue a better trade-off between the model performance and the parameter efficiency, we aim to maximize flexibility by utilizing the structure of transformer backbones. If only cross-modal sharing is considered, there will be upper bound for the compression rate (~ 50%) when sharing all layers of one backbone for another one. Another direction is to share layers within a single backbone similar Albert (Lan al., 2019), however, the downside is that there is no prior success of cross-layer sharing for the vision transformer. In addition to weight sharing, pruning is also an option; however, [i al. (2020); Yu et al. (2022) show that limits the capacity of the model when the compression rate is high. We argue that, especially in multimodal tasks, it can be hard to achieve a high compression rate while maintaining high accuracy by using only pruning or sharing.
In this direction, we unify cross-modal sharing, layer- wise sharing, and pruning into a single frame- work to maximize the flexibility for sharing and pruning. With the increased flexibility, finding which weights to prune or share can be very difficult. To solve this problem, we design hypernet- work for learning the binary structure vectors for sharing and pruning efficiently. The hypernetwork can effectively capture the complex interactions between pruning and sharing across different layers and modalities. Since we can not share and prune a weight vector at the same time, we apply con- straints on different parts of structure vectors. We use regularization term to softly align the learned structures and backbone weights instead of directly sharing and pruning by applying structure vec- tors. The benefit of such regularization term is that do not need to add separate fine-tuning process to the whole pre-training step. Finally, we iteratively train the hypernetwork and the model so that the learned structures can adapt to the changes in the model during training. With these novel designs, our method learns suitable structures for sharing and pruning with small extra costs.
We evaluate the effectiveness of our method with two state-of-the-ee vision and language ground- ing models: MDETR (Kamath et al. 202T), GLIP (Li el al. 2022). We use the CLIP text trans- former (Radford et al.) 2021) and DeiT (Tourron et al.) 2020) as the text backbone and the vision backbone, and thus two backbones have similar architectures. Our method is evaluated on differ- ent downstream tasks: RefCOOCOReFCOCOO
YYu et al., 2066/RefCOOO (Mao et al. 2016) for referring expression comprehension (RFC), GQA (Hudson & Manning, 2019) for visual question answering (VQA), MS-COCO object detection (Lin et al. (2014) and Flickr-30k (Plummer et al.) 2015) for phrase grounding. On these benchmarks, our method can remove around 40% 45% pa- rameters of the backbones and 35% ~ 40% of all the parameters with almost no accuracy drop. In some tasks, method even outperforms the original model. These results show that the proposed framework achieves a prominent rrfee-off between the number of parameters and accuracy.
2 RELATED WORKS
Vision and Language Models. Existing vision and language models can be broadly grouped into two categories: (1) two-stage and (2) single-stage. Two-stage methods (Yu et al. 2018 Chen et al.) 2020 Lu et al. 2019) utilize off-the-shelf object detectors, e.g., RCNN to detect objects and represent them with convolutional features. For referring expression comprehension, the referring expression is matched to its appropriate region (Yu et al. 2018). For other vision and language get image and language repre- sentations (Lu et al., 2019 Zhang et al.] 2021a Gan et al. 2020). Single-stage methods (Kamath) eet al. 2021 Deng et al. 2021 Yang et al. 2020 Chen et al. 2018 Li & Sigal (021) avoid using a detached off-the-shelf object detector and perform end-to-end training and inference, reducing the computational complexity of the two-stage methods. Another advantage of single-stage methods is that the full model can be pre-trained for text-conditioned object detection as in MDETR (Kamath
tasks, region descriptions are passed through two transformers
al. 202T). , GLIP (Li et al. 2022) GLIP-v2 (Zhang et al. 2022). etc.
Pruning with Transformers. With the increasing popularity of vision transformers, there are many works that perform pruming on different components, such as pruning tokens (Kong et al.) [021]] Xu
et al. 2021: Rao et al., 202T), pruning structures or weights (Chen et al.] [021] Yin et al. [023] Lou et al. (022b), or all of the above (Yang et al.) (02T). The pruning for language transformers (Li
al. 2020 Gale et al. 2019) can be traced back even carlier. Unlike these works, in the cross-modal
setting, we only prune weights that are not useful in both modalities.
Transformer-basse Unified Backbones. Transformers recently enjoyed tremendous popularity in processing different modalities, including images, videos (Girdhar et al. (019). speech (Dong et al, 2018), audio (Jaegle et al. 2021). and language (Jaegle et al. |0.]]222]]] | & Singh 2021 Zhang al. 2021b). The universality of transformers has been exploited for multi-tasking using the same vi- sion backbone for different vision-only tasks (Girdhar et al. 2022). the same vision and language
2

---

Published as a conference paper at ICLR 2023
backbone for different vision and language tasks (Hu & Singh 2021 Likhosherstov et al. (2021) on the other hand, performs multitasking with the same transformer backbone on different modal- ities, and Wang al. (202I) pre-trains the model vision-only, text-only, and lision-language tasks and fine-tune the model on vision or text-only or vision-language tasks. Similarly, [i et al. (202T): You al. (202T) pre-train unified transformer backbone with images and their captions and fine-tune the backbone for the single modality downstream tasks. Different from these studies, we propose structural cross-modal, layer-wise sharing and pruning across two transformer back- bones for grounding based vision and language downstream tasks with almost no loss in accuracy.
Weight Sharing with Transformers. There has been a number of studies that focus on sharing weights in a single transformer on language-only tasks (Lan et al., (2199) et al. [202]] Takase &
Kiyono, 202T Lou et al. 2022a). These studies mostly re-use a language transformer's full encoder (MSA+FFN) at different depths using some manually defined strategy. Parameter efficient cross- modal transformers (Lee et al., 202T) reduces parameters with manually designed weight sharing strategies. Another recent study, Kim et al. (2022), performs weight sharing in the multimodal fusion network, which can be orthogonal to our study as we explore weight sharing across the transformer backbones. Finally, a recent work, MS-CLIP You et al. 2022), shares weights across modalities by rigorously examining the architecture choices and adding early specialization layers. On the other hand, our method is general and flexible vision and language model. Additionally, unlike MS-CLIP, we do not change the architecture of the original model. As result, our method can be applied to any vision and language model with transformer backbones. Finally, different from these studies, we focus on grounding based vision and language models, which have shown impressive performance on many vision and language tasks (Kamath et al. 2021; Li et al. 2022. Zhang et al. 2022).
learn where to share and prune weight vectors in
3 LEARNING TO SHARE AND PRUNE STRUCTURES FOR VISION AND LANGUAGE TR TRANSFORMERS
3.1 OVERVIEW
In a typical vision and language model, we have two backbones: a vision backbone (v) and a text backbone (t). We use W E {u, f) to denote the weight matrix of I'th layer from the vision (W"1) or the text backbone (W"),, and 1, ... L. L is the total number of layers. This definition also applies to sharing and pruning vectors: 8 and m². Since we consider architecturally similar backbones, the vision and text backbone have the same L.
Usually, transformers are used for processing text and vision data. A classic transformer has two core components: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) layers. They are the targets for structural weight sharing and pruning since these components contain most of the weights in backbones. To find which structures to prune or share, we first need to optimize the sharing and pruning vectors produced by the hypernetwork during the pre-training process of MDETR (Kamath al.] 2021) and GLIP (Li et al. 2022). To further increase the flexibility of the pruning and sharing process, layer-wise sharing (inspired by Albert (Lan et al.) 2019) but at a fine-grained level) across the text backbone is also performed. A detailed comparison of different sharing and pruning options is given in Fig. - Since we learn discrete structure vectors, directly applying them in the fine-tuning step can result in severe accuracy drops. We instead softly align model weights and learned structures via a regularization term in the pre- trainini step. After the pre- training step, we conduct structural pruning and sharing based on learned structure vectors. Finally, we fine-tune the compressed model for the downstream task. An overview of how our method shares and prunes weights is provided in Fig.
3.2 SHARING AND PRUNING ACROSS TWO BACKBONES
For MSA layers, we have three weight matrices for query W Radd, key W E Rexd and value W E Rdxd and again E {u, t}. With input tokens X* E RN xd, the final Q", K* and V for self-attention is obtained by:
x K* = X V X'W;
(1)
where d is the embedding dimension, N represents the number of tokens given the modality. To make minimal changes to the original model, we can perform pruning on W and W and structural sharing are applied to all three weight matrix between backbones and across layers.
3

---

Published as a conference paper at ICLR 2023
ith layer
Sgenstee
Shared weights from the tedt backbone
I+1th
Pruning Staring According to m,
Bluck Seperation
1+2th
2+3th
Query Weights
Key Weights
Value Weights
Text Backbone
Vision Backbone
An overview of weights sharing and pruning across two backbones, Here, for simplicity, we use MSA layers as an example, and the process for the FFN layer is similar. The hypernetwork (HN) generates m, guide the pruning and sharing across modalities and within layers in block. As shown in the figure, to incorporate bock-wiss sharing, the text backbone is separated into several blocks. The confticts between sharing and pruning are resolved before they are applied to backbones' weights. Note that different from other weights, W, is not pruned.
Figure 1:
For pruning, we use binary vector m E {0, 1]d on W and W Let's use the vision backbone as an example for pruning. The production of the query and key when calculating MSA then becomes:
Q"(K"""²= X" @ Omm mW" @ mx")"""")"
(2)
where m is first resized to have the same dimension of W and @ is the element-wise production. x the feature map from the previous layer.
We perform cross-modal sharing between two backbones, and cross-layer sharing for the text back- bone. To achieve sharing, we modalities and different layers. Weights of the text backbone are used for cross-modal sharing. As result, a weight vector could be implicitly used in different layers across text and vision backbones. Let's use W as an example for cross-modal sharing:
binary vector s* E {0, 1] to share weights between two
W s o W +1 s'= @ W
where s is also first expanded to have the same size of W In short, weight vectors of index with sf 0 shared. Without loss of generality, omitted the modal notation (v t) for cross-modal sharing s', since the sharing direction is fixed. Similarly, cross-layer sharing has the following setup:
Wh = st @ Wh + (1 - s*) q
())
where Wtth represents the weights of ith layer in the text backhone, W is the assigned base weights, and s is the vector used for cross-layer sharing, As result, the final weights for the text backbone weights and shared weights from base layers.
divided into two parts:
A naive way for cross layer sharing layers (Lan et al. 2019). However, it can restrict the diversity of weights across layers. To enhance the flexibility of cross-layer sharing, divide the text backbone into several blocks, and each block has its own base weights. Specifically, we establish a set of base layers (in sorted order) B {bi, b...... , b All layers naturally split into several blocks based base layers, e.g., the first block contains layers with bi < I < b2. In our experiments, the base weight is set to the first layer of each block. When / base layer (1 E B), we force stt since assign the base layer its own weights and potentially reuse it for the other layers in the same block.
weights from certain layer as base weights for all other
Note that after cross-layer sharing, the base weights for cross-moda sharing is also changed. By putting cross-layer sharing and cross-modal sharing together, we have:
(5) As result, the final W consists weights from With & (weights from the text backbone of the same layer), Wi (weights from the text backbone of the base layer) and vision modality specific weights.
Wh @ W + +(1 s c W + (1 - s') -0) W..
4

---

Published as a conference paper at ICLR 2023
Text
techbono praned soordd parsnetenc
Vision
Bbolcateccaaee wabe
Prunce Weight
Cross-Modal Sharing and Pruning
Cross-Modal, Layer-Wise Sharing and Pruning
Cross-Modal, Block-Wise
Sharing and Pruning
More Flexibility
Figure 2: Overview of different settings of sharing and pruning- By including cross-modal, block- wise sharing, and pruning, we can reduce the number of parameters in different ways.
For FFN layers, have weight matrix W E Rdxd' expanding the feature dimension, and another weight matrix W E Rd xd compressing the feature dimension. keep the feature map size d unchanged, we conduct sharing and pruning along the dimension with size d' and the process similar to MSA layers. The detailed formulation is provided in the supplementary materials. We do not apply sharing and pruning for other layers including Layer Norm (Ba al 2016) since they contain very small amount of parameters.
3.3 CONFLICTS OF STRUCTURE VECTORS
Without any constraint, and could have conflicts. For example, if we prune weights, it becomes meaningless to share the same weights. To resolve such conflicts, we impose the following constraint for cross-modal sharing and pruning:
(m," x E C, (m² 8)) E C,, where C ((x, v)) (0,0))
where i is the index of sharing or pruning for certain weight vector. If (m; s;) E C. then the structure vector does not share and prune the same weight vector which resolves the conflicts. This constraint can also be applied for cross-layer sharing and pruning for the text backbone, but in more sophisticated way:
{ (ml) s()) C, where C (( + (0. B, (m)) 441 h41.... )/) E C, where C = ( (z, + (0,0)), 1 EB,
where b' represents the last element of the current block. For example, if b1 the current block consists layers of bj < I < by, and b' b2 1. With 8/' g ...8 represent all the shared elements from different layers in the current block, and these elements should be kept in the base layer.
We do not add this constraint between sf and s since there is no conflict between sharing. To directly apply this constraint, we can prioritize m; or s; and the other one manually. For For example, let m s 0. we can directly set m = 1 to comply with the constraint in Eq. 6 if s; is prioritized (shared weights will be pruned). practice, we prefer sharing weights to pruning weights, since sharing preserves the model capacity to some extent.
3.4
LEARNING STRUCTURE VECTORS
To generate m, 8, we use hypernetwork (HN) parameterized by 0 and Gumbel-Sigmoid (Jang et al., 2016) technique:
m, HN(2, 8),
(8)
where is the input the HN, and predefined vector. Basically, the HN is composed of GRUs (Chung et al. 2014) and multilayer perceptrons (MLPs). GRU can capture inter-layer inter- actions, and MLP used for intra-layer interactions. More details of the HN are presented in the supplementary materials. To learn the HN, we want to solve the following optimization problem:
min Cpre-trainin (x, y: m, s) + AR(P(m, s), pPuwaa):
where x is the input sample of the image and text pair, is its label, treeiniinigg is the original pre- training loss from MDETR (Kamath et al. 2021 GL.PP (Li (Liet al. (022), and its model structure is
5

---

Published as a conference paper at ICLR 2023
Algorithm 1: Learning to Jointly Share and Prune Weights for Grounding Based Vision and Language Tasks
Input: the pre-training dataset and sub-dataset for learning structure vectors: D. Dsub:
remained rate of parameters: pt hyper-parameter X, 7/ pre-training epochs: E; the model for pre-training: f; the hypernetwork HN parameterized by 0
for e : E do
/* Optimizing model weights. Freeze 0 in HN
for a mini-batch (z, y) in D do
1. generate m, from HN with Eq. 8
apply constraints on m, defined Eq. 6 and Eq. 7 and sharing is prioritized.
3. calculate R., (W, m, 8) given W and m,
calculate gradients w.rl W by minimizing Obj. and update W. end
/* Optimizing HN weights. Freeze W in the model.
for a mini-batch (z, y) in Duub do generate m, from HN with Eq. 8
2. apply constraints on calculate the parameter regularization term R (m, s). PPaaaa).
defined in Eq. 6 and Eq. 7 and sharing is prioritized.
4. calculate gradients w.r. 0 by minimizing Obj. 2 and update A.
end end
Get f' by pruning and sharing f based on m, Return f for task-specific fine-tuning.
decided by m, s. On the other hand, À controls the strength of (m,(m, s), p(ion)) The regulariza- tion loss, R (mP(m, s), plaioa)), controls how much parameters we should keep given by (0, 1]. P(m, 8) in the regularization loss represents the remaining number of parameters decided by m, s and Pioul is the total number of parameters of MSA+FFN layers in backbones. In practice, R(P(m, D.PPam = log (max(P s). PPPmaa) / P (Gao et al. 2021). On the other hand, R could be any regression loss functions, including MSE and MAE; however, it might be harder to push R small enough, especially when the number of parameters is large.
let
During the optimization of the HN, keep the weights of the whole model including backbones and the fusion part frozen. Since the time cost for pre-training is quite large, we use a small portion of the whole dataset to train HN. Next, we train model weights on the whole pre-training dataset while controlling its architecture with m, by training the hypernetwork on subset iteratively.
3.5 LEARNING TO JOINTLY SHARE AND PRUNE WEIGHTS
To share and prune the weights, we can directly apply m and (from the hypernetwork) vision and text transformers. However, it can drop the accuracy since the outputs of the vision and text backbones are drastically changed. Thus, alleviate this problem, we use selection based regu- larization mechanism to softly push selected weights closer (for sharing) or to zeros (for pruning):
Rw (W, m. s)
- [(1 s) 5) o
W (1 (1-s) 8) W
(10)
+ Σ
Wth - -1-5h (1- G w + 4mm=,, m4 Em.-I[Wi..lll
where the first two terms push selected weight vectors closer and the final term uses Group Lasso to push selected weight vectors to O. With this regularization term, it gradually aligns model weights to the learned structure vectors which creates a smooth process for reducing the number of model parameters. Given the regularization loss, we learn model weights W by optimizing the following objective function:
min uprettiining (z, y; W) + RR (W, m)
(11)
where 7 controls the strength of R. (W, m, m,,). s). After the pre-training process, the corresponding weights pruned and shared to compress the model. The compressed model can then be used for fine-tuning on downstream tasks.
We present the training process of the proposed method in Algorithm. I Note that (1) the forward calculation is different when optimizing the model (Obj. IT] and HN (Obj. D.. When optimizing HN, m, are applied in the forward calculation, and when optimizing the model, the forward calculation

---

Published as a conference paper at ICLR 2023
Settings
RefCOCO
RefCOCO+
ReiCOCOg
GQA
#PT
#PB
CLLL-I
75.1
65.3
67.7
50.9
53M
44M
DeiT-SmaallAAher
75.3
65.2
67.7
51.0
48M
34M
Backbones:
BeiTT-Small+CIL.
Baseline
77.5
67.3
69.9
52.9
72M
63M
FS
70.1
59.6
61.5
44.7
50M
41M
SP
76.2
65.8
68.5
52.3
52M
43M
LWSP
77.2
66.5
68.6
52.8
48M
39M
BWSP
77.6
67.3
69.4
53.0
48M
39M
BWSP w/o R.
71.6
61.4
62.2
46.4
48M
39M
Table I: Results tasks from MDETR. We use MDETR-Sman with CLIP+DeiT-Smanl backbones.
#PT/#PB in this table, Tab. 2 Tab. and Tab. represent the number of total parameters and back- bone parameters.
Settings
RefCOCO
RefCOCO+
RefCOCOg
GOA
#PT
#PB
Deil-Tiny+CLIIP
76.8
69.9
70.0
54.3
141M
124M
DeiT-SmalI+CLIP
77.5
70.6
70.3
54.7
159M
142M
Backbones: DeiT- CCll-Bas+CCLIP
Baseline
79.8
72.1
72.9
56.7
226M
209M
FS
72.8
62.6
63.0
46.1
141M
124M
BWSP
80,5
72.4
72.6
56.6
136M
119M
BWSP w/o R.,
73.9
63.4
64.6
48.1
136M
119M
Table 2: Results on tasks from MDETR. We use MDETR-Large with CLIP+DeiT-Band backbones.
remains intact. (2) To reduce the training time of the HN, we use a small subset Duub of the pre- training dataset, sub < D. (3) The goal of using HN is to accelerate the learning of m, and better capture the complicated interaction between pruning and sharing across modalities and layers. We note that can also directly set m, learnable parameters, however, it can slow down the learning process and thus lower the final performance.
4
EXPERIMENTS
4.1 SETTINGS
GLIP and MDETR. We apply our method to two recently published grounding based vision and language models: (1) MDETR (Kamath al.) (02T) and (2) GLIP (Li et al. (022). MDETR performs end-to-end pre-training on dataset of 1 I.3M image-text pairs for the task of text-conditioned object detection. On the other hand, GLIP scales up the pre-training dataset for text-conditioned object detection to 27M image-text pairs by training teacher network on 1.3M image-text pairs with ground truth bounding boxes and using the teacher on 27M image-texi pairs to get pseudo bounding boxes. Pre-trained GLIP and MDETR are then used for vision and language tasks.
Baselines. We constructed several alternative settings to verify the design choice of our method. The first setting is Fully cross-modal Sharing (FS). For FS, share all MSA+FFN layers across two backbones during both the pre-training stage and the fine-tuning stage without any regularization term. We note that FS can be considered to You et al. (2022); Lee et al. (202T). The second setting is cross-modal Sharing and Pruning (SP). For SP, we only apply cross-modal sharing and pruning for the vision and text backbones. The third setting is cross-modal, Layer-Wise Sharing and Pruning (LWSP). On top of SP, LWSP also shares weights across layers in the text backbone. The third setting is cross-modal, Block-Wise Sharing and Pruning (BWSP). For BWSP, the text backbone is split into three blocks, and each block contains 4 consecutive MSA and MLP layers. Fig. 2 shows the concept of these settings. By default, layer-wise sharing and block-wise sharing are applied for the text backbone, and the weights of the text backbone are reused for the vision backbone for the cross-modal sharing since Lu et al. 1 showed that language transformers can generalize to vision tasks.
manually designed weight sharing method similar
Implementation Details. In all experiments, we replace the original backbones used in MDETR and GLIP with the CLIP text transformer and DeiT to have architecturally similar backbones. Due to the resource limitation, we change the GLIP pre-training dataset to be the same as MDETR. and the pre-training dataset for MDETR is not changed. We note that MDETR and GLIP train the models on different size images and test the models on images with smaller size resized to 800 pixels. On the other hand, resize the images to 224x224 pixels in training and test time as the motivation of
7

---

Published as a conference paper at ICLR 2023
(a) FFN Vision Backhone
(b) MSA Vision Backbone
64
(c) FFN Text Backbone
(d) MSA Text Backbone
Figure 3: Ratios of weight sharing and pruning across the MSA and FFN layers in backbones. Dashed lines in the text backbone represent block separations.
Settings
AP
AP50
AP7s
#PT
#PB
Backbones: CLIP+DeiT-Small
Baseline
20.0
32.7
20.7
83M
63M
SP
19.1
31.0
19.7
63M
43M
LWSP
19.4
31.4
20.1
60M
40M
BWSP
19.9
32.4
20.4
57M
37M
Table COCO object detection results. We use GLIP with CLIP+DeiT-Sman as backbone.
our study to maintain the accuracy of the full model after compression. Due to space limitation, other implementation details are given in supplementary materials.
4.2 RESULTS
MDETR results. For MDETR, we construct two models, MDETR-Small and MDETR- Large, with different model settings. We report results on the validation dataset of Ref- COCO/R/CCCCO/ROOOCOO
is shown in Tab. [ From the table, we can see that the performance of BWSP is on par with the baseline model (relative changes:
and GQA. The result of MDETR-
for four tasks). At the same time, BWSP reduces 24M parameters (around 38% of two backbones) compared to the baseline model. LWSP and SP perform worse than BWSP, indicating that increasing flexibility is helpful when reducing the number of parameters. The results of MDETR- Large is shown in Tab. 2 On MDETR-Large, BWSP also performs similarly to the baseline model, but the parameter reduction rate is larger (40% of the baseline model). For both MDETR-Large and MDETR-Small. the manually designed sharing rule FS causes significant per- formance drops. In addition, given similar number of parameters, BWSP outperforms other full models with different backbones on both the large and small model setups. Finally, we remove R., during pre-training, and directly train the hypernetwork and fine-tune the model. We call this set- ting BWSP w/o Rm. As expected, BWSP w/o R.D results in much larger performance drop, which justifies the design choice of using R2 align model weights and structures during pre-training.
GLIP results. We evaluate our method with GLIP on MS-COCO object detection (COCO) and Flickr-30k for phrase grounding. The results on COCO are shown Tab. 3 From the table, can see that BWSP still outperforms SP and LWSP. Besides the performance, BWSP also removes more parameters. Tab. 4 we can have a similar observation. summary, BWSP provides more flexibility than SP and LWSP, which results in stronger performaanceparrameee trade-off.
4.3 DETAILED ANALYSIS
Effects R.... Fig. we plot the weight difference with without R. Let's consider cross- modal sharing as example, for ith layer, we plot Σn [[(1 8;) @ W4 (1 s:) @ W [|2 for regularized weights, and plot 1 Σ" ||s @ W - 4 1 for weights without regularization. The figure shows that the weight difference is much smaller with regularization for both pruning and cross-modal sharing. For layer-wise sharing, the difference is smaller since the ratio of layer-shared
8

---

Published as a conference paper at ICLR 2023
1/
007
101
Roga cual Woight
30"
Layer Indox
Layer Index (b) MSA CS
Layer Index (C) MSA LS
(a) MSA P
100
Weighhh
Indexe
Layorindas
(e) FFN
(f) FFN LS
Figure 4: Weights difference with or without the regularization R..
P represents Pruning, CS
represents modal Sharing, and LS represents Layer-wise wise Sharing.
Val
Test
Settings
#PT
#PB
R@I
R@5
R@10
R@
R@5
R@ 10
Backbones:
CLIP+DeiT
D--mmalll Small
Baseline
72.4
86.9
89.6
72.5
86.9
89.5
83M
63M
SP
71.6
86.4
89.0
71.8
86.3
89.3
63M
43M
LWSP
72.2
86.9
89.4
72.3
86.6
89.5
60M
40M
BWSP
72.4
86.9
89.4
72.7
86.7
89.5
57M
37M
Table 4: Flickr30k results. We use GLIP with CLIP+DeiT-Small backbones.
weights is larger. In conclusion, Fig. 4 shows that the proposed regularization R.w can effectively encourage sharing and pruning desired weights, and learning other weights is almost intact.
Analysis of Pruned and Shared Weights. In Fig. 3 we plot the details of pruned and shared weights of BWSP for MDETR-SIEE Small. We observe that the layer-wise sharing rate is large for both FFN and MSA layers of the text backbone. For the vision backhone, earlier layers are almost preserved both FFN and MSA layers. We also observe that FFN layers from the vision backbone shared more aggressively whereas MSA layers only shared for later layers. On the other hand, the ratio of pruning is not obvious except for the vision FFN layers. summary, the text backbone contributes more to the reduction of weights, which is reasonable since vision tasks more complex than language tasks in general for the tasks required for MDETR and GLIP models.
CONCLUSION
In this paper, we investigate how to perform structural sharing and pruning for grounding based vision and language models. Specifically, perform rross-modal sharing and pruning across the vision and text backbones and layer-wise sharing in the text backbone. With larger flexibility, becomes harder select the right set of weight vectors share and prune. To overcome this challenge, we propose an hypernttwork to capture the interactions of pruning and sharing across layers and modalities. Next, in the pre-training stage, we gradually align the backbones' weights with the desired architecture learned by the hypernetwork. Finally, the model is fine-tuned for downstream tasks. Our experiments on MDETR and GLIP show that we can reduce 35% 40% of model weights without significant loss in accuracy.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 06450, 2016.
arXiv preprint
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint /rXiv: htti::/30833322 2013.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34, 2021.
Xinpeng Chen, Lin Ma. Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint arXiv:18/2.03422 2018.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to- end visual grounding with transformers. arXiv preprint arXiv:2104.08 085441, 2021.
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: a no-recurrence geguence-o-sequeee model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888. IEFE, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview net. formm? d= cbFdNTTy
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.099 09574, 2019.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-languag representation learning. arXiv preprint arXiv:20066. 0995, 2020.
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9270-9280, 2021.
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer net- work. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2019.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. arXiv preprint arXiv:2201.08377 2022.
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified trans- former. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 1439-1449, 2021.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Andrew Jaegle, Sebastian Borgeaud, Jean- Baptiste Alayrac Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint 107v:2200..14795, 2021.
10

---

Published as a conference paper at ICLR 2023
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriiztioon with gumbel-softnax. arXiv preprint 1rXiv:1611.01 144, 2016.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr-modull detection for end-to-end multi-modal understanding. arXiv preprint vrXiv:2104.12765 2021.
Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution-free referring image segmentation using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18 18145-18154, 2022.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma. Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enabling faster vision transformers via soft token pruning, arXiv preprint 12. 2021.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: lite bert for self-supervised learning of language representations. arXiv preprint 1rXfv:1909.11942, 2019.
Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parame- ter efficient multimodal transformers for video representation learning. In International Confer- ence Learning Representations, 2021. URL 6JdgthhgJFDD
net forum?rd
Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. Efficient transformer-baset large scale language representations using hardware-friendly block structured pruning. arXiv preprint arXiv:2009.0800 2020.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Nent -eng Hwang, Kai-Wei Rei Chang, and Jianfeng Gao. Grounded ainuuag-image pre-training. In CVPR, 2022.
Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. arXiv preprint arXiv:2106.03003) 2021.
Qing Li, Boqing Gong, Yin Cui, Dan Kondratyuk, Xianzhi Du, Ming-Hsuan Yang, and Matthew Brown. Towards unified foundation model: Jointly pre-training transformers on unpaired images and text. arXiv preprint arXlv:2112.07074, 2021.
Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers images, vidcos and audio. 2021.
arXiv preprint arXiv:2111.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Qian Lou, Yen-Chang Hsu, Burak Uzkent, Ting Hua, Yilin Shen, and Hongxia Jin. Lite-mdetr: A lightweight multi-modal detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1P.1222 12206-12211 June 2022a.
Qian Lou, Yen-Chang Hsu, Burak Uzkent, Ting Hua, Yilin Shen, and Hongxia Jin. Lite-mdetr: A lightweight multi-modal detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12206-12215, 2022b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and- languuue----- tasks. arXiv preprint arXiv: 02265, 2019.
Kevin Lu, Aditya Grover, Picter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247: 20211
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 11-20, 2016.
11

---

Published as a conference paper at ICLR 2023
Bryan A Plummer. Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region to- to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pp- 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint S04iv:20030000200 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34, 2021.
Machel Reid, Edison Marrese- Marrese-Taylor, and Yutaka Matsuo. Subformer: Exploring weight sharing for parameter efficiency in generative transformers. arXiv preprint arXiv:2101.00234 2021.
Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. arXiv preprint arXiv:2104.06022, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint ar arXiv:2012.12877 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. arXiv preprint arXiv:2111.1000 2021.
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. arXiv preprint 1xXiv:200.001390 2021.
Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision transformer compression and parameter redistribution. arXiv preprint arXiv:2110.04869, 2021.
Zhengyuan Yang, Tianlang Chen, Liwei Wang. and Jiebo Luo. Improving one-stage visual ground- ing by recursive sub-query construction. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 387-404. Springer, 2020.
Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, and Bo Yuan. Gohsp: A unified framework of graph and optimization-based heterogeneous structured pruning for vision transformer. arXiv preprint N.Xiv:2301.05344 2023.
Haoxuan You, Luowei Zhou, Bin Xiao, Noel C Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Ma-clip: Towards podality-agnostie contrastive manguage-image pre-training 2021.
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre-training. arXiv preprint arXiv:2207. 12661, 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pp. 69 85-88. Springer, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara I. Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1300 1307-1315, 2018.
Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang. Ji Liu, and Zhangyang Wang. Unified visual transformer compression. In International Conference on Learning Repre- sentations, 2022. URL
12

---

Published as a conference paper at ICLR 2023
Haotian* Zhang, Pengchuan" Zhang, Xiaowei Hu, Yen-Chur Chen, Liumian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and 2022.
vision-language understanding. arXiv preprint unXiv:2206.000
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021a.
Zeyu Zhang, Thuy Vu, and Alessandro Moschitti. Joint models for answer verification in question answering systems. Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3252-3262. 2021b.
13

---

Published as a conference paper at ICLR 2023
A SUPPLEMENTARY MATERIALS
A.1 PRUNING AND SHARING FOR FFN LAYERS
We will provide more details for the pruning and sharing FFN Layers in this section. Recall that we have a weight matrix W E Rdxd' expanding the feature dimension, and another weight matrix W E Rd xd compressing the feature dimension for FFN layers. To keep the model dimension d unchanged, we conduct sharing and pruning along the dimension with size d'. For the cross-modal sharing, we then have:
E = s WE + (1-5 W
(12)
W s W T + (1 5²) W
like MSA layers, s is first resized have the same size of W For layer-wise sharing, have the following equations:
W = 5t o W + (1 50) o W E
(13)
W T = s O W T -11-s") o W
8 is resized have the same size of W Let's take the vision backbone as an example, the pruning for W and We is done by inserting the mask m" to the intermediate feature map between W and W X m @ X To simplify notations, we reuse the notations s' s and m for FFN layers. Similar to MSA layers, sharing and pruning for FFN layers also satisfy the constraints defined in 7q.7and Eq. 6
A.2 DETAILED STRUCTURE OF HYPERNETWORK
The detailed architecture summarized Tab. S1 The input the hypernetwork z E R21xx32
Table $1: The architecture of the hypernetwork.
Input
GRU(32,64) LayerNorm GeLU
(64, d')
Outputs df.. / = 1, L
(64, d%
Outputs o
... L
MLP; d:)
+Outputs of L 1....
(64, d - Outputs off I = 1,.
and is sampled from uniform distribution and it kept unchanged after the training starts. The outputs of GRUs are parallelly fed into MLP layers to get the outputs before the final pruning and sharing vectors. The final s m stt m ft is obtained by using the Gumbel-Sigmoid (Jang al. 2016) technique:
round(sigmoid 0, b)/7))))
m P2 round (simmoid + b)/)))
(14)
4 round(sigmoid o + g b)/T)),
m" ft roimmoidd + b)/T))
where sigmoid(:) ) is the sigmoid function, round(-) is the function that rounds its input to its nearest integer, follows Gumbel distribution: g ~ Gumbel(0, 1). is a constant and 7 is the temperature kept intact at the beginning. The
hyper-parameter. We set 3.0 that model structures set to 0.4 for all experiments. Since round(-) is not differentiable, we use the ttraight gradient estimator (Bengio et al. 2013) to circumvent this problem.
14

---

Published as a conference paper at ICLR 2023
L is the total number of MSA and FFN layers, and the order of MSA and FFN layers is followed by their natural order in the transformer. Recall that the model embedding dimension is d. and d' is the dimension after expansion for FFN layers. The dimension of each MLP is decided by whether is MSA or FFN layers. For sharing operations, d' and dh equal to 3d when they are MSA layers. For pruning operations, dy and dh equal to d, and they are shared for query weights and key weights. For FFN layers, dg d. d and d equal to d'.
A.3 ADDITIONAL DETAILS FOR EXPERIMENTS
Model
Backbones
Settings
p
SP
0.5
CLIP+DeiT-Small
LWSP
0.4
MDETR
BWSP
0,4
CLIP+DeiT-Base
BWSP
0.4
SP
0.5
GLIP
CLIP+DeiT-Small
LWSP 0,4
BWSP 0.33
Table S2: Choice of p for different models and settings.
The hyper-parametens À and for R and R.,. are set to 6.0 and 0.0005 for all datasets and tasks. We train the hyperettorkk with ADAM and a constant learning rate of 0.001 and weight decay 0.0001. For Dsub, we random sample 10% of samples from D. We found that this setting performs well for both MDETR and GLIP. Other pre-training and fine-tuning settings are identical to the original setting. In method, p controls how much compressible backbone pa- rameters should be removed. We list the detailed choice of in Tab. S2 ify p if we want change how much weight we want to compress. The results of all MDETR tasks are obtained by fine-tuning on downstream tasks. Results of GLIP on COCO detection task Tab. 3 are also obtained by fine-tuning. We follow the fine-tuning setting in their original pa- pers and codes") Let's take GLIP pre-taining as an example. We use the hyper-parameter given http:///ccoootiiii///////aaaaaaa
We only need to mod-
Swin T (3mml We re- placed BACKBONE, LANGUAGE BACKBONE, and DATASETS in the configuration file, and for training the hyperettwrk. For Flickr-30k results in Tab, we
we inserted add very small fine-tuning period (~ 3 epochs) to obtain them (including the bascline). For more information on the pre-training dataset, we refer the readers to the MDETR (Kamath et al. 2021
and GLIP (Li et al. 2022) papers.
In addition, the implementation of Eq. and Eq. 7 may not be straightforward at first glance. We provide PyTorch style pseudo code in this section to give example for Eq. The implementation of Eq. can be done similarly.
for in range [ cross_modal_gharede if tttps__mccaa__aaeeeeeeeae
contains indices of shared weights for all layers .neemeen () !- 0:
shared index contains indices of shared weights for the current layer matssmooaa___areeeas list
chared_index
apply the conscraint for vision praming vectore
vision_pruning vector contains aid pruning vectors,
dision_prunine wew.ioo_rrunnngeeeoooo.ooooo veclor_clone wision_pruming.uee sesuiing_vecto__la
cclone() (I).oneee))
apply the conseraint defined in 8g.6
=
the process is applied for text proning
Lext_pruning vee_oor_coooeeeeeeeeeeeeeeeeeeee
wtw.pprniing_ecctor_coooomm htt.cccoooosssse___iiii (adaaredddddxx]]] [i]]
catt_pruminn_rerrrr_lsttiii ott.rroe....sooooooooooooonnneen
clone
https:/ggihhhhccm///ssaaaaaaaaaa and http://giiaaa.ccmmmmieess///ht http://....a..aaoooooo...ooc/ine
15

---

Published as a conference paper at ICLR 2023
A.4 STUDY OF DIFFERENT SETTINGS
Settings
AP
AP50
AP
#PT
#PB
Backbones:
CLIP+DeiT-Smn -maall
Baseline
17.3
28.7
17.9
83M
63M
7 = 1 X 10
16.7
27.6
17.1
57M
37M
7 = 5 x 10
17.2
28.6
17.6
57M
37M
7 1 10-3
17.2
28.5
17.5
57M
37M
2
17.0
28.3
17.3
57M
37M
17.2
28.6
17.6
57M
37M
À 10
16.3
27.4
16.5
57M
37M
Daub
= 0.01
16.4
27.4
16.9
57M
37M
Duub 0.10
17.2
28.6
17.6
57M
37M
Daub = 0.20 D
17.3
28.7
17.7
57M
37M
16.8
28.1
17.2
57M
37M
B 3
17.2
28.6
17.6
57M
37M
16.4
27.5
16.7
57M
37M
Table S3: COCO object detection results. We use GLIP with CLIP+DeiT-Sman as the backbone. The results are lower than Tab. since the pre-training dataset is smaller due to the pre-training costs. Rows with blue color are our default settings.
To see how different choices of Hyper-parameterr affect our method, we adjust A. % Dubb (the number of samples of Dsab) and [B] (the number of blocks) for our method. The overall pretraining cost quite high for GLIP and MDETR. As result, we only use Flickr-30k (Plummer et al. 2015) as the pre-tran pre-training dataset. The experiments are conducted on GLIP with COCO object detection as the evaluation task. All other settings are the same as Tab.
is used to control the regularization strength of R.,. in Obj. From Tab. S3 we can see that smaller y will negatively affect the final results, and this probably because weights from different layers and different modalities are not well aligned, which creates difficulty for fine-tuning. This observation is consistent with the results BWSSP w/o R. (7 = 0) in Tab. Tand Tab. which all indicates that poor alignment will be harmful to the final performance.
used to control the regularization strength of the parameter constraint R(P(m, s),pP Pooa)) in Obj. 9 From Tab. [S]] we because too large À will ignore the pre- pre-training loss Cpeetraiiing and only focus on reducing the number parameters. As result, it hinders the hypernetwork from finding the ideal sharing and pruning structures.
that too large À (A 10) is harmful
method. This is
[Dubll controls the number of samples in |Dabll- From Tab. S3
that increasing the size of D is beneficial to the final performance. At the same time, it increases the pre-training costs. If D is too small, it will definitely negatively affect the results, as shown in the table. Daubll 0.10 creates a good trade-off between pre-training costs and the final performance.
[B] controls the number of blocks for cross-layer sharing. We tested three settings, and the results shown in Tab. S3 When B = 2, the final model has fewer unique base weights, which restricts the weight diversity. When [B] 6, the potential compression is limited. Clearly, when [B] 3, we can have good trade-off between the weight diversity and the potential compression rate for the text transformer.
16


Reviews:
[{"review_id": "zclfXDZ23A9", "reviewer": "Reviewer_FNmr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the weight pruning and weight sharing for image-text retrieval model. Contrary to previous works that applies the pruning and sharing on the layer level, the main idea of this paper is to apply weight sharing and weight pruning on the weight level. Therefore, the pruned model could achieve better compression rate and performance. The pruning and sharing is controlled by a hyper network, which is trained jointly with the image-text retrieval model. The proposed approach achieves better performance and better compression rate than the previous state-of-the=art.", "summary_of_the_review": "Given the improved performance and the novelty, I would recommend the rating of 8.", "strength_and_weaknesses": "The paper is interesting to read and the proposed idea is novel.\nI just have several quick questions and comments.\n1. The proposed approach seems could be apply not only on the image-text retrieval model, but also on single modal models (e.g. image only model and text only model). I wonder why would the author choose the image-text retrieval model?\n2. Just curious, as the ConvNet models also shares unified structure across the model, would the proposed approach work on the ConvNet models?\n3. How to set the base layers? Is it set heuristically?\n4. I might miss some details, but what is the 'z' in Eq.8 in implementation?\n5. Just curious, how to enforce the Eq.6 and Eq.7 during training?\n6. I wonder did the author observe any training instabilities for image-text retrieval model pre-training? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cDBJ7Ol8AL", "reviewer": "Reviewer_U8mM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel compression algorithm for transformer-based vision and language models. It trains a hypernetwork to identify whether a network weight is able to share across multimodal backbones and across layers within one backbone or is able to be pruned off the original networks. Experiments show that the proposed model compression algorithm can reduce up to 40% of parameters without significant loss of accuracy in several downstream tasks.", "summary_of_the_review": "The paper is well written and the proposed method is clearly explained.", "strength_and_weaknesses": "1. Strengths:\n\n+ The paper addresses an important topic and the proposed algorithm is well motivated. The paper is well written making it easy to follow.\n+ Experiments are done carefully and the results are convincing. \n\n2. Weaknesses:\n\n+ Some details are missing in the current version. In page 4, the authors divide a backbone into several blocks to identify whether network weights in those different blocks can be shared. By what criteria did you divide into these blocks? What is the optimal number of blocks? There is no justification by experiments or intuition for this in the paper. \n\n+ In page 4, after Eq. 5, the final W_q^vl is a combination of only two elements since s^l and s_{t_l} are binary. This makes the statement inaccurate.\n\n+ In page 6, the authors mentioned that they only used a subset of training data to train HN. How does this affect the compression performance? What proportion of the subset is sufficient? Again, there are no justifications for why the authors do this.\n+ The experiments conducted on the much smaller image size (224x224) compared to the original baselines, which raises doubts about the baselines' results.\n\n+ Have the authors tried methods such as grid search to find the best hyperparameters of the backbone for that small image input?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UB89y9TcA_", "reviewer": "Reviewer_p7Rq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a framework to perform weight sharing and pruning across two transformer backbones and within the same transformer backbone. The framework is evaluated on vision and language tasks, including Referring Expression Comprehension (REC), Visual Question Answering (VQA), and Object Detection. The results show that the proposed framework achieves a prominent trade-off between number of parameters and accuracy.", "summary_of_the_review": "The paper is well written and easy to follow and the proposed method is clear and well motivated. \nThe experiments are well designed and the results are convincing, but the paper could provide a more detailed analysis of the proposed method.", "strength_and_weaknesses": "Strength\n- The paper is well organized and convincing\n- The paper proposes a novel framework for weight sharing and pruning of transformer backbones\n- The method achieves a prominent trade-off between the number of parameters and the accuracy\n- The method is tested on various vision and language grounding tasks, including Referring Expression Comprehension, Visual Question Answering, and Object Detection\n\nWeakness\n- The paper does not provide a detailed analysis of the proposed method\n- Also, they only share part of hyper-parameters which makes the result hard to reproduce\n- Lack of justification for experimental setup, hyper-parameter selection for model and baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster