ADVL: Adaptive Distillation for Vision-Language Tasks
Authors: Zhecan Wang
Noel C Codella
Haoxuan You
Long Chen
Yen-Chun Chen
Yulei Niu
Jianwei Yang
Luowei Zhou
Lu Yuan
Kai-Wei Chang
Shih-Fu Chang
Venue: Submitted to ICLR 2023 (2023)
Keywords: vision language
kowledge distillation
vcr
vqa
snli-ve
visual question answering
commonsense reasoning
pretraining
multimodal
robust
low-shot
zero-shot
domain-shift
debiased

Abstract: Large-scale image-text pairs, such as image-captions and image-phrases, enable the strong representation of vision-language (VL) models. Nevertheless, they lose diversity and complexity due to the constraints in collecting data. Meanwhile, models pre-trained with image-only or text-only data (we call them unimodal pretrained models) continue to flourish and impress the community. Compared to image-text pairs, unimodal data has less constraints during the collection process resulting in more diverse styles. A natural question is how to leverage unimodal pretrained models to benefit downstream VL tasks? Most existing works focus on fusing VL information in the expensive pre-training stage. They directly plug in unimodal pre-trained encoders into a VL framework and redo an additional pre-training step on paired image-text data. This causes additional computation expense and the unimodal pretrained knowledge might be forgotten. In this paper, we take a different route and investigate how to fuse VL information in the finetuning stage oaly. To directly transfer pretrained knowledge from unimodal models to belp downstream VL tasks, we propose $\mathrm{ADVL}$, which avoids redoing any pre-training step and is generalizable to be applied of top of various VL models. To comprehensively demonstrate the effectiveness of ADVL, we conduct evaluation across three mostly recognized highly semantic VL benchmarks: VCR, VQA, and SNLI-VE under three settings, low-shot, full-shot and domainshifted settings. Results show that ADVL consistently improves the performance with different VL base models across all settings. It even achieves state-of-theart (SOTA) performance on VCR among models pre-trained with image-text data and delivers competitive results on VQA and SNLI-VE, Based on our analysis, we also discover that ADVL can improve the robustness of VL models and regulate them to better use vision information.

Under review as a conference paper at ICLR 2023
ADVL: ADAPTIVE DISTILLATION FOR VISION- LANGUAGE TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Large-scale image-text pairs, such as image-captions and smage-phrases, enable the strong representation of rision-language (VL) models. Nevertheless, they lose diversity and complexity due to the constraints in collecting data. Meanwhile, models pre-trained with image-only or text-only data (we call them unimodal pre- trained models) continue to flourish and impress the community. Compared image-text pairs, unimodal data has less constraints during the collection process resulting in more diverse styles. A natural question is how to leverage unimodal pre-trained models to benefit downstream VL tasks? Most existing works focus fusing VL information in the expensive pre-training stage. They directly plug in unimodal pre-trained encoders into a VL framework and redo an additional pre-training step on paired image-text data. This causes additional computation expense and the unimodal pre-trained knowledge might be forgotten. In this pa- per, we take different route and investigate how to fuse VL information the finetuning stage only. To directly transfer pre-trained knowledge from unimodal models to help downstream VL tasks, we propose ADVL, which avoids redo- ing any pre-training step and is generalizable to be applied on top of various VL models. To comprehensively demonstrate the effectiveness of ADVL, we con- duct evaluation across three mostly recognized highly semantic VL benchmarks: VCR, VQA, and SNLI-VE under three settings, low-shot, full-shot and domain- shifted settings. Results show that ADVL consistently improves the performance with different VI. base models across all settings. It even achieves state-of-the- art (SOTA) performance VCR among models pre-trained with image-text data and delivers competitive results on VQA and SNLI-VE. Based on our analysis, we also discover that ADVL can improve the robustness of VL models and regulate them to better use vision information.
INTRODUCTION
Recently, Vision-I Vision-Language (VL) models (Radford et al. 2021 Jia et al.
2021b Pham et al. 2021
Wang et al. 2021 Chen al. 2020; Su et al.) 2020 Gan al. 2020) pre-trained on paired image-
text data have achieved great success on many VL tasks (Zellers et al. 2019 Xie et al. 2019 Antol
al. 2015 Deng al. 2009). These paired data generally fall into two categories: curated image-
caption Sharma et al.
2018 Lin et al. 2014) or noisy online image-text data (Radford et al. 2021
Jia et al.
2021b Pham et al. 2021 Wang et al., 2021 Yu al. 2022).
However, primarily using paired image-text data for pre-training restricts the knowledge the model can learn. Image-text data are challenging to collect and their styles are limited. Image captions are typically short, template-like and with limited vocabulary (Chen et al.) 2015] [Sharma et al.) 2018). The left part of Fig. [ shows an example of image captions, "A glass of beer on a table" This description differs from texts used in many downstream applications, like VCR (Zellers al. 2019) and SNLI-VE (Xie et al. 2019). For these tasks, event inference, spatial/temporal scene understanding essential.
On the other hand, unimodal data, such as text, are relatively easier to collect and they cover a wider range of domains. For example, web-crawled text corporas (Raffel et al.) 2020 Aaron Gokaslan" contain longer sentences, paragraphs, stories, and articles with full context, In fact, there exist many rvaiile- unimodal models preited billion-scale data.

---

Under review as a conference paper at ICLR 2023
[BookCorpps)] T.The soceas to whose care ste was had per dreek, and poer listl thing exgesed to the orid, so flot dill....
(penued) Lamanily drank? Poaiilyy
[CC News) i.. ,..Cospae dee prices of beer at the supersalle and bas ciiz..."
Caption of
[OperWeb]
pastiiely own realy fur your
Painel Test
Unpaiedl
Unpaieed
Examples of paired, unpaired and downstream image and text data. Unpaired text and image have more diverse distribution in terms of contents and structures than paired data.
Figure 1:
Considering this, we posit that the diverse styles and sources of text and image data in unimodal pre- training can benefit downstream VL tasks, in addition to paired image-text data. Inspired by this, we study how to effectively and efficiently transfer pre-trained knowledge from unimodal encoders to improve downstream VL models.
In this paper, we propose ADVL, Adaptive Distillation that leverages unimodal pre-trained models for improving performance VL tasks. ADVL takes unimodal encoders as the teacher models and distill their knowledge into a student VI. model that pre-trained on multimodal data. To allow dis- tillation from vision encoders and text encoders from different pre-training sources, ADVL includes separate distillation pathways for vision and language. Further, two adaptive mechanisms, Adaptive Confidence-bul Weighting and Adaptive Text Token Selection, are proposed in instance-level and token-level to dynamically adjust the significance of the distilled knowledge. Finally, an adaptive two-step finetuning strategy is introduced to mitigate the domain gap in the distillation.
Prior works (Shen et al.) 2021| Tan & Bansal, 2019; Li et al. 2019) has proposed to initialize weights from unimodal vision or text encoders when pre- traiing the models with paired image-text data. Some other studies (Li et al. 20200 /Zhou et al. 2022) have focused on joint pre-training with both multimodal and unimodal data. As opposed these works, ADVI. leverages unimodal pre-trained models when fine-tuning VL model for a downstream task. This setting has the following benefits: (I) Computation and energy efficient: pre-training VL models is resource-hungry and may not be practical when computation resource is limited. ADVL avoids redoing the pre-training step. (2) Modularization and future proof: ADVL allows researchers to integrate new pre- tre-trained vision, text and multimodal encoders when they become available. This is essential more and more large unimodal and multimodal pre-trained models are released. (3) Flexibility and generalization: The flexibility of selecting various combinations of pre-trained encoders teacher models enables researchers to explore and find the optimized teacher models (c.g.. in the same domain) for specific downstream task.
To verify the effectiveness of ADVL, we implement ADVL on top of several high-performing VL models and evaluate the results on 3 popular VL tasks, VCR (Zelers et al. 2019), SNLI-VE (Xie) et al.] 2019), and VQA (Antol et al. 2015), under low-shot, full-data, and domain-shifted settings. With extensive experiments, we show that ADVL improves the performance of VI models on var- ious tasks across all settings by levering unimodel encoders. Moreover, it achieves SOTA perfor- mance on VCR- VCR-Q2A compared to other non-ensembled models pre-trained with large image-text data. It also achieves competitive performance SNLI-VE and VQA. Furthermore. the existing VL models tend to under-utilize vision information. With ADVL, the vision information can be better used. As result, the model is more robust in the domain-shifter setting.
discover
We plan to release the code to facilitate future research in the community.
RELATED WORK
Pre-training with image and text data: Existing pre- mre-training frameworks leveraging image and
text data can fall into three essential categories: (1) Unimodal pre-training including image-only
pre-training, egg. (Dosovitskiy et al. 2020: Liu et al. 2021b Dai al. 2021) or text-only pre-
2

---

Under review as a conference paper at ICLR 2023
training e.g., (Devlin et al.) 2018: Raffel et al. 2020 Radford et al. 2019b a Liu et al.
2019b);
(2) VL pre-training with vision and text encoders, e.g. (Shen et al.
2021] lan Bansal 2019 Li
et al.
2019. Radford et al. 2021 Mu et al.) 2021: Li et al.) 2021b Jia et al., [02 laj Pham et al. Yu et al. 2022). Although they are pre-trained with paired image-text data, their structures incorporate independent vision and text encoders. For example, CLIP (Radford et al.] 2021] has one vision encoder and one text encoder loosely connected by a contrastive loss in the end. Among them, frameworks like (Shen et al. 2021] Tan & Bansal 2019 [i et al. 2019) utilize the unimodal pre-trained encoders by directly plugging them into the VL frameworks. Following, they have to conduct an additional step of pre-training on image-caption data before finetuning for downstream tasks. (L) pre-training with cross-modal structures only, e.g. (Su et al. 2020; Chen et al.) 2020)
2021
Gan et al. 2020 Zhang et al., 2021: Wang et al. 202T). They enable in-depth integration between vision and language information in the pre-training stage and thus often achieve leading performance
in highly semantic VL benchmarks like VCR.
In this work, we emphasize utilizing the learned knowledge from unimodal pre-trained encoders like RoBERTa (Liu et al.) 2019bb and ViT (Dosovitskiy et al. 20200). We would further leverage the pre-trained vision and text encoders from VL pre-training like CLIP (Radford et al. 2021 for the comparison purpose among experiments.
Knowledge Distillation: Former works use knowledge distillation for model compression (Buciluă
et al. 2006; Hinton et al., 2015: Wang & Yoon| 2021 Xu et al., 2020 Liu et al. 2019a). Among them. Sanh et al. (2019): Jiao al. alll (2019); 1 (Snn al. (2020b) focus compressing language mod- els and Tian et al. 2020); Fang et al. (b021b) focus on vision models. Past methods leverage various
representations for transferring embeded knowledge, e.g. logit distribution, intermediate feature em-
beddings and attentions (Kim & Rush [016]] Sun et al. [2019] Liu et al.] 20200] Yang et al.) [020]]
Chen al., 202T). Other methods, [Tian et al. (2019); Sun et al. (2020a) also try create more robust distil lation mechanisms like contrastive distillation. Li et al. (202 la); Niu & Zhang (2021);
Wu et al. (202T); Furlanello et al.] (2018); Kang et al. (202)) ; Xiang et al. (2020) create different
dynamic and adaptive distillation methods like adjusting distillation weights. Related, our method pushes further to propose adaptive mechanisms like instance-level confidence- paceeee-ppedddd weighting and token-level text-token selection. Together, they help adjust the distillation weights among text tokens and teacher models. With the popularity of VL learning, recent methods investigate on transferring
knowledge between modalities (Jin et al. [2022] Gupta et al., 2016). Prior works (Do et al. 2019
Fang et al. 2021a] Mun et al.] [208]] Huang et al. 2022) also experiment with transfering VL knowl- edge between models. However, their approaches do not separate distillation procedures between modalities thus they require both the teacher and student models to be VL models with similar structures. Differently, ADVL incorporates independent distillation procedures for both vision and language, and this allows it to distill knowledge from different pre-trained models. Further, Tang jet al. (202T): Liu et al. (0211)); Khan et al. (2021): Croitoru et al. (202T) exam distillation between a single-modal model and a VL framework. Similarly, concurrent works (Zhang et al., 2022 Dai jet al. 2022, Gu et al.) 2021) explore distilling knowledge from the single-modal encoders within VL contrastive pre-training frameworks. However, they both can utilize at-most one single-modal model as the teacher during distillation. In contrast, our method can simultaneously distill knowl- edge from vision and text encoders separately the dual-teachers, ADVL is generalizable it not only can distill from unimodal pre-trainte trained encoders like RoBERTa (Liu et al. 2019b) but also from vision text encoders within VL pre-training frameworks like CL.IP (Radford al. 202T).
To our best knowledge, ADVL is the first approach to distill knowledge from mulitple unimodal pre-trained encoders assist downstream VL tasks. We also conduct the first study the chal- lenging low-shot evaluation setting for VCR and SNLI-VE. See A.4.2 for additional evaluation and discussion.
METHOD
Problem Definition: We assume two pre-trained teacher models, fú, and fú, and pre-trained student model f.. Our goal is to distill the knowledge from the teacher models to the student on a specific task D with input image i, and text t.
3

---

Under review as a conference paper at ICLR 2023
Nera
1g Nove
4.1, Nore
Kieine Uiionee
(0) 4/-3
Stobal Pooking
Global Pooling
Cip
Clip Text Encoder
Keywond Extraction
VL Studeat
T Teacher
Vision
Tescher
0.00
o*
Imoge
Text
Imroge
Text
Image
(Why is she unhappy?? ... loud nuise.)
(+)
Figure 2: Structure diagrams of Modality-specifie Distillation (a) and Text Token Selection (b). On the left diagram (a), within ADVL, every VL model the student would have vision teacher and text teacher to conduct knowledge distillation separately. One the right diagram (b), the text would be pre-processed via tokenization and stopwords removal before input into the Clip text encoder. The keyword extraction is conducted by off-shelf keyword extractor.
Overview: In the following, we first introduce distillation in transformer-basee models and how ADVL separates distillation pathways for vision and language. Then, we discuss the adaptive mech- anisms of ADVL. Finally, we introduce the adaptive two-step finetuning strategy that further helps VL models to adapt the unimodal pre-trained knowledge gradually in the finetuning stage. On both of the finetuning steps, all the adaptive mechanisms of ADVL are applied at the same time.
3.1 MODALITY-SPECIIC DISTILLATION
First, we identify attention layer in the student VL model and attention layer in the teacher model. Then, we conduct knowledge distillation between them. Typically, this would be between the last layers of the teacher and the student model (Sanh et al. 2019; Jiao et al., 2019 Kim & Rush, 2016)) [unn et al.) 2019). Different from former methods that do not differentiate distillation process between modalities, tion so that we can flexibly distill knowledge from various pre-trained vision and text encoders from different sources.
propose separate the distillation of vision and language informa-
For language distillation, in conventional methods (Sanh et al. 2019 Jiao et al. 2019 Kim & Rush 2016 Sun al. 2019), the hidden features of the leading classification token, [CLS], are commonly extracted and used for predicting final labels. Similarly, on either the teacher or student model, we also extract the [CLS] token features from the identified attention layer to represent the information of text sequence. As for vision distillation, in the target distillation layer of either a teacher or student model, following [u et al. (019), we, on purpose, globally average-pool across features of all the visual tokens to create global representative visual features to represent the visual sequence.
After extracting the representative text and visual features, we first normalize them by dividing them by the L2 norm and then calculate the feature distance separately for the two modalities via LI measure to obtain the corresponding distillation losses. For the text modality, we have
(t) (1)
fa. (t)
La,-
(1)
(1a
where t refers to the imput text; fú, and fa, refer to the feature extraction of the representative text features in the teacher and student model; Similarly, when the teacher model is a vision encoder, then define Ta accordingly.
4

---

Under review as a conference paper at ICLR 2023
What
percant dong
(SEF) le In sending
takagu
Derj
Taken Selection Por
Figure 3: Token-level attention values before and after distillation. The top and bottom rows visu- alilze aggregated token-level attention values of heads from VL-BERT before and after distillation. The middle row indicates the emphasized and selected tokens by using language prior.
3.2
ADAPTIVE MECHANISM
3.2.1
ADAPTIVE CONFIDENCE-BASER WEIGHITIGG
As the VL student model improves, it may become more confident and accurate in generating predic- tions for certain instances than the teacher models. With those instances, prefer less interference from the teacher models. Considering this, we design a confidence-bart weighting mechanism to allow the student model to adjust the distillation weight based its confidence.
For each input instance, we first extract the logit vector, 1' output by the student model. We then compute the probability distribution over all classes, (?*), by applying a softmax function, o, top. The class with the highest probability is selected and the probability can be regarded as the highest class confidence score, C max (a (7*)), of the model's prediction for that instance.
Inspired by Xiang et al. (2020), we further evaluate the student model in the validation set at the end of each training epoch and record the model's highest class confidence score when the student model predicts correctly. Based on these scores, we calculate the average class confidence score, Cavg when the student makes the correct prediction. We can regard Cavg reference value with which the student is confident and accurate in its predictions.
During training, for each input instance, we also calculate the student model's highest class confi- dence score, C. Comparing against the pre-computed reference score, = can tell us how much the student is confident about the classification prediction for that instance. As train the student model from epoch to epoch, the reference score, Cavg will also be updated accordingly to reflect the student's s confidence in accurate classification predictions. Considering the student model may have low confidence at the initial epochs, we also set a minimum value, 0.5 for the reference score. Following, we utilize the ratio in the equation below adjust the distillation weight during training:
w T Ssigmoid mmax(Cavg 0.5) -1)
(2)
C'
where fsigmoid represents the sigmoid function. is a learnable temperature value. We directly compare the ratio against 1 via subtraction to determine how confident the student model is for an input instance. If the resulting value smaller than 0, the student is confident about the prediction on the instance. Hence the overall distillation weight would be decreased accordingly. Otherwise, the student is considered less confident than usual. Then the student may need more help from the teacher model via distillation for that specific instance.
Formally, the distillation loss would then be:
La = w (F T.d, + F. Laas) where and F are learnable weights for vision and language distillation respectively.
(3)
3.2.2 ADAPTIVE TEXT TOKEN SELECTION
Former works Ye & Kovashka 202 Cao et al. 2020 Ramakrishnan et al. 2018 Kovaleva et al. 2019) point out V+L models are prone to over-rely on trivial information within text data. For instance, for an input text sequence, existing models still struggle in differentiating semantically important text tokens from the trivial ones. As example in Fig. 3 among the input text sequence, tokens like "person4", "doing", "sending" and "telegram" are semantically important since they con- tain crucial information for understanding the visual scene and answering the question. Yet, from
5

---

Under review as a conference paper at ICLR 2023
the top row of visualized attention aggregated over heads from VL-BERT, mistakenly focuses on trivial tokens like "is" and "[SEP]" and neglects the important tokens. This may further result in failure to understand the visual scene. To resolve this, we intend to incorporate language priors to help differentiate semantically important tokens in every input text sequence. As in the second row of Fig [3] relevant tokens emphasized and selected with language prior. Conse- quently, we can further utilize the selected tokens by guiding the knowledge distillation procedure more refined token-level. This may help the student model in learning the semantic information of the selected tokens instead of shortcuts or biases from the trivial tokens, as shown in the third row of Fig. 3
observe that it
To achieve it, we measure the semantic importance of each token based on the input image and text. Specifically, we generate the semantic importance score 8², for each token e',j E [1, based on its visual relevance against the image and syntactic importance within the text sequence. Therefore, every z is summation of two subsequent scores, visual relevance score, s and syntactic
importance score, sy- We first apply pre-trained CLIP model (Radford et al. 2021 to measure the visual relevance score between every token and the input image. Thereby, S. = { = {cos (fr. fc.t (e')) [1, =1}. Here, fe,: represents the global feature extraction by CLIP's visual encoder and fa, denotes output features local tokens from the text encoder. The cosine distance between the extracted visual and text features would be served as the visual relevance score. Further, we also adapt a pre-trained off-shelf keyword extraction model (Campos et al. 2020) to produce the syntactic importance score, S = { After summation, S |s.h
S. + S
St We
then rank text tokens based on S and select tokens (A.1.I) with the highest scores, e Their corresponding features of both teacher and student models would be then compared with an L1 measure calculate their difference:
{e" {e"r==
m Σ
(er lffa (e))
far (ef) Ifaa
La
(4)
La,'' would be furhter added update the distillation loss in Eq. B The final loss consists of the downstream task loss, Leask; e.g., classification loss in VCR and the newly updated distillation loss with the distillation weight, wt
Lifinat = Leask + w La= = Leask + w (E Ld,o + F. (Ia, + La,t))
(5)
3.3 ADAPTIVE TWO-STEP FINETUNING STRATEGY
Domain gap exists when directly transfering unimodal pre-trained knowledge specific down- stream task. This may be due to the distribution differences between the pre-training and down- stream data and discrepancy in optimization directions. With the aim ease the gap, design an Adaptive Two-step Finetuning.
Adaptive Finetuning with Contrastive Learning: We first finetune the base student VL model on a downstream dataset with Image-Texx Matching (ITM). Moreover, we also incorporate the knowl- edge distillation on parallel as an additional task. As the base student model is adapting to the downstream dataset, the distilled knowledge from pre-trained teacher models can assist the student learn nataee differeetiiaton,, differences between image-text pairs. Specifically, on VCR, we convert every question and correct answer pair into a text statement with heuristic rules (details A.1.3) and regard each image-text pair as the positive sample. We also apply similar procedures on VQA and SNLI-VE. For each positive sample, create two corresponding negative samples. In the first negative sample, we swap the image with another randomly selected one; the second, swap the text.
During finetuning, we extract the representative text features and visual features to represent the text and visual sequence respectively. These two features then input into prediction head, MLP module, to predict the probability that the input image and text is a correct pair. The contrastive learning of ITM is via calculating the InfoNCE loss (Oord al. 2018):

---

Under review as a conference paper at ICLR 2023
Vision
Lang
Conf. W.
T.T. Select.
T.S. Fine.
100TTyp 30.85
1000/Type
Full
53,48
75.53
34.24
54.53
75.92
35.96
55.04
76.15
37.21
57.64
76.83
28
58.68
77.15
40.43
58.98
77.61
Table I: Ablation experiments using VL-BERT as the base student model and evaluated on the VCR standard validation set. The visual teacher is pre-trained SwimTransformer V2 and the lan- guage teacher is a pre-trained RoBERTa. Vision represents vision distillation and Lang. represents language distillation. Conf. W. represents Distillation with Confidence Weighting, T.T. Select. rep- resents Distillation with Text Token Selection. Fin.Fine. represents Adaptive Two-Step Finetuning.
exp sim (2)
LITM = - log
(6)
exp (iim (2) 2f) /T) + exp (sim (2) 22f
/T) + exp (sim (2k 24)
/T)
where i * and i * k. sim represents the predicted probability by the prediction head. is hyperparameter temperature 1A.1.2). 2 and =!" are the visual and text representative features for positive pair. z and zif are the visual and text representative features of other samples.
st stage,
Lffint = LITM + w La Lftnnl = Leack + w Ls
2nd stage,
Final Finetuning: Lastly, we finetune the base student model with the downstream task specific loss and also keep the knowledge distillation on parallel. The final loss is listed as above.
4 EXPERIMENT
4.1 MODEL
Teacher Model: As mentioned Sec. 2 for comprehensive comparison, we experiment with wide range of encoders from unimodal pre-training or VL pre-training with image-text pairs. For the pre-trained visual encoders, we utilize ViT, V2 CLIP-V and SLIP-V; For the text pre-trained encoders, utilize RoBERTa, CLIP-T and SLIP-T.
Student Model: We adapt top-performing and publicly available VL models for highly-semant VL tasks, including UNITER (Chen al. 2020), VL-BERT (Su al. 2020), and VILLA (Gan et al. 2020). All these models represent variations of multimodal transformer- based architectures.
4.2 DATASET AND EVALUATION
To verify our claim that the unimodal pre-trained vision and language models can help to improve the permanence on downstream VL tasks. We evaluate ADVL across the three popular
semantic VL datasets: VCR, VQA and SNLI-VE under three settings: low-shot, full-data and domain-shifter settings.
Low-shot: Thre are question types in VCR, 3 in SNLI-VE and 8 in VQA. For a better comparison, we design have two standardized evaluation settings for all them: (1) Selecting 100 images for each type, (100/Type) and (2) Selecting 1,000 images for each type, (1000/Type) (More details in Appendix A.3)). Few-shot results are averaged over runs (see Appendix A.4. A.4.2 for standard deviations).
Domain-shift: For VCR, we follow an existing Nomainn-siifeen setting Ye & Kovashka 2021) (Ap- pendix A.3.I. For SNLI-VE and VQA, also follow similar procedures in (Ye & Kovashka 2021) to replace pronouns based on list of heuristic rules to create domain-shifted settings. (Ap-
pendix A.3.2 A.3.3)

---

Under review as a conference paper at ICLR 2023
Teacher Model
VCR 1000/Type
SNLI-VE 10007Type
VQA
Vision
Lang.
100/Type
Full
100/Type
Full
100TTpp
10000/Type
34.84
57.01
78.27
58.47
67.16
79.64
37.18
65.75
72.11
SLIP-V
SLIP-T
38.92
57.38
78.28
58.67
67.23
79.37
39.07
65.89
72.14
CLIP-V
CLIP-T
42.04
59.42
39.04
68.37
79.96
39.34
66.10
72.43
CLIP-V
RoBERTa
40.71
59.36
78.93
58.77
68.69
79.75
39.12
66.34
72.79
VIT RoBERTa
58,14
78.42
58.44
67.56
79.24
39.01
66.17
72.49
SwinV2
RoBERTa
41.87
60.18
79.13
59.07
68.83
80.87
39.67
66.96
73.04
Table 2: Knowledge distillation from combinations of paired encoders for VI. downstream tasks. Vision represents vision teacher model and Lang. represents language teacher model.
4.3 ABLATION
As in Tab. 1 we performed an ablation study of ADVL with VL-BERT the student model VCR under the low-shot and full-data settings. The goal here to understand how each ADVL component improves the performance. These results demonstrate 2 key observations: 1) Language distillation contributes more than vision distillation (as studied in previous works (Shen et al. 202I Hence it is incredibly important to include it in multimodal distillation. Further, distilling both vision and language perform better than either one alone. 2) Each component inside ADVL can bring consistent improvement over the course of evaluation the validation set of VCR-Q2A.
Distillation with Various Teachers: Our method is flexible that text encoders as teacher models and conduct modality-specific distillation separately.
have separate vision and
For demonstrating the benefits of distilling knowledge from unimodal pre-trained encoders for downstream VL tasks, we experiment ADVL on top of VILLA Gan et al. (2020) as the base model distill from mutliple combinations of teacher models. Comparing among the first three rows, observe that when applying ADVL to distill from dual encoders of CLIP and SLIP the downstream performance can improved. Moreover, vision or text encoder from a VL pre-training can also pair with unimodal encoder pre-trained with image-only or text-only data to serve as the dual teachers. As in the 4th row of Tab. P ADVL even enables downstream VL. models to flexibly dis- till learned knowledge of CLIP-V from VL pre-training and learned knowledge of RoBERTa from unimodal pre-training. Lastly, we also pair two independently unimodal pre-trained encoders as in the final two rows. We can spot that with Swin Transformer V2 and RoBERTa, we can achieve the best performance among the different combinations of paired teal-re This further verifies the value of leveraging knowledge in unimodal pre pre-training downstream VL tasks.
During experiments, we discover that combinations of teacher models correspond to different set- tings of picking attention layers within the student VL. model the optimal target distillation layers. When distilling from two independently unimodal pre-trained encoders, we find that selecting the center intermediate layer of the student VL model would deliver the best performance. For instance, select the 12th intermediate layer as the target distillation layer out of VL-BERT, ,, s 24 layers. When distilling from vision and text encoders of a VL pre- pre-training framework, such CLIPs, selecting the final hidden layer of the student VL model is optimal. Refer to Appendix A.4. for more ablation.
4.4 COMPARISON WITH BENCHMARKS
4.4.1
VCR
Results on the VCR dataset for Q2A. QA2R and Q2AR are shown in Tab. For Q2A, we also conduct evaluations with several data availability and domain-shifted constraints. Based on our former evaluation with different tual-teachers, Tab. 2
we apply ADVL. with the overall best dual- teacher combination: SwinV2 and RoBERTa. Although ADVL avoids redoing any pre-training, for comparison, we also include a recent po--erforming method utilizing CLIP vision encoder with additional pre-training here for comparison (CLIP- ViL..) We find ADVL can improve downstream VL models consistently across different settings, and surprisingly bringing stronger boost on low- scale and Homain--shitee settings.
ADVL with VILLA delivers high performance on two public leaderboards: (1) On the A12 hosted public leaderboad, it achieves new SOTA across three metrics with (Q2A: 80.9% QA2R: 83.3%
8

---

Under review as a conference paper at ICLR 2023
Q2A
QAZR
Q2AR
Model
Val.
Test
Val.
Test
Val.
Test
Topo/Typp
Full
FallDomai-shffiee
BERERT
30.85
53.48
75.53
71.13
75.8
77.91
78.4
58.85
39.7
VL-BERT+
40.43 (+9.58)
58,98 (+5.5)
77.61(+2.08)
74.55 (+3.42)
79.02 (+1.1))
61.33 (+2.48)
UNITER
31.43
54.24
76.67
73.84
77.3
79.98
80.8
61.32
62.8
UNITER
42.23 (+10.8)
59.88 (+5.64)
77.05 (+0.38)
74.94 (1.1)
80.57 (+0.59)
62.08 (+0.76)
VILLA
34,84
$7.01
78.2
75.43
78.9
82.33
82.8
64.44
65.7
VILLA-
41.87 (+7.03)
60.18 (+3.17)
79.13 (+0.86)
76.32 (+0.89)
79.6
82.57 (+0.24)
82,9
65.34 (+0.90)
66.2
CLIP VILP
34.63
53.54
68.36
66.83
Table 3: Results on VCR with different VI. architectures. A model with * represents we apply
ADVL on top of it. Baseline results are based re-implementation of the VL models, Val. represents evaluation on validation sets.
Model
[0000Type
VSl.
Full
SNLE-VE
FullDemain shilled
Text
1001ypp
[000FType
Val.
VQA
FulFDomann shilted
Test-Val
Test-Std
DERT
62.31
14.55
7096
15:33
63.29
69,05
71.75
72.22
VL-BERT
56.78
65.37
75.75 (+1.09)
73.19 (+2.3)
75.43
37.12
66.71
71.42 (+2.37)
70.58 (+2.87)
UNITER
79.02
76.72
36.46
71.26
73.82
74.02
UNITER
68.34
90.14
78.54
39.75
65.93
71.94
70.18 (+0.3)
77.22
37.18
74.60
74.87
VILLA
(+1.23)
79.56
72.74
75.81
Table 4: Results on SNLI-VE and VQA. A model with represents we apply ADVL top of it. Baseline results are based on our re-implementation of the VL models.
Q2AR: 67.7%) compared to other published single models pre-trained with image-text data (2) On UW public leaderboard: With (Q2A: 79.6% % QA2R: 82.9 % Q2AR: 66.2%) ), it also achieves new SOTA of Q2A performance compared to other published single models pre-trained with image-text data, as well overall competitive performance for QA2R and Q2AR.
4.4.2 SNLI-VE AND VQA
Results on SNLI-VE and VQA are shown Tab. We observe that ADVL improves downstream VL models consistently across different settings both SNLI-VE and VQA. The improvement low-scale and domain-shifted settings is even more significant. Besides, on SNLI-VE, ADVL out- performs concurrent work CLIP-ViL, under all evaluated conditions. On VQA, ADVL outperforms finetuning approach CLIP-ViL, under the few-shot settings and achieves comparable performance CLIP-VIL, under the full-shot setting. Furthermore, under the domain-shifiee setting, ADVL outperforms CLIP-VILE by large margin. Applying ADVL on top of an ensemble of VL models, such as VIILA and VI.-BETT, lead to 74.29 on local validation, outperforming CII.
4.5 INCREASED UTILIZATION OF VISION MODALITY
ADVL improves VI. models on varioius downstream tasks. After carefully examining why ADVI. improves VI. models, we are surprised to discover that existing VL. models underutilized vision information. In particular, following Cao et al. (2020), vision and language modalities and plot the result in Fig (see appendix). We observe that ADVL increases the Vision MI shown in Fig 3 (a) and the MI difference between Vision and Text is decreased as shown in Fig (b). Both observations indicate that ADVL encourages base VL models to better utilize vision information. Details are in the Appendix.
measure Modality Importance (MI) of
5 CONCLUSIONS
We initiated this study with the motivation to explore how to utilize the unimodal pre-trained knowl- edge to help the finetuning in downstream VL tasks. Instead of fusing VL information in the pre- training stage, we focus on the finetuning stage only. With our new approach, ADVL, we can flexibly distill pre-trained knowledge from different pairs of vision and text encoders. With exten- sive experiments, we verify that unimodal pre-trained knowledge is beneficial to direct finetuning on downstream VI. tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ellie Pavlick Stefanie Tellex Aaron Gokaslan*, Vanya Cohen Openwebtext corpus.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference Computer Vision (ICCV), 2015.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Ja- towt. Yake! keyword extraction from single documents using multiple local features. In- formation Sciences, 509:257-289, 2020. ISSN 0020-0255. doi: http://ddi.org/1001016...
2019.09.013. URL https: www. secenceiireecc.................. www.//o....s.eeeeeeeecccc.eeeeee cct.//cleeecee/rriiel/////o
550200255193045585
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained vision-and- and-language models. In European Conference on Computer Vision, pp. 565-580. Springer, 2020.
Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 16296-16305, 2021.
Xinlci Chen, Hao Fang, Tsung Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325,, 2015. URL htttp://// wwwpexiwv.org//b.o. Bttg/abs//s0 hwt.ab//115.00...
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
Ioana Croitoru, Simion-Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11583- 11593, October 2021.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung, Enabling multimodal generation clip via vision-language knowledge distillation. arXiv preprint arXiv:2203.063: 2022.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 10.10009RRR22000.55222 5206848.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Tuong Do, Thanh- Toan Do. Huy Tran, Erman Tjiputra, and Quang D. Tran. Compact trilinear inter- action for visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:20I0.I 11929, 2020.
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Com- pressing visual- lingul-ing model via knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1428-1433 2021a.
10

---

Under review as a conference paper at ICLR 2023
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint arXiv:2101.0473, 2021b.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616.
PMLR, 2018.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-languaal representation learning. arXiv preprint arXiv:2006. 200i:::000.069999 06195, 2020.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language
knowledge distillation. CoRR, abs/2104.13924 2021. URL https::/arxiv.orgg/
2104.1392.
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2827- 2836, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint 350300/500.053/// 2(7), 2015.
Bo Huang, Shuai Zhang, Jitao Huang, Yijun Yu, Zhicai Shi, and Yujie Xiong. Knowledge distilled pre-training model for lawplllaggnggggggatiiiii Applied Intelligence, pp. 1-13, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. CoRR, abs/2102.059188. 2021a. URL https::/arxiv.cr
abs/2102..55 05918
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and langual representation learning with noisy text supervision. arXiv preprint arXiv:2102.059// 2021b.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:/909 I, 2019.
Woojeong Jin, Dong-Ho Lee, Chenguang Zhu, Jay Pujara, and Xiang Ren. Leveraging vi- sual knowledge in language tasks: An empirical study intermediate pre-training for cross- modal knowledge transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2750-2762, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/218 199. 196. URL https://aeLaatthooooo..oo////// www.oo/oo......ooooo.......... 196
Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. Instance-condittooal knowledge distillation for object detection. Advances in Neural Information Processing Systems, 34:16468-16480, 2021.
Humair Raj Khan, Deepak Gupta, and Asif Ekbal. Towards developing a multilingual and code-mixed visual question answering system by knowledge distillation. 1.Xiv:2109.04633, 2021.
arXiv preprint
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. 1rXiv:1606.00947 2016.
arXiv preprint
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets 2019.
of bert. arXiv preprint
Lei Li. Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Dynamic knowledge distillation for pre-trained language models. arXiv preprint arXiv:2 /295, 2021a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: (cX.vv/908.03557 2019.
11

---

Under review as a conference paper at ICLR 2023
Liunian Harold Li, Haoxuan You, Zhecan Wang, Alireza Zareian, Shih-Fu Chang, and Kai- Wei Chang. Unsupervised vision-and- languugg pre-training without parallel images and captions. arXiv preprint w.Xv:2010..22311, 2020.
Yangguang Li, Feng Liang. Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre- training paradigm. arXiv preprini arXiv:2110.05208 2021b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pp. 2604-2613 2019a.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 2019b.
Yongfei Liu, Chenfei Wu, Shao-yen Tseng, Vasudev Lal, Xuming He, and Nan Duan. Kd-vlp: Improving end-to-end sasion-and-lnngunn pretraining with object knowledge distillation. arXiv preprint arXiv:2 /09. 10504, 2021a.
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocompuling, 415:106-113, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012-10022, 2021b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for assion-and-lagguaa tasks. Advances in neural information processing systems, 32, 2019.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, arXiv preprint arXiv:2112. 127.5000,, 2021.
Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. Learning to specialize with knowledge distillation for visual question answering. In NIPS, 2018.
Yulei Niu and Hanwang Zhang. Introspective distillation for robust question answering. In
M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 16292-16304. Curran As- sociates, Inc., 2021. URL https:ttt//..m...... http:::ess///poceeeeeeeee igs. wew.ing....nnuureeee neurips...... wttpr/pps.c//ppppe/////... pctppp//e//222////1ll//////// 87 17 Papeedddddpppp..........
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 10X.v:1807.03746 2018
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingx- ing Tan, and Quoe V Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.1005 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019a.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020) 2021. htt.https://arxt wwwaaaxiivvooogaaaae /tw.r////////// 22w.. 00020
12

---

Under review as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming language priors in vi- sual question answering with adversarial regularization. arXiv preprint arXiv:/810.0366 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 108, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACI., 2018 8.
Sheng Shen, Liumian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit
and-language tasks?, 2021.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre- International Conference on Learning training of generic disual-linguiste representations. net wttp://eeeeooonooooooooeeeeeree Representations, 2020. URL https: http://ppperevveeeeee httpps////pp...e.ee.......
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: / (al:::908.09555 2019.
Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. arXiv preprint arXiv:2009.14100 14167, 2020a.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact task-agnostic bert for sesource-limiteer devices. arXiv preprint 2rXiv:0004.02984, 2020b.
Hao Tan and Mohit Bansal. LXMERT: learning cross-mddality repeesenttooooostttaiiiiiiooo from trans- formers. EMNLP, htt//190.074440 2019. URL
/ htttt:://axx.i.oo//bb/////// 07490
Zineng Tang, Jaemin Cho, Hao Tan, and Mohit Bansal. Vidlankd: Improving language understand- ing via video-distilled knowledge transfer. Advances in Neural Information Processing Systems, 34:24468 24481, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 10699, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual in- telligence: A review and new outlooks. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. One teacher is enough? pre-trained language model distillation from multiple teachers. arXiv preprint arXiv:2106.01022 2021.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowl- edge distillation for long-tailed classification. In European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint arXiv:1901.06706, 2019.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. of Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
13

---

Under review as a conference paper at ICLR 2023
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via adaptive instance normalization. arXiv preprint arXiv:2003.04289, 2020.
Keren Ye and Adriana Kovashka. case study the shorteut effects visual commonsense rea- soning, In Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE CVF Conference on Computer Vision and Puttern Recognition, pp. 55795557555555 5588, 2021.
Zhengkun Zhang, Xiaojun Meng, Yasheng Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. Unims: A unified framework for multimodal summarization with knowledge distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 11757-11766 1174, 2022.
Mingyang Zhou, Licheng Yu, Amanpreet Singh, Mengjiao Wang. Zhou Yu, and Ning Zhang. Unsu- pervised Dision-and-languaee pre-training via retrieval-based multi-granular alignment. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16485- 16494, 2022.
A APPENDIX
A.1 TRAINING/MPPLEMENTTTII DETAILS
A.1.1 ADAPTIVE TEXT TOKEN SELECTION
With experiments. we find that when the number of selected tokens is (m = 2), Adaptive Text Token Selection would produce the highest performance.
A.1.2 ADAPTIVE TWO-STEP FINETUNING
T is hyperparameter temperature and set to in all experiments.
A.1.3 HEURISTIC RULES FOR CONVERSION
When conducting Adaptive Finetuning with Contrastive Learning on VCR, we convert every pair of question and correct answer choice into statement with heuristic rules in Fig. 4 When conduct Finetuning with Contrastive Learning VQA, we directly concatenate the question and the correct answer together as the statement. When we conduct Finetuning with Contrastive Learning on SNLI-VE, we only keep the images with entailed hypothesises positive pairs. The hypothesises are already in statement format so we do not need to further convert them.
A.2 EXPERIMENT SET-UP
All the baseline experiment results with models including VL-BERT, UNITER, VILLA and CLIP- ViL, are based code provided by the authors, which modified to include our distillation methods. CLIP-ViL, did not originally evaluate on VCR in their pre-print. However, we evaluated it the VCR dataset.
14

---

Under review as a conference paper at ICLR 2023
Question "what: 15 going to happen next?"
ansver "Pe-sonz gaing to tay how cute the childr are."
question.spitt"
anover answer-
if "what" question [8]:
11 'vItt" = question[1] and 'hoppan' questior(21 (2] and 'next" question|83 and "P" =s question(4]s
in
statement questionnquesston.nnee Index; 111) answer
teap Late, Laiee_tpe type "1b"
6llf 'after' in question:
qusstionlquestiion
"ofter")
teaplate, type
question[1]]
cunstion|:]
question
questior [1] or question or 'did' quest (131
or "does" question(1) or 'vere' -- ques iintt]::
statement answer
tesplate. "Id"
queskiin(1]:
statement
template, type "Ie"
elid "will" m question[]] [ 'night' questior [1]:
'it'
questionlal:
in questin ion:
statement unswer question leues lon, index(" is") =1111
temp type_type a '2a"
el1f 'are"
ion. [[=1]
toxp conplate_type
61.7 Infty" question(@):
11 'cons' =n question [1] or 'de' sz question [1] "ane" on question(s) or 'ss' question [1] or "did' question(1] or 'sould' = qusstion
"vas' (>-question or thas cuestion(3]:
[question(21) [question(al] cuestion(Br-1) [ ('becouss")
type
alif "how' quest qunstion(a):
11 'if' In question:
statement suestsoolqucciioninnee. 11dex)"1"'1=11 answer
Late,type '4a"
question):
statement
tesplate, type "db"
niif "do" mn qurstion[1] or 'will' en quest inn[1] or 'is" m qusstimn:]]
statement answer
texplate_type
question[a]:
statemmnt anser
Somplate__ type '5a'
el1r ('1s" questioni91) or caaa" as questiont0)) I'does" question(Bl: or I'se" questionte1) or UNas" questioniel)
(8 'di't question [8] "wil"" question 'soesn' question "would" poulddlmmpee (80
statement Iquestion [questiontel] susstion[2:-11 ansver(1:]
else:
statwaent [question [:]] [qpastinn(s]] ["not*] question [P:=1] amsoen [5:]
semplate__ 'b2'
statement anserr
type '/2"
elir "whsch" = questionlel:
Figure 4: Heuristic rules for converting question and answer pairs into statements. The input ques- tion and answer by default are in format of string.
A.2.1 TEACHER MODEL
For VIT, we utilize ViT La2 version pre-trained on ImageNet22k:
For Swin Transformer V2, we utilize the SwinV2- version pre-trained on ImageNer22k.
For RoBERTa, we utilize the pre-trained RoBERTa J. without further finetuning.
For CLIP, we utilize the ViT-By Byg version pre-trained with 400 Million paired image-text data.
A.2.2 STUDENT MODEL
VL-BERT: We train our model for 30 epochs with warm-up steps 1000, SGD optimizer. Initial learning rate is 7.0e and decays by 0.1 at the 14th, 18th and 26th epoch. The gradient
15

---

Under review as a conference paper at ICLR 2023
accumulation steps is set to be 4 8 NVIDIA V100 GPUs (32GB VRAM). The total number of attention layers for LarB is 24.
UNITER: Started with warm up steps of 800, the model is trained with total steps of 8000. With AdamW optimizer, the intial learning rate set to be 6e 05 with weight decay of 0.01 and batch size of 4000. The gradient accumulation steps is set to be 5 4 NVIDIA TTTAN RTX GPUs (24GB VRAM). The total number of attention layers for UNITER is 24.
VILLA: Warm up steps is set to be 1000 and total training steps is 10000. The intial learning rate is 6e 05 with weight decay of 0.01 and AdamW optimizer. The training batch size is 1250. The gradient accumulation steps is to be 8 on 8 NVIDIA TITAN RTX GPUs (24GB VRAM). The total number of attention layers for VILLA 24.
A.2.3 COMPARISON MODEL
CLIP-ViLp: The model is trained for 20 epochs with batch size of 24. The optimizer is Adam with a peak learning rate of x 10-5.
A.3
EVALUATION SETTING
A.3.1 VISUAL COMMONSENSE REASONING (VCR)
The VCR benchmark presents images along with paired question, a set of candidate answers, and a set candidate rationales (Zellers et al., 2019). The dataset includes 290k questions, in reference to 110k unique visual scenes. The questions constituted into categories based on patterns in the questions. Please see the supplementary material for a full list.
VCR Dataset Question Sub-Types According to Zellers et al. (201)) among VCR questions, 38% fall into explanation (why, how come, etc.), 24% activity (doing, looking, event, etc.), 13% temporal (happened, before, after, etc.), 8% mental (feeling, thinking, etc.), 7% role (relations, occupations, etc.), 5% scene (where, near, etc.). and 5% hypothetical (if, would, could, etc.). Details can be referred to Zellers et al. (2019).
Low-Shot: the low-shot setting, we have 2 training set partitions of varying sizes. Since VCR has types of questions thus (1) select 100 samples per question category, (1000/Type), to- talling 700 pairs, or 0.3% of the entire dataset, and (2) 1,000 samples per category, (1000/Type), totalling 7, 7,000 pairs, 3%, Each experiment is run 4 times.
We refer to existing domain-shifted evaluation configuration Ye & Kovashkal (022 that focuses on changing pronouns between question and answers.
Domain-shifted:
In questions and answer choices of VCR, person can be referred by person tag [[person2] [2]) or a pronoun (he, she). The method applies simple rule-based modification on the person tags and pronouns in VCR answer choices.
For the correct answer choice, the rule is to remove the person tag and swap it with a pronoun, instead.
For the incorrect answer choice, the rule is to make the option more associated to the question by repeating the person that is asked, as in Fig.
Some examples are shown Fig. and Fig.
A.3.2 VISUAL ENTAILMENT (SNLI-VE)
The Stanford Natural Language Inference Visual Entailment (SNLI-VE) task Xie et al. (2019) presents images as a premise, with a paired hypothesis text. The goal is predict whether the image entails or contradicts the hypothesis, or whether neither is the case (neutral).
Low-shot: There are only 3 classes of answer labels corresponding to types of relationship be- tween the image premise and the text hypothesis (entailment, neutral, and contradiction). Different
16

---

Under review as a conference paper at ICLR 2023
[val-54) 54) [2) going
original
rule-singular
[2] is going into the store..
He is going into the store
[2] is geiting into carrisge..
[2] is getting into carriage.
going bathroom.
bathroom. (1] is going outside to play after
[1] Is go outsidee to play after
the conversation with [2] is over
the conversation with is over
Figure 5: Rule-based modification for one person.
[val-270] What are [1,2] feeling? ?
original
rule-plural
[1,2] do not like the restaurant.
[1,2] do not like the restaurant.
They are spprehensive.
[1,2] are apprehensive.
They are both feellng happy
They are both feeling happy
[1,2] are fooling drunk.
[1,2] are foeling drunk.
Figure Rule-based modification for multiple people.
Examples- Modifying the distracting options.
Question
Original
Changed
He used the wrong ingredients to make the meal. [person]] | is very excited.
[person2] used the wrong ingredients
Why is [person2] such a rush?
to make the meal. [person2] very excited.
How is ]person]] feeling?
Figure 7: Examples of modifying the incorrect answer choices.
from VCR, image can pair with around 5 text premises in SNLI-VE. Therefore, choose the image-based selection. We also have two settings: (I) We select 100 random images for each class, 300 images in total paired with around 1,500 text premises. (2) 1,000 randomly sampled for each class label, then 30, 00 samples in total with around 15,000 premises. They both correspond to 0.3% and 3% of SNLI-VE. Each experiment is run 4 times.
Domain-shifted: We follow similar procedures as in Ye & Kovashka (2021) to only replace pro- nouns based on list of heuristic rules create Nomain--siffed settings, in Tab.
17

---

Under review as a conference paper at ICLR 2023
Original Words
Replaced Words
Man
Male
Boy
Woman
Female
Girl
People
Persons
Table 5: Heuristic rules of creating domain- shifted setting on SNLI-VE.
Types
Original Words
Replaced Words
Man
Woman
Pronouns
Person
He
She
How many XXx?
What is the number of xxx?
What is xxx doing?
What is the action of xxx?
Question Types
Who xxx?
Which person xxx?
What color xXx?
Which color xxx?
Table 6: Heuristic rules of creating domain-shifted setting on VQA.
A.3.3 VISUAL QUESTION ANSWERING (VQA)
Different from VCR and VE, for every image-question pair in VQA Antol et (2015), question- specific multiple choices are not provided. Instead, the global set of all possible answer choices for all the questions are provided (more than 3,000). The challenge is then select the correct answer choice from this set for the given image-question pair.
Low-shot: There is not clear categorization of VQA questions. Based on our analysis of the first n-gram words of questions, we group and finalize 8 types in total. In VQA, an image is also paired with several questions, 5.4 on average. Thus, we also rely on image-based sampling. We have two settings of image sampling in low-shot settings: 100 random images per question type and 1,000 random image per question type. After collecting up 2 paired questions per each selected image, we have two low-shot set: a set with around 1,600 questions and another set with 16,000 questions. They both correspond to 0.3% and 3% of VQA. Each experiment is run 4 times.
Domain-shifted Setting: We follow similar procedures as in Ye & Kovashka (2021) to replace pronouns and initial words of questions based on list of heuristic rules to create dooainn-siffi settings, in Tab. 6
A.4 EVALUATION RESULT
A.4.1 ABLATION EVALUATION
Different Teacher Models
Distillation with Different Layers We conduct ablation study with ADVL on top of VILLA as
the student model. The ablation study only evaluated VCR. As in Tab. 9 we can observe that: (1) when the pre-trained image and text encoders are from a VI. pre-training, the optimized target attention layer should be set the last hidden layer of the VL student model. (2) when the pre-trained image and text encoders are from unimodal pre-training, the optimized target attention layer should be set at intermediate layer of the VL student model. Typically, this would be one half of the VL student model's 's total number of attention layers.
A.4.2 LOW-SHOT EVALUATION
Refer Tab. 7 Tab.
Tab. 9 Tab. and 10
18

---

Under review as a conference paper at ICLR 2023
Type
1000/
Type
Vision
Lang.
Conf. W.
T.T. Select.
T.S. Fine.
Accuracy
Var
Accuracy
Var
30.85
1.12
53.48
0.74
34.24
0.93
54.53
0.54
/
35.96
0.59
55.04
0.70
36.78
0.91
55.91
0.82
/
/
37.21
1.40
57.64
1.03
38.28
1.04
58.68
0.68
40.43
1.36
58.98
1.09
Table 7: Ablation experiments using VL-BERT as the base student model and evaluated on the VCR standard validation set. The visual teacher is pre-trained SwimTTransformer V2 and the lan- guage teacher is a pre-trained RoBERTa. Vision represents vision distillation and Lang. represents language distillation. Conf. W. represents Distillation with Confidence Weighting. T.T. Select. rep- resents Distillation with Text Token Selection. Tin.Fine. represents Adaptive Two-step Finetuning.
Teacher Model
VCR
SNLL-VE
VQA
100TType
1000/Ty ype
100/Type
1000/Type
100/ Type
10007 Type
Vision
Lang.
Accuracy
Var
Accuracy
Var
Accuracy
Var
Accuracy
Var
Accuracy
Var
Accuracy
Var
34,84
0.83
57.01
0.69
58.47
1.32
67.16
1.51
37.18
0.90
65.75
1.23
SLIP-V
SLIP-T
38.92
0.54
57.38
0.38
58.67
0.95
67.23
1.19
39.07
0.85
65.89
0.84
CLIP-V
CLIP-T
42.04
0.88
59.42
1.06
59.04
1.47
68.37
1.07
39.34
0.75
66.10
1.41
CLIP-V
RoBERTa
40.71
1,26
59.36
1.05
58.77
1.85
68.69
1.44
39.12
1.13
66.34
0.77
VIT
RoBERTS
39.05
1.01
58.14
0.92
58.44
0.86
67.56
0.75
39.01
0.52
66.17
0.54
SwinV2
RoBERTa
41.87
0.93
60.18
0.78
59.07
0.70
68.83
39.67
0.99
66.96
1.06
Table 8: Knowledge distillation from combinations of paired encoders for VI. downstream tasks. Vision represents vision teacher model and Lang. represents language teacher model.
A.4.3 ENSEMBLE RESULTS
VCR: As our approach is model agnostic, further apply ADVL on top of multiple ensembled VL base models (i.e., VL-BERT and VILLA) and achieve further significant gains in performance (Q2A: 80.93%, QA2R: 84.01 %) in the fully-sampled validation set.
VQA: Further experiments with applying ADVL on top of an ensemble of VL models, i.e.,
VILLA and VL-BERT can even achieve 74.29 % local validation, outperforming CLIP-ViLp.
A.S ANALYSIS
Based on our results, we have seen the impact end task performance of distilling knowledge from largely pre-trained unimodal encoders into student VL models. However, a key question still remains: how much the improvement comes from the distillation of each modality respectively?
Following Cao et al. (2020), we measure the Modality Importance (MI) of both visual modality and textual modality. This approach sums the attention weights across heads of each modality to understand how much each modality is weighted by the model. Fig. g shows the average the MI values of all the heads for each layer VL-BERT, both with and without MD, trained the VCR dataset. One can clearly observe that prior to distillation, the model more heavily weights the text modality as being important correctly choosing answers. After distillation, both vision and text modalities are more equally considered. This may also explain why the model yields such impressive performance improvements low-shot and domain shifted scenarios.
Fig. g. we plot the MI values of all the heads across 12 layers in VL-BERT Base and VL-BERT Base with MD. It obvious that, at the last layer, the textual MI heatmap on the right is denser than the visual MI heatmap on the left. This shows common flaw from existing V+L models that they heavily rely on the textual information than the visual part indicating the shallow understanding of the visual scene in downstream tasks. However, in the bottom row, the difference between the left and right heatmaps much smaller and the visual MI heatmap the bottom is also clearly more denser than the one at the top.
19

---

Under review as a conference paper at ICLR 2023
Teacher Model Lang.
VCR 1000/Type
Target Attention Layer / Total of Attention Layers
Vision
100/Type
Full
34.84
57.01
78.27
24th
42.04
39.42
78.61
20th
40.06
59.73
78.03
CLIP-V
CLIP-T
16th 24
38.53
58.60
78.46
12th 24
37.70
58.96
78.00
24th 24
26.74
38.99
78.03
20th 24
28.81
45.39
78.28
SwinV2
RoBERTa
16th / 24
35.93
53.10
78.67
12th 24
41.87
60.18
79.13
Table Knowledge distillation different target layers of VL models for VCR. Vision represents vision teacher model and Lang. represents language teacher model.
SNLI-VE
VOA
Model
100/Type
1000/Type
1000/Type
1000/Type
Accuracy
Var
Accuracy
Var
Accuracy
Var
Accuracy
Var
VL-BERT
53.28
1.04
62.31
1.28
35.33
0.75
63.29
0.69
VL-BERT
56.78
1.48
65.37
0.51
37.12
0.85
65.71
0.49
UNITER
58.36
0.87
66.23
0.85
36.46
0.24
64.43
0.48
UNITER*
59.42
1.14
68.34
1.32
39.75
0.51
65.93
1.07
VILLA
58.47
1.32
67.16
1.51
37.18
0.90
65.75
1.23
VILLA*
59.96
0.70
68.83
0.89
39.67
0.99
66.96
1.06
CLIP-ViL,
59.48
0.89
68.32
0.63
39.01
0.13
66.84
0.81
Table Low-shot evaluation on SNLI-VE and VQA. model with represents we apply ADVL on top of it. Baseline results are based our re-implementation of the VL models. Few-shot results are averaged over runs. 1000/Type and 1000/Type represent two low-shot settings of 100 and 1000 samples per question type.
Laper Nurtor
W--ERT
LoWe NUMEE
Tod M1 Vousl MI Vieaa od Datilation
Number
Head Mumber
Loye HUMDOT
Layer Number
44
(4)
Figure 8: (a) Heatmap of MI values for both vision and text modality. Each cell represents combi- nation between 12 heads and 12 layers VL-BERT model. The top row corresponds to the orig- inal VL-BERT model and the bottom row corresponds VL-BERT ø with ADVL. Index starts with 0. (b) Average Modality Importance (MI) values for each layer of VL-BERT, with ADVL (green) and baseline (blue). Shaded areas represent differences between vision and text modalities.
20

---

Under review as a conference paper at ICLR 2023
Q2A
Model
100/Type
1000/Type
Accuracy
Var
Accuracy
Var
VL-BERT
30.85
1.12
53.48
0.74
VL-BERT
40.43 (+9.58)
1.36
58.98 (+5.5)
1.09
UNITER
31.43
1.32
54.24
0.49
UNITER*
42.23 (+10.8)
0.76
59.88
(+5.64)
0.56
VILLA
34.84
0.83
57.01
0.69
VILLA*
41.87 (+7.03)
0.93
60.18 (+3.17)
0.78
CLIP-ViL P
34.63
1.06
53.54
0.80
Table 11: Low-shot evaluation on VCR. model with * represents we apply ADVL on top of it. Baseline results are based on our re-implementation of the VL models. Few-shot results are averaged over runs. 1000/Type and 1000/Type represent two low-shot settings of 100 and 1000 samples per question type.
21


Reviews:
[{"review_id": "yZ5xwc-8sy", "reviewer": "Reviewer_LFJd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to utilize unimodal pretrained models to improve vision-language downstream tasks through distillation. In particular, the paper proposes to use both visual and textual teachers to supervise a student VL model during the finetuning stage. A group of methods/tricks have been explored, including adaptive confidence reweighting, adaptive token selections, and two-stage finetuning. The resultant framework improves the downstream VL performance on the student model.", "summary_of_the_review": "In general, this paper presents a seemingly working framework to improve VL tasks with distillations from both vision and language pretrained checkpoints. However, the method is quite complicated and appears to be a bag of tricks. On the other hand, there is no clear and comprehensive understanding of the source of improvement. Therefore, I wish to see more studies to understand the trade-off between the additional training costs and true benefits the method brought.", "strength_and_weaknesses": "Pros:\n1. Experiments are conducted on many pretrained variants and datasets, and clearly show gains over VL baselines. This shows that the proposed method is effective.\n2. Ablation studies are performed and it is clear that each proposed component is beneficial.\n3. The paper is clear and easy to follow.\n\nCons:\n1. The resulting student model is a large-scale VL pretrained checkpoint itself. It will be more interesting if the method can get strong performance on non-pretrained or smaller-scale models. As of now, it is hard to say the method brings 'efficiency' given everything is pretrained anyway.\n2. The method contains too many specialized components. Although ablation does show the benefit of each, this still makes the method hard to use in practice. In fact, I am a bit hesitant to call it a 'method' and feel it is more like a bag of 'methods'.\n3. Although some analysis is given, it is still hard to understand why the method is better. Since it is not intuitive why such framework can/should improve visual attention, it seems a must to dig further to understand details such as which exact step brings this benefit and why.\n4. The proposed method is also designed for fusion-based VL models and not directly applicable for other trending architectures such as encoder-decoder.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tz6Y2xUCueC", "reviewer": "Reviewer_JhCz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes methods to distill unimodal teacher models of vision and language onto vision-language (VL) student models for utilizing the benefit of unimodal pre-trained models. \nThe authors present ideas to improve the performance: (1) VL student models are not changed, but additional learnable components are attached on the top of them, (2) instance-level weighting and (3) text token selection to consider for distillation. \nThe authors claim that the methods have good advantages to improve the performance and to avoid pre-training large models, and achieve new state-of-the-art performance.\nThe two former claims are validated with some experimental results on 3 kinds of vision-linguistic tasks (Vision Commonsense Reasoning, Visual QA, SNLI-SE) including ablation study and comparative analysis. \nHowever, the latter one is NOT justified since the supporting evidences in the web-based leaderboards have some problems.", "summary_of_the_review": "I think that the weaknesses on the significance of this work is so critical, and it is difficult to accept the authors' claims.", "strength_and_weaknesses": "*Strength\n- The proposed ideas are good to achieve the research goal. I think the proposed methods would be practically useful if they are generally applicable and the benefit is clear in most cases.\n- The paper presents interesting findings on the performance improvements in the low-shot area.\n- Various settings of experiments are performed and the results are consistent as the authors’ claims.\n\n\n*Weaknesses\n- I'm concerned with that the significance of this work would be limited. This method is not for building compact student models (in normal scenarios of knowledge distillation), but for improving the performance based on VL student models, unimodal teacher models for vision, and unimodal teacher models for language. The generality of the performance improvement is not clearly validated yet, and the low-shot setting, the area that the benefit is significant, needs the criteria of how low is appropriate to apply this method.\n- One of the main claims is that the proposed methods achieve new SOTA on the VCR tasks. However, there are some problems on that paragraph. I checked the two leaderboards (1: https://leaderboard.allenai.org/vcr/submissions/public, 2: https://visualcommonsense.com/leaderboard/ ). Following the AI2 leaderboard, I realized that the numbers from the page 8 of the manuscript, “Q2A: 80.9%, QA2R:83.3%, Q2AR: 67.7%”, is the 1st one’s and it is for MERLOT, not for ADVL. The ADVL’s is “80.4%, 82.3%, 66.2%”. Following the VCR leaderboard, ADVL is the 18th runner and the score is “79.6, 82.9, 66.2”. Considering the submission dates of the others and the ranking, I cannot accept the claim currently. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3bp81zn4IQ", "reviewer": "Reviewer_chYB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is concerned with vision-language representation learning. Main contribution is a new distillation technique that uses stronger uni-modal encoders as teacher network, to improve the finetuning performance of pre-trained VL models. \n\nIn specific, the work proposes Adaptive Distillation for Vision-Language (ADVL) tasks. ADVL leverages unimodal pre-trained models (e.g. ViT for vision, Roberta for text). The distillation is achieved by minimizing the L1 distance between features from the uni-modal teacher network, and the ones from the pre-trained VL student models. Authors also propose additional mechanisms, such as (i) adaptive confidence-based weighting, which adjust distillation weights based on student networks' confidence scores; (ii) adaptive text token selection, which enables the distillation process to focus on semantically more important text tokens; (iii) finetuning with contrastive learning to enhance cross-modal alignment.\n\nExperiments are conducted using UNITER, VL-BERT, VILLA as student networks, and various unimodal encoders as teacher networks, including SLIP, CLIP ViT, Swin transformer. The proposed technique improves student networks on visual commonsense reasoning (VCR), visual entailment (SNLI-VE), and visual question answering (VQA) significantly in low-shot setups, and marginally in full-shot setups.", "summary_of_the_review": "The proposed technique provides marginal empirical benefit while the method itself is technically incremental. Experiments are flawed, especially the choice of teacher networks falsifies the motivation of relying on uni-modal data only. It is also not clear whether the proposed technique will push the limits of more recent VL pre-trained models, especially those trained end-to-end. Therefore, I'd not recommend a rejection.", "strength_and_weaknesses": "**Strengths**:\n1. The motivation is clear. It is naturally expected that vision-language models can benefit from stronger unimodal encoders, for which the best strategies are not very well explored by the community.\n2. The idea is simple and easy to implement. According to the evaluation, the proposed method shows its effectiveness especially when the finetuning data is limited.\n3. Although the presentation can be polished, the overall narrative and explanation is clear and easy to follow.\n\n**Weakness**\n1. **Flawed experiment setup**. The motivation is to leverage unimodal data, which are assumed easier to obtain than image-text pairs. However, the teacher networks used in the experiments are not always uni-modal. For example, CLIP, SLIP are trained on a large amount of image-text pairs, meaning that their unimodal encoders are also aligned across modals. This contradicts with the motivation.\n2. **Limited technical novelty and incremental empirical gains**. The proposed distillation loss is standard and by itself is not technically new. In addition, improvements on full-shot cases are mostly marginal. Considering this is the combined benefit of multiple techniques, e.g. distillation, text token selection, contrastive learning, I am not fully convinced by the empirical value of the proposed method.\n3. **Student networks** are too weak to prove the proposed techniques are useful for the more recent (and more powerful) models. The selected student networks, VL-BERT, UNITER, VILLA, while they are great and highly reputable works in the community, their performance is not as competitive as for today. Therefore, the obtained task performance is far from state-of-the-art. For example, on VQA, more recent works (BEiT-3) achieve 84+, while the best reported result in the manuscript is ~76. To make the method more convincing, authors may consider use more recent VL models as student works, and try to further push their limits. For example, to use LLM such as GPT to improve the text representation of BEiT.\n5. **More recent VL-pretrained works are not well acknowledged**. Although I understand the experiment setup, missing reference to more recent VL works prevent readers from getting a good research landscape in the multimodal pre-training. For example, recent great VL works, ALIGN, ALBEF, OFA, Frozen, Flamingo, Florence. BLIP, BEiT3. Authors are suggested to better position their work among the more recent ones.\n4. It is not fully clarified what is the difference between the so-called \"ITM\" (image-text matching) and the contrastive losses used in other  VL pretrained models, such as ALIGN, ALBEF. Conventionally the ITM loss is a binary prediction task, while the particular one used in this work is more often referred as contrastive learning loss. This may create unnecessary confusions.\n6. Presentation can be improved. Typos are not uncommon. Experiment setups are not entirely clear. For example, what is the experiment environment and training receipts.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject