ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval
Year: 2023

Abstract:
Large-scale pre-trained text-image models with dual-encoder architectures (such as CLIP (Radford et al., 2021)) are typically adopted for various vision-language applications, including text-image retrieval. However, these models are still less practical on edge devices or for real-time situations, due to the substantial indexing and inference time and the large consumption of computational resources. Although knowledge distillation techniques have been widely utilized for uni-modal model compression, how to expand them to the situation when the numbers of modalities and teachers/students are doubled has been rarely studied. In this paper, we conduct comprehensive experiments on this topic and propose the fully-Connected knowledge interaction graph (Cona) technique for cross-modal pre-training distillation. Based on our findings, the resulting ConaCLIP achieves SOTA performances on the widely-used Flickr30K and MSCOCO benchmarks under the lightweight setting. An industry application of our method on an ecommercial platform further demonstrates the significant effectiveness of ConaCLIP.1

heading: 1 Introduction
text: Text-image retrieval (TIR) aims at retrieving a list of the most relevant images from a large image collection when a specific text query is given. With the rapid development of information interaction and social intercourse, it has been regarded as a crucial component of cross-modal applications and required by various real-world scenarios, such as e-commercial platforms (sites).
Recently, inspired by the great success of pretrained language models (Devlin et al., 2019; Liu
∗Contribution during internship at Alibaba Group. †Co-corresponding authors.
1Related resources will be publicly available in the EasyNLP framework (Wang et al., 2022a). URL: https: //github.com/alibaba/EasyNLP.
et al., 2019; Brown et al., 2020), research on largescale vision-language pre-training (Tan and Bansal, 2019; Li et al., 2020; Radford et al., 2021; Li et al., 2022; Wang et al., 2022b, 2023) has achieved remarkable progress on a variety of vision-language tasks, including text-image retrieval. These existing methods can be typically classified into two categories according to the model architecture: crossencoder and dual-encoder. Cross-encoder typically adds additional Transformer (Vaswani et al., 2017) layers to model the deep interaction between image and text representations. It can generally boost the retrieval performance, while resulting in an unbearably slow retrieval speed when applied to the entire image collection since the cross-modal costs are required for each image sample whenever a new text query is given. In contrast, dual-encoder encodes the visual and textual inputs in a wholly decoupled manner. The image representation is allowed to be pre-computed and re-used independent of the text queries. Such approaches can also utilize fast approximate nearest neighbor (ANN) search (Muja and Lowe, 2009; Jegou et al., 2010; Johnson et al., 2019) at runtime.
Although dual-encoder is usually preferred for real-world applications, the existing related models such as CLIP (Radford et al., 2021) are still less practical on edge devices with limited computing resources, or for the dynamic indexing scenario, e.g., private photos/messages collections (sites). To address this issue, we aim to start from the largescale pre-trained dual-encoder models and focus on the pre-training distillation to present a series of much smaller, faster, and effective counterparts. Knowledge distillation (KD) (Hinton et al., 2014) is proposed to transfer knowledge with soft targets from a teacher to a student in the same modality. MoTIS (Ren and Zhu, 2022) simply repeats intramodal InfoNCE-based (Oord et al., 2018) learning in both text and image domains for distillation. Nevertheless, when the number of modalities dou-
71
bles for dual-encoder structure, which means text and image teachers as well as text and image students, these methods still only involve intra-modal teacher-student knowledge interaction learning. Instead, in this paper, we comprehensively explore the fully-Connected knowledge interaction graph (Cona) between every possible teacher-student or student-student pair. As shown in Fig. 1, each two-way arrow represents the knowledge interaction learning between the two models it points to. And the aforementioned KD and MoTIS belong to a single blue arrow and the two blue arrows, respectively. Moreover, in order to better explore the potential of Cona, we implement and investigate various supervision strategies to guide the model optimization, which finally makes each type of learning contribute to the overall improvement.
We release various sizes of lightweight dualencoder models named ConaCLIP for different real-world scenarios. Compared with the previous SOTA method (Ren and Zhu, 2022), our ConaCLIP achieves 10.6/12.9/12.8 R@1 gains on Flickr30K/MSCOCO (1K)/MSCOCO (5K) benchmarks under the same model setting. We have also verified its effectiveness on an e-commerce platform. It can achieve 1.44×/1.92∼4.86× inference speed-up with competitive performances given image/text queries. The main contributions of this paper can be summarized as follows:
• We propose a new pre-training distillation method with the fully-connected knowledge interaction graph (Cona) for lightweight dualencoder models.
• We release a series of lightweight ConaCLIP models to the open-source community, which can significantly surpass previous SOTA models on the widely-used Flickr30K and MSCOCO benchmarks.
• We provide a real-world application of this method in real industrial scenarios to further demonstrate its practical values.

heading: 2 Related Work
text: Cross-encoder (Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020; Li et al., 2020; Chen et al., 2022) refers to multiple layers of dense crossmodal interactions, e.g., cross-attention (Vaswani et al., 2017), are typically employed to image and text representations for more fine-grained merge
and alignment. Although it often achieves superior retrieval accuracy thanks to the patch/token-level integration, the high memory cost and computation inefficiency make it impractical under time-critical real-world settings.
Oppositely, for dual-encoder (Zhang et al., 2020; Jia et al., 2021; Radford et al., 2021; Dou et al., 2022), image and text features are encoded into a joint embedding space separately, and the modality interaction is only handled by a simple cosine similarity of the final image and text feature vectors. Such approaches can be regarded as scalable and indexable: the specific choices of encoder architectures can be independent and dynamic, and the late-interaction scheme allows for efficient largescale searching.
Pre-training distillation for lightweight dualencoder architecture has been rarely studied. Vanilla knowledge distillation (Hinton et al., 2014) can be referred to as the knowledge transfer from a teacher to a student in the same modality based on soft targets. However, it is a general procedure without awareness and pertinence for crossmodal learning. MoTIS (Ren and Zhu, 2022) separately compresses text or image encoder with an intra-modal contrastive objective that aligns the output embeddings of the student and teacher of each modality, which can be seen as an alternative form of knowledge distillation. Nevertheless, these methods ignore or do not find an appropriate approach to leverage the cross-modal distillation process. Further than them, our method is dedicated to exploring the fully-connected knowledge interaction graph for dual-encoder distillation, which is a natural and effective extension.

heading: 3 Methodology
text: In this section, we first give the preliminary knowledge, then propose our pre-training distillation framework with Cona. Finally, we introduce various supervision strategies.

heading: 3.1 Preliminary
text: For the sake of explanation, we abbreviate text, image, teacher and student as T , I , tch and stu respectively. F represents the L2-normalized feature vector outputted by the encoder architecture E.
Before student learning, the teachers ETtch and EItch are commonly first pre-trained using an objective that pushes the embeddings of matched text-image pairs closer while pushing those of non-
matched ones apart, with large model capacity and massive data. Specifically, CLIP (Radford et al., 2021) takes the InfoNCE (Oord et al., 2018) loss as the supervision form. Without losing generality, given two outputted feature vectors F a and F b ∈ RN×d, we define that:
pi,j(F a, F b) =
exp(F ai F b j ⊤ /τ)
∑ k exp(F a i F b k ⊤ /τ) , (1)
LInfoNCEFa→F b = − 1
N
N∑
i=1
log(pi,i(F a, F b)), (2)
where N is the mini-batch size, d is the channel size and τ is the temperature hyper-parameter. The final loss of CLIP can be formulated as:
LCLIP = LInfoNCE FTtch→F Itch + LInfoNCE F Itch→FTtch . (3)
Next, the pre-training distillation of students ETstu and E I stu begins, with parameters of teachers ETtch and E I tch frozen. MoTIS (Ren and Zhu, 2022) also adopts the InfoNCE-based loss at this stage, and implements it in both text and image domains separately:
LMoTIS = LInfoNCE FTstu→FTtch + LInfoNCE F Istu→F Itch . (4)
According to the subscript in Eq. (4), it is easy to see that MoTIS only involves intra-modal teacherstudent learning.

heading: 3.2 Pre-training Distillation with Cona
text: Unlike existing works, our method introduces the fully-connected knowledge interaction graph
(Cona) for pre-training distillation. Apart from intra-modal teacher-student learning, our method also includes intra-modal student-student learning, inter-modal teacher-student learning and intermodal student-student learning, as shown in Fig. 1. This fully-connected learning graph established for students ETstu and E I stu serves as an integration of multi-view and multi-task learning schemes, which can strengthen the robustness and effectiveness (Caruana, 1997; Luong et al., 2016; Aghajanyan et al., 2021) required by pre-trained models.
We suggest that each type of learning process in Cona should be concretely implemented in detailed supervision strategies. Therefore, we propose and investigate various supervision strategies in the next subsection.

heading: 3.3 Supervision Strategies
text: Here we continue to use F a and F b (prediction) along with F̃ a and F̃ b (target) as placeholders for illustration, and present the following effective supervision strategies: InfoNCE loss is a type of contrastive loss function. It has been formulated in Eq. (2), and successfully applied for distillation by Eq. (4). Feature-wise distance (FD) loss directly minimizes the distance between feature vectors. We utilize squared L2-norm as the measure:
LFDFa⇔F b = 1
2
1
Nd
N∑
i=1
d∑
j=1
(F ai,j − F bi,j)2. (5)
Similarity-wise distance (SD) loss minimizes the distance criterion between similarity matrices:
LSD Fa→F b⇔F̃a→F̃ b = 12 1 N2 N∑ i=1 N∑ j=1 (F ai F b j ⊤ − F̃ ai F̃ bj ⊤ )2.
(6)
Since F a, F b, F̃ a and F̃ b have been L2normalized, the values of cosine-similarities F ai F b j ⊤ and F̃ ai F̃ b j ⊤ are in the range [−1, 1]. The distance between prediction F ai F b j ⊤ and target
F̃ ai F̃ b j
⊤ needs to be shortened. Hence, the squared
L2-norm is also adopted here. KL-Div loss uses the Kullback–Leibler divergence to measure the difference between the predicted and the target probability distributions. Given pi,j acquired by softmax operation shown in Eq. (1), it
minimizes the following optimization objective:
LKL-Div Fa→F b∥F̃a→F̃ b = 1N N∑ i=1 N∑ j=1 pi,j(F a, F b)log pi,j(F a,F b) pi,j(F̃a,F̃ b) .
(7)
It is worth noting that, when performing the learning process indicated by an arrow shown in Fig. 1, the common practice is to use teachers’ outputs F Ttch and F I tch as target in Eq. (6)(7) that students learn from. While in our case with two modalities available, we propose to use the paired arrow as the target, and we call this the symmetric version (for SD loss and KL-Div loss). For example, inter-modal teacher-student learning implemented with KL-Div loss can be formulated as
LKL-Div FTstu→F Itch∥FTtch→F Itch + LKL-Div F Istu→FTtch∥F Itch→FTtch ,
(8)
while its symmetric version is
LKL-Div FTstu→F Itch∥F Istu→FTtch + LKL-Div F Istu→FTtch∥FTstu→F Itch .
(9)
This modification deepens the interaction between the four encoders during optimization.
So far, any one of the learning types can be concretely implemented by any one of the supervision strategies, except for a few meaningless combinations. Detailed loss functions are listed in Tab. 1.

heading: 4 Experiments

heading: 4.1 Setup
text: We use Conceptual Caption (CC3M) (Sharma et al., 2018) and Conceptual 12M (CC12M) (Changpinyo et al., 2021) for pre-training distillation, which consist of 3M and 12M noisy text-image pairs respectively. During fine-tuning, we use MSCOCO (Lin
et al., 2014) and Flickr30K (Plummer et al., 2015) as benchmarks. MSCOCO has 113,287 images for training, 5K images for validation, and both 5K and 1K for testing. Flickr30K has 28,783 images for training, 1K images for validation, and 1K for testing. Following previous works, we use recall R@k (k=1,5,10) as the main metric.
We use the open-source CLIP (Radford et al., 2021) with ViT-B/32 (Dosovitskiy et al., 2020) as the teacher model. Its image encoder is a 12-layer ViT with the hidden size to be 768 and 12 attention heads. Its text encoder is a 12-layer Transformer with hidden size to be 512 and 8 attention heads.
For the student model, we use ViT-S/16 with hidden size to be 384 as the image encoder, and initialize it from the pre-trained weights on ImageNet21K (Ridnik et al., 2021). For the text encoder, we experiment with 2, 4 and 6-layer Transformer, of which the weights are initialized from the first corresponding layers of the teacher’s text encoder. The details of model settings are shown in Tab. 6.
In pre-training distillation, we train the student models in 4 epochs using AdamW (Loshchilov and Hutter, 2018) with a batch size of 1024 for both images and texts, the learning rate of 3e-4, and the weight decay of 0.1. We employ a cosine learning rate scheduler with 10,000 warm-up steps. In finetuning, we use the same optimization setting as in MoTIS (Ren and Zhu, 2022). Experiments are conducted on 4 NVIDIA TESLA V100 32G GPUs.

heading: 4.2 Ablation Study
text: Considering our complete pre-training distillation takes a relatively long time, we follow the setup of (Ren and Zhu, 2022) and train ConaCLIP on CC3M for 1 epoch with batch size 84 to conduct the ablation study. Taking Eq. (4) as the naive baseline, we aim to find out which of the proposed
combinations of learning types and supervision strategies can bring further improvements. The fine-tuned results on Flickr30K is shown in Tab. 2.
We can make some observations that: 1) With an appropriate choice of detailed supervision strategies, each type of learning can further bring obvious improvements on the basis of the baseline. 2) The effect of each learning type is greatly affected by the implemented loss function. It also indicates that the pre-training distillation process should be carefully explored regarding the supervision strategy. 3) Our proposed symmetric version losses (Sym-SD and Sym-KL-Div) can generally achieve superior performances to the standard ones for (intra/inter-modal) teacher-student learning.
We can also attain several findings that: 1) For (intra/inter-modal) student-student learning where students first make knowledge interaction and then learn together from teachers, SD loss performs the best. Because the actual retrieval application uses this cosine similarity to rank candidates, it can help students acquire goal-oriented knowledge more directly. It also relaxes the learning task of students from teachers’ feature space to the similarity space. 2) For (intra/inter-modal) teacher-student learning, our proposed symmetric version losses are more
suitable. Compared with the standard losses, they make the knowledge interaction between teachers and students closer during optimization. In this regard, student encoders can cooperate more intimately in downstream tasks. 3) Although the naive intra-modal teacher-student learning with InfoNCE loss can serve as a competent baseline, the addition of SD and Sym-SD losses of the same learning type can complement its effectiveness. On the other hand, the other three different learning types with proper loss choices can also benefit the effect of pre-training distillation. More findings on distilling intermediate layers are shown in A.2.
Our method has been established with the further integration of the highlight (in bold) combinations in Tab. 2 based on the baseline. The effect after full integration is shown in Tab. 3(a).

heading: 4.3 Performance
text: Fair Comparisons. In order to better verify the effectiveness of ConaCLIP, besides the previous SOTA, we also experiment with two strong baseline methods. As shown in Tab. 3(a), InfoNCE-based indicates the naive cross-modal contrastive learning procedure. Cross-modal KD represents distilling the cross-modal in-batch probability distribution of teachers into students. All these experiments
are conducted under the pre-training setup of (Ren and Zhu, 2022) for fair comparisons. As can be observed, 1) Cross-modal KD which introduces the knowledge distillation process obviously outperforms the standard InfoNCE-based approach. 2) MoTIS greatly surpasses InfoNCE-based and Cross-modal KD. This reveals the superiority of intra-modal teacher-student learning over intermodal student-student learning in the case of dualencoder distillation. 3) Our ConaCLIP shows significant improvements compared with competitors on all evaluation metrics: 3.6/3.1/2.4 R@1/5/10 gains on Flickr30K, 5.9/4.2/2.2 R@1/5/10 gains on MSCOCO (1K) and 4.7/6.5/5.8 R@1/5/10 gains on MSCOCO (5K). This fully demonstrates the effectiveness of our distillation framework with Cona.
Model Zoo and Benchmarks. In order to better promote the development of cross-modal textimage research, we release a series of lightweight dual-encoder models. Their benchmark results are shown in Tab. 3(b). In this case, the power of ConaCLIP is further unlocked and brings further improvements. Specifically, even ConaCLIP-2L can achieve 8.6/7.1/5.1 R@1/5/10 gains on Flickr30K, 12.0/6.1/2.8 R@1/5/10 gains on MSCOCO (1K) and 11.5/12.6/10.0 R@1/5/10 gains on MSCOCO (5K) compared with the previous SOTA. We have also found that the capacity of the text encoder may have limited effects on these performances. For example, ConaCLIP-4L can achieve competitive results with ConaCLIP-6L, and ConaCLIP-2L has only minor drops.

heading: 5 Industry Application
text: We apply the proposed technique to end-to-end cross-modal retrieval in an e-commerce platform, where we vectorize the search queries and the products and then perform product retrieval and ranking with nearest-neighbor search (Muja and Lowe,
2009; Jegou et al., 2010; Johnson et al., 2019), as shown in Fig. 2. We first collect massive data of text-image pairs from e-commerce products in our platform, where the titles of products can act as text information. We utilize most of the data to pre-train an e-commerce version of the CLIP model (denoted as EC-CLIP) with ViT-B/32 as the image encoder, which is overly large for online deployment. For the remaining data, we utilize 3M pairs for distilling the lightweight EC-ConaCLIP. To evaluate its effectiveness, we hold out a separate set of 100K pairs for fine-tuning and 5K/5K pairs used in validating/testing. In this set of experiments, we train EC-ConaCLIP for 20 epochs in pretraining distillation, and fine-tune both EC-CLIP and EC-ConaCLIP for 5 epochs. The remaining settings are the same as in Section 4.1.
In apart to the R@k metric, we also report the disk space (MB) and the acceleration rate of Query Per Second (QPSi for image and QPSt for text) to evaluate model’s memory footprints and inference speed. In Tab. 4, we report the averaged results where the inference speed is tested on an NVIDIA TESLA V100 (16G) GPU. As seen, the compressed EC-ConaCLIP-6L only takes 44% disk space (254MB) of EC-CLIP meanwhile being 1.44×/1.92× faster with image/text queries. It also performs on par with EC-CLIP. Our ECConaCLIP-2L can further achieve up to 4.86× inference speed-up with text queries, and 64% size reduction (from 578MB to 206MB). We provide some case studies in A.4.

heading: 6 Conclusion
text: In this paper, we propose Cona for pre-training distillation with dual-encoder architecture. It gathers every type of knowledge interaction learning with appropriate supervision choice to benefit the cross-modal distillation. The resulting ConaCLIP
achieves superior performances on both general benchmarks and industry applications.
For future work, we will explore more variants of visual encoders, and continue to tap the potential of dual-encoder distillation.

heading: Acknowledgements
text: This research is supported in part by Alibaba Innovative Research Foundation (No. D8200510), NSFC (Grant No. 61936003), Zhuhai Industry Core and Key Technology Research Project (No. 2220004002350). This work is also supported by Alibaba Cloud Group, through Research Talent Program with South China University of Technology.

heading: A Appendix
text: A.1 Model Settings We give detailed parameters on the settings of our ConaCLIP models in Tab. 6, such as the number of parameters, layers, heads, etc.
A.2 Negative Results on Distilling Intermediate Layers
We also present an exploratory study on distilling the knowledge of intermediate layers from teacher encoders. We first evenly divide the encoder of each student/teacher into six parts along the number of layers, and then perform our distillation technique on the feature representations of each part. The experiment results are shown in Tab. 5.
We can observe that additional distillation with features of the intermediate layers does not bring about positive improvement. This inspires us that we should mainly focus on the representation matching ability of the output of the last layer for the cross-modal retrieval task. Due to the difference of capabilities between models of different sizes, they can choose different paths to learn the goal-oriented features in the same task during distillation (Li et al., 2021; Zhu and Wang, 2021; Xu
et al., 2022). In our application, we suggest that it can be inappropriate to force small models to learn the same path as the large ones.
A.3 Application in E-Commerce Product Retrieval
We apply the proposed distillation technique to end-to-end cross-modal retrieval in an e-commerce platform, where we vectorize the search queries and the products and then perform product retrieval and ranking with nearest-neighbor search. The whole framework is shown in Fig. 2.
A.4 Case Study
Tab. 7 shows the case studies in our e-commerce retrieval scenario. For the same text query, we
show the top-1 image retrieval results of the opensource CLIP model and our EC-ConaCLIP model respectively.
From these cases, we can find that our model can better capture conceptual and fine-grained fashion information during cross-modal text-image retrieval, and maintain the cross-modal alignment effect of text-image samples after the lightweight distillation. For example, in Case 1, our model more accurately captures the cartoon subject in the target commodity as "unicorn". In Case 2, our model pays more attention to fine-grained information "2 layers double handle", while maintaining the correct perception of other information such as "Stainless steel", "steamers pot" and "with lid". In Case 3, our EC-ConaCLIP better captures the color clue of "Hot red". Although the retrieval result of CLIP also conforms to the information of "sports tights high waist yoga pants", its color is more like "dark red".
Based on our distillation technique, the resulting model can sufficiently learn the perception ability of the teacher model about commodity fashion concepts and reduce matching errors.

heading: Future Work
text: For future work, we will explore more variants of visual encoders, and continue to tap the potential of dual-encoder distillation