On the Language Encoder of Contrastive Cross-modal Models (2024)

Abstract: Contrastive cross-modal models such as CLIP and CLAP aid various vision-language (VL) and audio-language (AL) tasks. However, there has been limited investigation of and improvement in their language encoder – the central component of encoding natural language descriptions of image/audio into vector representations. We extensively evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. In VL pretraining, we found that sentence embedding training enhances language encoder quality and aids in cross-modal tasks, improving contrastive VL models such as CyCLIP. Sentence embedding training also benefits AL tasks when the amount of training data is large. We analyze the representation spaces to understand the strengths of sentence embedding training, and find that it improves text-space uniformity, at the cost of decreased cross-modal alignment.

heading: 1 Introduction
text: Significant progresses have been made in pretraining large-scale cross-modal models, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), for various vision-language (VL) applications such as retrieval and zero-shot image classification. These models are often pretrained with large amounts of data, e.g., OpenAI leverages «400M caption-image pairs to train CLIP while LAION-AI scaled up this number to 5B (Schuhmann et al., 2022; Cherti et al., 2023). Such a large amount of multi-modal pretraining data contains text captions at the same scale as the pretraining corpora of large language models (LLMs) such as BERT, which is pretrained on 3.3B words (Devlin et al., 2019).
The success of VL pretraining encourages research on contrastive learning models for other modalities like audio. Pretrained audio-language
(AL) models such as AudioCLIP (Guzhov et al., 2022) and CLAP (Wu et al., 2023; Elizalde et al., 2023) show promising results on AL retrieval and zero-shot audio classification tasks.
It is clear that the language encoder in crossmodal contrastive models plays a central role when scaling-up pretraining of a specific modality and/or the amount of modalities. Therefore, analyzing and improving the language encoder become increasingly crucial. CLIP’s language encoder (CLIP LM) – a decoder-only language model similar to GPT-2 (Radford et al., 2019) – has been investigated. Yan et al. (2022) showed that the CLIP LM outperforms BERT (Devlin et al., 2019) in clustering entities with prompting. Wolfe and Caliskan (2022) probed the CLIP LM, showing its word representations are less anisotropic (Ethayarajh, 2019), i.e., more uniformly distributed with respect to direction, than GPT-2. Complementary to these research on CLIP LM, we focus on pretraining. CLIP-like models are often pretrained with crossmodal contrastive learning. We measure – during pretraining – the effectiveness of systematically modeling the captions with sentence embedding training (Reimers and Gurevych, 2019; Gao et al., 2021), which is a natural fit to the captions.
We pretrain CLIP and one of its new variants CyCLIP (Goel et al., 2022) with sentence embedding training, as well as the conventional cross-modal contrastive learning. In addition to CLIP’s crossmodal contrastive learning objective, CyCLIP explicitly optimizes for geometry consistency between the text and image representation spaces, making it a suitable model for validating the effectiveness of NLP methods. We evaluate pretrained models on an array of tasks involving one or two modalities such as SentEval, zero-shot VL retrieval, and image classification. We find that unsupervised sentence embedding training improves the language encoder quality and VL tasks. Supervised sentence embedding training improves language en-
4923
coder quality, but the benefit does not necessarily transfer to VL tasks. We analyze the learned representation spaces and find that sentence embedding training improves text-space uniformity (Wang and Isola, 2020) and reduces anisotropy (Ethayarajh, 2019; Wolfe and Caliskan, 2022).
We also investigate AL contrastive models such as CLAP (Wu et al., 2023). In contrast to VL pretraining, AL pretraining suffers from data scarcity and often leverages pretrained LLMs and audio encoders. We determine the effectiveness of sentence embedding training in both scenarios: continued pretraining with LLMs and audio encoders, and pretraining from scratch. We find that sentence embedding training improves AL tasks when the amount of data is large, while the benefits become less noticeable on small datasets. To the best of our knowledge, this is the first study on investigating and trying to improve the language encoder of AL contrastive learning, and we expect our results will encourage more research in this direction.
In summary, our contributions are as follows: (i) We extensively evaluate how unsupervised and supervised sentence embedding trainings affect VL and AL contrastive pretraining. Experimental results indicate improved VL performance. AL tasks see improvements when the amount of training data is large, while the benefits become less noticeable on small datasets. (ii) We show that unsupervised sentence embedding training improves the language encoder of CyCLIP (Goel et al., 2022), hence improves performance of cross-modal tasks. (iii) We conduct a comprehensive analysis on the alignment and uniformity of learned representation spaces following Wang and Isola (2020), and show that sentence embedding training improves uniformity of the text representation space, but at the cost of decreased cross-modal alignment.

heading: 2 Related work
text: CLIP LM. Research has focused on the language encoder of OpenAI CLIP (Radford et al., 2021). The model consists of a language encoder (CLIP LM) and an image encoder that are jointly trained on Web-scale caption-image pairs. Yan et al. (2022) stressed the importance of CLIP LM, showing that it outperforms BERT (Devlin et al., 2019) in tasks, such as entity clustering, through prompting. Bielawski et al. (2022) showed that CLIP LM outperforms BERT in “human-centric” tasks such as genre classification on books or movies. Santurkar
et al. (2023) highlighted the importance of text captions for representation learning of CLIP by comparing it with SimCLR (Chen et al., 2020) in which no language supervisions is present. Training signals from language are shown to be detrimental, worthing any number of images in a sufficiently large dataset. Our work follows this direction, with a focus on determining how supervised or unsupervised sentence embedding trainings affect CLIP LM and VL contrastive learning.
Goel et al. (2022) introduced CyCLIP, incorporating extra training objectives than cross-modal contrastive learning such that the geometry consistency between the text and image spaces is improved. One of CyCLIP’s training objectives is computing similarities between captions; this motivates us to determine how systematically modeling the captions through supervised or unsupervised sentence embedding training affects CyCLIP/CLIP.
Contrastive audio-language pretraining models have also been recently proposed. Guzhov et al. (2022) extended CLIP to audio tasks by adding an extra module and continued training on audio datasets. Similar distillation methods such as Wav2CLIP (Wu et al., 2022), have also been proposed. Elizalde et al. (2023) and Wu et al. (2023) independently proposed CLAP, in which a language encoder and an audio encoder are jointly trained on AL datasets, which resembles CLIP. We focused on the language encoder in the AL models, and demonstrated the impact of sentence embedding training. To the best of our knowledge, this is the first step in this direction.
Sentence embedding is an extensively investigated NLP topic. Methods ranging from bagof-word averaging non-contextualized embeddings (Mikolov et al., 2013; Pennington et al., 2014) to training LSTMs (Hochreiter and Schmidhuber, 1997), e.g., SkipThought (Kiros et al., 2015) and InferSent (Conneau et al., 2017), have been proposed to effectively compose individual tokens to meaningful sentence representations. Methods that leverage post-hoc transforming (Li et al., 2020; Su et al., 2021) or finetuning the pretrained BERT in supervised (Reimers and Gurevych, 2019) or unsupervised scenarios (Gao et al., 2021) are also introduced. Zhang et al. (2022) show that grounding sentence embedding learning to images improves semantic textual similarity tasks. We present a focused investigation on the LM in CLIP/CyCLIP. We pretrained from scratch an LM and ResNet-50 (He et al., 2016) with cross-modal contrastive learn-
ing, as well as unsupervised or supervised sentence embedding training with image captions. Verifying the effectiveness of sentence embeddings – a critical component for retrieval and clustering (Reimers and Gurevych, 2019; Gao et al., 2021; Wang et al., 2021; Thakur et al., 2021; Geigle et al., 2022) – is of great importance because retrieval has been one of the main applications of CLIP-like models.

heading: 3 Method
text: Cross-modal contrastive learning plays a key role in training models such as CLIP/CLAP. We take the image modality as an example for introducing this method. Consider a caption-image dataset tpIi, TiquNi“1 that includes N caption-image pairs, and denote Ie and T e as the output representations from an image and language encoder, respectively. The cross-modal contrastive loss (Radford et al., 2021) is defined as
Lcontra.pτq “ ´ Nÿ
j“1 log
exp ´ xIej , T ej y{τ ¯
ř k exp ´ xIej , T ek y{τ ¯
´ Nÿ
k“1 log exp pxIek, T ek y{τqř j exp ´ xIej , T ek y{τ ¯ ,
where τ is a trainable temperature parameter initialized to 0.07 and x¨, ¨y computes cosine similarity.
CLIP’s training objective solely stresses the alignment between the two modalities. CyCLIP (Goel et al., 2022) has improved CLIP (Radford et al., 2021) by additionally optimizing for improved representation space geometry, such that the image and text spaces are more consistent with each other. Concretely, CyCLIP explicitly optimizes two additional objectives for cross-modal and in-modal consistency as well as Lcontra.:
LC-cyclic “ ÿ
j
ÿ
k
`xIej , T ek y ´ xIek, T ej y ˘2 ,
LI-cyclic “ ÿ
j
ÿ
k
`xIej , Ieky ´ xT ek , T ej y ˘2 .
Intuitively, decreasing the cross-modal consistency loss LC-cyclic makes the cross-modal similarity matrix more symmetric, as shown in Figure 1. Note that solely optimizing Lcontra. is expected to symmetrize the cross-modal similarity matrix because the similarity of non-diagonal pairs are trained to be zero. Goel et al. (2022) showed that this scenario does not occur in practice and explicitly optimizing LC-cyclic is beneficial.
Optimizing LI-cyclic, however, reduces the inconsistency between the overall geometry of the text and image spaces. Computing LI-cyclic requires calculating the similarity between two text captions, i.e., xT ek , T ej y. It is thus reasonable to hypothesize that accurately computing caption similarities is beneficial for optimizing LI-cyclic. We use unsupervised and supervised sentence embedding training methods and test this hypothesis (§4).
We use the widely used SimCSE (Gao et al., 2021) method for unsupervise learning caption representations. SimCSE also uses contrastive learning: a caption is input to a language encoder twice to obtain two vectors T e and T e`. With dropout (Srivastava et al., 2014) enabled, T e and T e` are generally different. These paired vectors serve as the positive training pairs for contrastive learning, while mismatched captions form negative pairs. We denote the unsupervised SimCSE loss as
Lspτq “ ´ ÿ
j
log exp
´ xT ej , T ej,`y{τ ¯
ř k exp ´ xT ej , T ek,`y{τ ¯ ,
τ is fixed to 0.05 following Gao et al. (2021). Another direction of sentence embedding training is supervised training (Reimers and Gurevych, 2019) on natural language inference (NLI) datasets, e.g., SNLI and MNLI (Bowman et al., 2015; Williams et al., 2018). We denote the objective as Ln and follow Gao et al. (2021) in using entailment pairs as T e and T e` and the contradiction sentence as a hard negative.
As a result, the overall training objective considered is:
Ltotal “ Lcontra. ` LC-cyclic ` LI-cyclic ` Ls/n.
Table 1 lists various models we use in our experiments as well as their training objectives. During the experiments, we sum up the objectives but
weight them with different hyperparameters (λ) depending on the combinations, which are shown in §4. We add a suffix “s” to the name of models trained with Ls and “n” to models trained with Ln.

heading: 4 Experiments

heading: 4.1 Datasets
text: To pretrain VL models such as CLIP and CyCLIP, we follow Bugliarello et al. (2021); Goel et al. (2022) and use the Conceptual Captions dataset, which consists of approximately1 3M captionimage pairs (CC3M; Sharma et al. (2018)). CC3M has a reasonable size for pretraining and contains a broad coverage of Web content, making it a good option for learning generic VL representations (Bugliarello et al., 2021).
To evaluate the trained VL models, we follow (Radford et al., 2021) and conduct evaluations with zero-shot image-text retrieval on the Karpathy (Karpathy and Fei-Fei, 2015) test splits of Flickr30K (Plummer et al., 2015) and MSCOCO (Chen et al., 2015). We skip the evaluation on Flickr30K when supervised sentence embedding training is used, i.e., when Ln is considered in training. This is because Flickr30K captions are the premises in the SNLI dataset (Bowman et al., 2015), overlapping with the supervised sentence embedding training data. For zero-shot image classification, we use the standard benchmarks CIFAR10, CIFAR100 (Krizhevsky et al., 2009), and ImageNet1K (Russakovsky et al., 2015). Zeroshot image classification with domain shift, out-ofdomain, and adversarial examples are also considered: ImageNetV2 (Recht et al., 2019), ImageNetSketch (Wang et al., 2019), ImageNet-O, ImageNetA, and ImageNet-R (Hendrycks et al., 2021b,a).
For training AL models, e.g., CLAP and CyCLAP, we conduct experiments with Clotho
1CC3M images need to be downloaded by users. Due to broken URLs, the exact amount of data varies from time to time; Table 2 shows the exact number of images.
(Drossos et al., 2020) consisting of «6K captionaudio pairs, AudioCaps consisting of «50K2 caption-audio pairs (Kim et al., 2019), and FreeSound (Fonseca et al., 2017; Mei et al., 2023) consisting of «195K caption-audio pairs. In contrast to the VL scenario, AL pretraining is known to be challenging due to data scarcity (Wu et al., 2023). We explore the effectiveness of sentence embedding training for AL with datasets with various scales of size.
To evaluate trained AL models, we conduct cross-modal retrieval and zero-shot audio classification tasks. For Clotho, we train the models on the training split and report retrieval results on the validation split. For AudioCaps and FreeSound, we select the best-performing checkpoint on the validation split and report test split results. We conduct zero-shot classification on the Environmental Sound Classification dataset (ESC50; Piczak (2015)) and UrbanSound8K (US8K; Salamon et al. (2014)), which have been widely used (Wu et al., 2023; Elizalde et al., 2023). ESC50 consists of short audio clips containing the sound of different common events such as cats meowing and dogs barking; the clips are categorized into 50 classes, and US8K contains audio clips of urban event sounds such as drilling and street music; the clips are categorized into ten classes.
Table 2 lists all the cross-modal datasets and their usage. We follow Goel et al. (2022) in processing the VL datasets and Wu et al. (2023) in
2Similar to CC3M, AudioCaps only provides audio captions while users need to download corresponding YouTube videos and convert their audio to waveforms and Table 2 shows the exact amount of waveforms we used.
processing the AL datasets; the detailed steps of these processes are shown in Appendix §A.1.

heading: 4.2 Experiment settings
text: To pretrain the VL models, we use the same model architecture as CyCLIP (Goel et al., 2022), i.e., a ResNet-50 as the image encoder and Transformer (Vaswani et al., 2017) as the language encoder. We pretrain the model from scratch and largely reuse CyCLIP’s hyperparameters. To weight different training objectives (Table 1) in CyCLIP, we set λI-cyclic and λC-cyclic to 0.25, λcontra. is set to 1.0, and we empirically set λs and λn to 0.1. We use a batch size of 80, and each pretraining trial is run for 64 epochs, taking four days with four A100 GPUs. We enable dropout in the language encoder and use dropout rate of 0.1. Appendix §A.1 lists the details of the hyperparameters.
The VL pretraining dataset CC3M has a validation split consisting of «15K text-image pairs. We select the best-performing checkpoint on this validation split for downstream task evaluations.
Due to data scarcity, AL pretraining often leverages pretrained language and audio encoders. We use the same model architecture as LAION-CLAP (Wu et al., 2023): pretrained RoBERTa-base (Liu et al., 2020) as the language encoder and pretrained Hierarchical Token-Semantic Audio Transformer (HTSAT; Chen et al. (2022)) as the audio encoder. HTSAT has shown to outperform CNNs in various audio tasks (Chen et al., 2022). We use the HTSAT-tiny variant with 31M parameters. Due to smaller dataset size, each pretraining experiment takes less than one day on a single A100 GPU. We use the default dropout rate of 0.1 of RoBERTabase in our experiments when the unsupervised sentence embedding training objective is used. Table 10 lists other hyperparameters used in the experiments.
For weighting the training objectives, due to the small model size and dataset size, we grid search the optimal λI-cyclic, λC-cyclic, and λs from [0.1, 0.25, 0.5]; λcontra. is set to 1.0. Supervised sentence embedding training (λn) is not considered due to the small size of the AL datasets.
For AL datasets Clotho, AudioCaps, and FreeSound, we select the best-performing checkpoint on the validation splits then conduct evaluation on downstream tasks.

heading: 5 Results and analyses

heading: 5.1 Results
text: Table 3 lists the zero-shot VL retrieval results on MSCOCO and Flickr30K, following the settings in Radford et al. (2021). We first utilize the pretrained image and language encoders in a VL model to encode images and captions into vectors. In text retrieval, we then input the image vector to retrieve the aligned captions and vice-versa for image retrieval. We report Recall@N for N in [1, 5, 10].
Comparing CyCLIP and CLIP variants. We see that CyCLIP clearly outperforms CLIP across the board for both datasets, highlighting the significant value of incorporating LC-cyclic and LI-cyclic in optimizing for consistent geometry of the text and image representation spaces (Goel et al., 2022).
When comparing CyCLIPs/CLIPs to CyCLIP/CLIP, we observe the effectiveness of improving the language encoder with unsupervised sentence embedding training Ls. CyCLIPs/CLIPs clearly surpass CyCLIP/CLIP in all configurations except Flickr30K-CLIPs-ImageRetrieval-R@10. This suggests that, we improve CyCLIP on zeroshot vision-language retrieval tasks through learning better representations of the captions. We also observe more gains in text retrieval than in image retrieval. For example, on Flickr30K@1, CyCLIPs outperforms CyCLIP by 2.70% (absolute) in text retrieval and by 1.52% (absolute) in image retrieval.
This observation reflects the effectiveness of improved caption representations.
In contrast to the unsupervised embedding training scenario (Ls and CyCLIPs/CLIPs), supervised sentence embedding training (Ln and CyCLIPn/CLIPn) results in a neutral to negative impact on the overall retrieval results. Investigating the text and image representation spaces, we find that Ln extensively enforces a uniform text representation space such that the alignment between the text and image spaces is negatively affected; we provide more in-depth analyses in §5.2.
AL retrieval results on Clotho, AudioCaps, and FreeSound are listed in Table 4. Supervised sentence embedding training objective Ln is not considered because NLI datasets are much larger than AL datasets (e.g., MNLI: 433K; Clotho: 6K); subsampling introduces extra random factors that are difficult to control.
On FreeSound, we observe improvements when comparing CyCLAP to CLAP, demonstrating that explicitly optimizing for the consistency between the audio and text spaces as in CyCLIP (Goel et al., 2022) is also promising for improving AL retrieval tasks. We also observe consistent improvements of CLAPs/CyCLAPs to CLAP/CyCLAP,
this shows the benefits of integrating unsupervised sentence embedding training objective during AL contrastive learning.
On Clotho, we observe overall improvements when comparing CyCLAP to CLAP. We see only one exception on text-retrieval-R@10 (i.e., 48.90% for CLAP and 48.52% for CyCLAP); When comparing CyCLAPs/CLAPs with CyCLAP/CLAP, we see clear improvements on text retrieval. However, this comes with a decreased performance on audio retrieval results of R@5 and R@10.
On AudioCaps, CyCLAP falls behind CLAP, showing that optimizing for geometry consitency brings no improvements on AudioCaps. The HTSAT audio encoder has already been pretrained with audio classification tasks on AudioSet (Gemmeke et al., 2017), from which AudioCaps is derived. This may contribute to the noisy results. Similarly, LAION-CLAP (Wu et al., 2023) reported that adding additional 630K AL pairs largely boosts AL retrieval performance on Clotho, but hurts on AudioCaps. We observe similar results when comparing CyCLAPs/CLAPs with CyCLAP/CLAP. We further conduct in-depth analyses (c.f. §5.2) on the audio caption properties of different datasets, and find that AudioCaps captions have a small vocabulary and the language use has very small variations, which likely limits the effectiveness of sentence embedding training.
Comparing VL and AL retrieval results in Table 3 and Table 4, we observe that (1) CyCLIP noticeably improves over CLIP than CyCLAP over CLAP; (2) improving the language encoder with sentence embedding training is more beneficial to VL than AL. We hypothesize that this is because AL pretraining starts with pretrained encoders, which have geometry that is difficult to alter due to the small AL dataset size. We further conduct AL pretraining from scratch, where the language and audio encoders are randomly re-initialized. Table 5 shows the results. We observe that sentence embedding training brings more consistent and noticeable results, especially on FreeSound and Clotho. However, not utilizing the pretrained encoders leads to inferior absolute performances due to the small dataset sizes. We believe that resolving the data scarcity issue is still a critical step for future work in AL pretraining.
For zero-shot image classification, Table 6 lists the Top1 accuracy on standard image classification datasets (top) and datasets with distribution shifts or adversarial examples (bottom). We fol-
low Radford et al. (2021) and use their prompts for zero-shot classification. For an image to be classified, we compute the cosine similarity between its vector and the encoded vector of all classes. Each of the classes is reformulated with various prompts. E.g., the ImageNet class “plane” is reformulated with 80 templates3 such as “a photo of a ” and “a blurry photo of a ”, resulting in prompts “a photo of a plane” and “a blurry photo of a plane” (Radford et al., 2021). The vectors of encoded prompts of a class are averaged; we select the class with the maximum cosine similarity with the image vector.
Similar trends as in the retrieval tasks are observed. CyCLIP variants outperform their CLIP counterparts; unsupervised sentence embedding training benefits both CyCLIP/CLIP while supervised sentence embedding training does not result
3OpenAI templates are public on GitHub link.
in consistent improvement or deterioration. For zero-shot audio classification, we follow the VL scenario to write several prompts (c.f. Appendix §A.3) such as “a sound of dog barking” and conduct similar experiments. Table 7 list the Top1 accuracy on ESC50 and US8K of models pretrained on FreeSound, Clotho, and AudioCaps respectively. CyCLAP generally outperforms CLAP, except for AudioCaps-US8K. We observe mixed performance when comparing different model confirgurations, and CyCLAPs/CLAPs perform on par with CyCLAP/CLAP across different datasets. Prompting inherently leads to performance with large variances (Zhao et al., 2021); we leave the extensive “prompt engineering” of designing more prompts for future work.

heading: 5.2 Analyses
text: Alignment and uniformity of representation spaces. In this section, we take a closer look at the learned representation spaces. Following Wang and Isola (2020), we inspect the alignment and uniformity on the hypersphere of learned spaces. Considering a caption-image dataset tpIi, TiquNi“1, we can compute the alignment and uniformity scores defined as:
Lalign fi EpI,T q „ ppos }Ie ´ T e}22 ,
LT,uniform fi log E Ti,Tj i.i.d.„ pdata e´2}T ei ´T ej } 2 2 ,
LI,uniform fi log E Ii,Ij i.i.d.„ pdata e´2}Iei ´Iej } 2 2 ,
where pI, T q „ ppos refers to aligned text-images pairs, pTi, Tjq „ pdata refers to independent and identically distributed (IID) sampled text pairs, pIi, Ijq „ pdata refers to IID sampled image pairs, and Ie, T e respectively refer to the encoded image and text vectors. Recall that the trained models output ℓ-2 normalized vectors residing on the unit ball.
Intuitively, we want vectors of aligned pairs of two modalities to be well aligned in the representation space, such that Lalign is close to zero. On the other hand, we want the space of a single modality to be more uniform than anisotropic (Ethayarajh, 2019; Wolfe and Caliskan, 2022), such that the overall representation space capacity is well used. This results in a near minus infinite Luniform.
Figure 2 illustrates the results. We only show Lalign w.r.t. LT,uniform since our main focus is the language encoder. We see that unsupervised sentence embedding training trades cross-modal alignment for improved text space uniformity. For VL pretraining, supervised sentence embedding training (CLIPn/CyCLIPn) overly focuses on text space uniformity while the VL space alignment deteriorates, as evidenced when visualizing them using the Flickr30K (›) dataset in which the captions largely overlap with the dataset for supervised sentence embedding training (§4.1).
Another interesting observation is that the text encoder of CyCLIP/CyCLAP outputs representation space less uniform than that of CLIP/CLAP, as shown in Figure 2 (x-axis). Liang et al. (2022) show that randomly initialized encoders output vectors residing in different cones. The in-modal cyclic loss LI-cyclic (§3) stresses consistency between cones; it is thus expected to be challenging to learn a uniform space while simultaneously preserving consistency between two spaces4. Sentence embedding training provides extra training signals.
Audio dataset analyses. We show in Table 4 that sentence embedding training consistently improves on the largest AL dataset FreeSound; the benefits diminish on smaller datasets Clotho and AudioCaps. In addition to dataset size, we fur-
4We find that improving text space uniformity also benefits the image space for CyCLIP. More discussions are presented in Appendix §A.2.
ther conduct in-depth analyses on the three AL datasets. We compute word frequency (then normalized by total number of words) of captions in the AL datasets, and then sort the words in decreasing order. Figure 3 shows the results in log scale. AudioCaps has the smallest vocabulary size, and there are little variations on the word use (i.e., a small group of words dominant the captions). This property could hinder improving the uniformity of the text space. Clotho, «9 times smaller than AudioCaps, has a larger vocabulary and a more uniform word frequency distribution of its captions. FreeSound has the largest number of and most diverse captions, and integrating sentence embedding training consistently improves the AL tasks. We also show the results of CC3M as a reference.
LM quality. We evaluate the language encoder quality of pretrained VL models. Our motivation is two-fold. First, as a sanity check, we want to verify that incorporating sentence embedding training in VL contrastive learning still improves the language encoder’s ability of representing general sentences. Second, the evaluation results will help us measure the compatibility and possible interferences among the various training objectives (Pfeiffer et al., 2023).
To this goal, we use the sentence embedding benchmark SentEval (Conneau and Kiela, 2018). Default SentEval configurations are used in all experiments, and we conduct both intrinsic (e.g., semantic textual similarity) and extrinsic tasks (e.g., sentiment analysis). Table 8 lists the averaged results on SentEval (Appendix §A.6 shows individual results). We observe that unsupervised sentence embedding training is generally beneficial for CyCLIP/CLIP on both intrinsic and extrinsic tasks. Supervised sentence embedding training results in more significant improvements5, however, it negatively affects CyCLIPn on the sensitive intrinsic tasks.

heading: 6 Conclusion
text: We extensively investigate the effectiveness of sentence embedding training for pretraining contrastive vision-language and audio-language models. We show that it improves vision-language pretraining, resulting in a better CyCLIP. Sentence embedding training also improves audio-language pretraining on large datasets, while the benefits diminish on small datasets. We conduct comprehensive analyses and show that sentence embedding training increases text space uniformity, but with a cost of reduced cross-modal alignment.

heading: 7 Limitations
text: We restrict our scope to cross-modal contrastive models with three most common modalities: language, image, and audio. While contrastive learning has been successfully extended to other modalities such as music, incorporating music poses additional challenges, particularly regarding licensing and the heterogeneity of music sources. Downloading music from the internet and mining reliable music-language pairs are time-consuming tasks, which we did not consider in detail in this study. Nevertheless, we conducted initial experiments on the music modality using MusicCaps (Agostinelli
5Improvements of supervised sentence embedding training may due to the observation that NLI datasets have similar domains and language use as SentEval tasks. In Appendix §A.5, we show that the improvements are indeed from supervised training, rather than domain similarity.
et al., 2023) and show promising results in §A.7. Another limitation of our work is that models with decoder such as BLIP (Li et al., 2022) and tasks such as visual question answering (Antol et al., 2015) are not considered; we leave them to future work.
In our audio-language pretraining experiments, we explored both pretraining from scratch and pretraining from publicly available language and audio encoders. However, we believe that a more promising direction would involve adapting the pretrained language encoder to the audio domain by performing additional pretraining on audio descriptions before engaging in cross-modal contrastive learning. Nevertheless, we chose to follow the current methods in the literature to ensure consistent evaluations and facilitate meaningful comparisons.

heading: A Appendix
text: A.1 Datasets and hyperparameters For VL pretraining, our experiments largely follow those for CyCLIP (Goel et al., 2022) and for CLIP (Radford et al., 2021). We also directly reuse CyCLIP training hyperparameters but with a smaller batch size, as listed in Table 9. We highlight that Goel et al. (2022) use more pretraining data than us. Concretely, Goel et al. (2022) use 2,631,703 text-image pairs from CC3M and randomly sampled 368,297 text-image pairs from CC12M. As a result, 3,000,000 text-image pairs are used. In our case, we only use our version of CC3M without randomly sampling from CC12M (which is difficult to control). Our downloaded CC3M has 2,806,641 images. Thus, our pretraining dataset is 7% smaller than Goel et al. (2022).
For AL pretraining, our experiments largely follow that of for LAION-CLAP (Wu et al., 2023). To process the audio data, we sample the wavefiles at a rate of 48kHz and then convert them to FLAC
format using FFmpeg6. We then use a hop size of 480, window size of 1024, and 64 mel-bins for computing Short-time Fourier transform (STFT) and mel-spectrograms. The audio encoder input thus has a dimension of 1024 for time steps and 64 for frequency bins. We list the hyperparameters in Table 10.
A.2 Text and image space consistency
CyCLIP variants ensure cross-modal consistency, such that improving the uniformity of the text space with sentence embedding training also benefits image space uniformity as shown in Figure 4. As expected, this observation does not hold for CLIP.
A.3 Zero-shot audio classification with prompts
To resemble zero-shot image classification in VL experiments, we write several prompts for zeroshot audio classification, as listed in Table 11.
A.4 Extended results of audio-language models
We observe that the differences between Table 4 results are smaller than in the VL scenario. As a result, we repeat each experiment three times and then report mean and variance of the results in Table 12. Similar observations as in Table 4 are obtained.
A.5 Unsupervised sentence embedding training with NLI datasets
When evaluting the language encoder on SentEval tasks (§5.2), it is possible that the improvements brought by supervised sentence embedding training is due to the fact that NLI datasets have similar domain and language use as the SentEval tasks. We thus condcut a new type of training, where we use sentences in the NLI datasets for unsupervised sentence embedding training with SimCSE, in addition to VL contrastive learning. We name this new training scheme as CLIPe and CyCLIPe.
Table 16 shows that the new training schemes, CLIPe and CyCLIPe fall behind the supervised
6https://ffmpeg.org/

heading: 2.25 2.00 1.75 1.50 1.25
text: sentence embedding training counterparts CLIPn and CyCLIPn on SentEval. This confirms that the gains of supervised sentence embedding trainings is from the NLI task supervision, e.g., premise and hypothesis relations, instead of other factors such as domain. For completeness, we also report CLIPe/CyCLIPe performance on VL retrieval tasks in Table 13 and zero-shot image classification in Table 17.
A.6 Complete results on SentEval
Our intrinsic evaluation tasks are the semantic textual similarity tasks: STS12-STS16, STS-B, SICKR (Marelli et al., 2014; Cer et al., 2017; Agirre et al., 2012, 2013, 2014, 2015, 2016). Ex-
trinsic evaluation tasks are movie review (MR; Pang and Lee (2005)) product review (CR; Hu and Liu (2004)) subjectivity status (SUBJ; Pang and Lee (2004)), opinion polarity (MPQA; Wiebe et al. (2005)), sentiment analysis on SST2 (Socher et al., 2013), question-type classification (TREC; Voorhees and Tice (2000)), and paraphrase detection (MRPC; Dolan et al. (2004)). Table 14 shows individual task performances.
A.7 Preliminary experiments on the music modality
We further conducted a new experiment with the music modality: music-text retrieval on the MusicCaps dataset introduced by MusicLM (Agostinelli et al., 2023). MusicCaps consists of 5521 musiccaption pairs, of which 2858 pairs are for training and 2663 are for validation. Each music clip is associated with hand-curated English descriptions (including genre, mood, tempo, singer voices etc.) from expert musicians. We use MusicCaps because it is open-sourced and publicly available. Following tables show the retrieval results (the same experiment configurations as the audio modality are used; cf. §A.1).
It can be observed from Table 15 that improving the text encoder with unsupervised sentence embedding training also helps music-text retrieval in the music modality, especially in the text retrieval scenario (CLAPs generally outperforms CLAP; CyCLAPs generally outperforms CyCLAP). These music modality results are consistent with our previous findings on the image and audio modalities, and we plan to explore more in this direction in future work.

heading: Future Work
text: These music modality results are consistent with our previous findings on the image and audio modalities, and we plan to explore more in this direction in future work