UniCLIP: Unified Framework for Contrastive Language-Image Pre-training
Authors: Janghyeon Lee
Jongsuk Kim
Hyounguk Shon
Bumsoo Kim
Seung Hwan Kim
Honglak Lee
Junmo Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Contrastive Learning
Vision-Language Pre-training
Self-Supervised Learning

Abstract: Pre-training vision-language models with contrastive objectives has shown promising results that are both scalable to large uncurated datasets and transferable to many downstream applications. Some following works have targeted to improve data efficiency by adding self-supervision terms, but inter-domain (image-text) contrastive loss and intra-domain (image-image) contrastive loss are defined on individual spaces in those works, so many feasible combinations of supervision are overlooked. To overcome this issue, we propose UniCLIP, a Unified framework for Contrastive Language-Image Pre-training. UniCLIP integrates the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space. The discrepancies that occur when integrating contrastive loss between different domains are resolved by the three key components of UniCLIP: (1) augmentation-aware feature embedding, (2) MP-NCE loss, and (3) domain dependent similarity measure. UniCLIP outperforms previous vision-language pre-training methods on various single- and multi-modality downstream tasks. In our experiments, we show that each component that comprises UniCLIP contributes well to the final performance.

UniCLIP: Unified Framework for Contrastive Language-Image Pre-training
Janghyeon Lee" LG Al Research
Jongsuk Kim*t KAIST j jkkpop0kaist.ac ac.kr
j aaghyeon... 1ee@1greeearch..a. ai
Hyounguk Shon i
Bumsoo Kim
KAIST hyounguk. shon ac. kr
LG AI Research
bumsoo. kin@lgresearch.cc ai
Seung Hwan Kim
Honglak Lee
Junmo Kim KAIST
LG Al Research
LG Al Research
sh. krim@ggresearch.
hongl akkl gra.iudgggeeeee......
j junmo. kimkkaist. ac. kr
Abstract
Pre-training rision-language models with contrastive objectives has shown promis- ing results that are both scalable to large uncurated datasets and transferable to many downstream applications. Some following works have targeted to improve data efficiency by adding self-supervision terms, but inter-domain (image- text) contrastive loss and intra-domain (image- image) contrastive loss are defined on individual spaces in those works, so many feasible combinations of supervision are overlooked. To overcome this issue, we propose UniCLIIP a Unified framework Pre-training. UniCLIP integrates the contrastive
for Contrastive Laaguage--maa loss of both inter-domain pairs and intra-domain pairs into single universal space. The discrepancies that occur when integrating contrastive loss between different domains are resolved by the three key components of UniCLIP: (I) augmentation- aware feature embedding, (2) MP-NCE loss, and (3) domain dependent similarity measure. UniCLIP outperforms previous vision-language pre-training methods various single- and multi-modalig modality downstream tasks. In experiments, we show that each component that comprises UniCLIP contributes well to the final performance.
Introduction
Recent advances in deep learning have shown significant progress in pre-training large-scale models that transfer well to various downstream applications. Following the success of this paradigm in both fields of computer vision and natural language processing, vision-language pre-training models [9, 20] that learn image representations from natural language supervision have been proposed. In those works, pre-traiinn pre-training is done under simple contrastive loss that makes the embedding of an image and its matching text description (positive pair) more similar each other than other arbitrary image-text pairs (negative pairs).
Towards data-efficient pre-training objective, subsequent works [13, 16] introduced additional self-supervision terms to the image- text contrastive loss, including self-supervision for augmented
'Equal contribution. Alphabetical order. Work done during an internship at LG Al Research.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

images [3, 4], augmented texts [28], and masked texts [13]. Involving more pairs of positive/negative supervisions into the final contrastive loss leads to a more mathematically pleasing objective [3], thus enabling the model to be more data-efficient. Yet, these works entail a major limitation since the contrastive loss for intra-domain pairs, such as image-image pairs, and inter-domain pairs, such as image- text pairs, are defined independently in separated spaces. This means that the contrastive loss is unaware of a substantial set of feasible combinations for negative supervision, for instance image-image pairs are not included when calculating the contrastive loss for image-lext supervision, leaving a huge room for improvement in terms of data-efficiency and feature-diversity, Based on this observation, we set the goal of this paper to build contrastive image- text pre- pre-training framework where the contrastive learning of all possible intra-domain and inter-domain pairs is defined in the same single unified embedding space.
possible
Though this goal sounds intuitive, defining a contrastive loss between the multiple modalities in a unified space has several challenges. First, misalignments can occur between the image-text semantics when applying image augmentations. For example in Figure I, the semantic of "a red apple is on the right of sliced green apples' can be easily broken by simple image augmentations like horizon- tal flipping, converting grayscale, cropping, whereas they are fundamental augmentations used in image-image contrastive self- supervueee learning [3]. We validate from
A not spple on tu of slol greee applis
our experiments that leaving this discrepancy unattended hinders training and degrades final performance. Secondly, existing contrastive losses in literature for multi-positivg pairs [10, 15] are not compatible with our training ob- jective that deals with embedding of different modalities, This is because intra-domain pairs, like two different aug- mented views of a single image, serve as relatively casier examples than inter-domain pairs like image- text pairs. Existing losses [10, 15] are vulnerable to this condition as casy-positive examples and hard- sord-posiiive examples inter- fere with each other. Lastly, we discovered that applying the same similarity measure between embeddings from different modalities in our contrastive loss results in suboptimal performance, because there are inherent differences in similarity measures between inter-domain and intra-domain pairs, i.e., samples in an intra-domain pair can be arbitrarily close but samples in an inter-domain pair cannot.
and apple inon the right
A aeple on the right
repin of sooo grosiiggles
Image-text misalignments caused by data augmentations. The mis- aligned texts are highlighted in red (best viewed in color).
Figure 1:
In this paper, we propose UniCLIP: a Unified framework for Contrastive Language Language-Image Pre- training, that unifies contrastive objectives between multiple modalities on single embedding space. Each challenge above is addressed with our key components of UniCLIP: (I) augmentation-aawar
feature embedding that makes UniCLIP aware of misalignments caused by data augmentations, (2)
MP-NCE loss that is designed to stabilize training for both easy- and hard-positive pairs, and (3) domain dependent similarity measure that adjusts the difference in similarity scales between inter- domain pairs and intra-domain pairs. UniCLIIP outperforms existing vision- languae pre-training methods in various single- and multi-modal downstream tasks such as linear probing, zero-shot classification, fine-tuning, and image-text retrieval, by addressing the three problems described above. We validate that each component of UniCLIP successfully addresses the issues of contrastive learning in a unified space and meaningfully contributes to the final performance. Our contribution is summarized as follows:
We propose UniCLIP, a unified framework for visual-language pre-training that improves data-efficiency by integrating contrastive losses defined across multiple domains into a single universal space. We study new technical challenges that occur from this integration.
We design new components for UniCLIP to address the aforementioned challenges: feature embedding, MP-NCE loss, and domain dependent similar- ity measure. Our extensive experiments show that each of our proposed components serves a key role in the final performance.
augmentation-avaal
UniCLIP outperforms existing vision- langgage pre-training methods across multiple down- stream tasks that include various modalities.

---

Text caption
Text Projection Head A22
Text fx
red apple on the right sloced green apples
Unified Embedding Space
MiO
Augmentation Enzodde fA
odke
image
image augmentte wiew
Image Encodor
Image Projection gi
M-thh image augmented wkew
concat
crp prysicde
Augmentation fA
misalignment adjustments by UniCLIP
Figure 2: Overview of the UniCLIP framework.
2 Methods
The UniCLIP architecture (Figure 2) consists of augmentation encoder S'A, image encoder Jr, a text encoder Jr, and corresponding projection heads and gr. J1 encodes an image to an sugmentation-agnostt image representation and then ØI outputs an augmentation-awee -ware image embedding. For text caption data, fr and produce text embeddings on the same embedding space the image embedding space. Image and text representations are learned by our multi-positive NCE loss with Nomain-depended similarity scores measured on the unified embedding space. Each element of our method described in detail in the following sections.
2.1 Architecture
Augmentation Encoder To enable an augmentation instruction A to be used as an input to a network, we first describe it real vector containing information about how much each basic transformation A is applied to data. For example, image augmentations that frequently appear in contrastive learning can be converted real vectors follows:
Crop & Resize: RandomResizzedrrop augmentation is encoded a four-dimensional vector of (ar, y, h), where (r. y) is the top left corner coordinate of cropped image and (w, h) is the size of the cropped image, in a normalized coordinate system (i.e., the top left corner of the original image is (0, 0) and the bottom right corner is (1, 1)).
Color Jitter: As a ColorJitted augmentation changes the brightness, contrast, saturation, and hue of an image, this augmentation is encoded to four-dimensional vector consisting the changes those four factors.
Gaussian Blur: A GaussianBlur augmentation is encoded to the standard deviation of its Gaussian blurring kernel.
Horizontal Flip: RandomHorrzzotallFlip augmentation encoded if an image actually flipped and 0 otherwise.
Grayscale Convert: A RandomGrayscale augmentation is encoded to 1 if an image is actually converted grayscale and ( otherwise.
an image augmentation A is composed of all five augmentations described above, will be first encoded to 11-dimensional vector according to the above rules and then pass through an MLP to obtain the augmentation embedding fa (AA). Note that (A(A) will different for each forward and each sample because of the randomness of the augmentation.

---

Image Encoder & Image Projection Head For the model to learn how to adjust for image-tery text misalignment caused by image augmentations, the image encoder or projection head must take the augmentation information as input. However, the encoder cannot fully benefit from augmented data if it knows which augmentation was applied to the image. For example, when the encoder trained with horizontal flip augmentation, and if it takes an augmented image and an flag of whether the image is flipped or not as input of the form (image, not flipped flag) or (flipped image, flipped flag), then the encoder may exhibit undesirable behavior when it has to encode (flipped image, not flipped flag) from some downstream task, since the encoder was not trained on this kind of data, which means that the model has lost some generalization ability. Therefore, the image encoder must be augmentation-agee agnnsttcc and the image projection head must be augmentation-awen awaar. In this way, the encoder can fully enjoy the benefits of data augmentation and generalizes better, while the projection head is still able to correct inter-domain misalignments caused by the augmentations.
To make image representations augmentation-agman aggnostic and image embeddings augmentation-awaat the augmentation information is provided only to the projection head, whereas the encoder only sees the augmented image without knowing which augmentation has been applied. Therefore, for an image I, the image encoder fI takes an augmented image A(z) as input to get an augmentation- agnostic image representation h = fr(()r)). Then, an augmentation-aver
image embedding 2 gu (fr (AA))), fa (A)) in the unified embedding space is obtained from the image representation h and the augmentation embedding (AA) by the image projection head g1- We adopt ViT (Vision Transformer) [7] as the image encoder fi with learnable positional embeddings and the image projection head 9, is composed of three residual blocks. The last activation value of the [cls] token is used as the image representation h.
Text Encoder & Text Projection Head A raw text is first tokenized by byte pair encoding and wrapped with a start token and an end token, resulting in a tokenized text æ. Any text augmentation method can also be applied here as in the case of the image embedding, but we do not create multiple augmented views for a text as we found it not very helpful. So, the text representation h = fr(x) and the text embedding emhedding. We use Transformer [27] for the text encoder fr with learnable positional embeddings and linear layer for the text projection head gr. The last activation value of the start token used as the text representation h.
gr (x)) in the unified latent space are obtained without any augmentation
2.2 Contrastive Loss Functions for Multiple Positive Pairs
Contrastive loss functions can be classified according the number of positive and negative pairs
taken by the loss for one data point. For example, triplet loss [22] takes only a single positive pair
and single negative pair, N N-pair loss [24] and InfoNCE loss [26] take single positive pair and
multiple negative pairs, and MIL-NCE loss [15] and SupCon loss [10] take multiple positive pairs
and multiple negative pairs. As there are multiple positive pairs in our unified framework, we first review MIL-NCE loss and SupCon loss functions and discuss their drawbacks.
For i-th embedding 2; in a batch of embeddings { let P. be the set of all positive sample indices of the i-th sample excluding i itself and N, be the set of all negative sample indices of the i-th sample.
P: [j](z,, zz) is positive pair and * i}
(1)
N₂ (2, 23) a negative pair)
(2)
A similarity score between the i-th and j-tjjthh j-th embedding is denoted by $ty > 0. A contrastive loss function will try to maximize the similarity scores of positive pairs, while minimize the similarity scores of negative pairs. For example, if there is only one positive sample for each sample in batch, say P.- = } then InfoNCE loss [26] or NT-Xent loss [3] for the i-th sample can be described by
C ClafeNCE
log Si..
8i.p
(3)
E Si,
MIL-NCE Loss
MIL-NCE loss [15] for the i-th embedding is defined by
EpEp: Si.p
C CMI.NCE = -log Epep. 8i,p + EmEN, 8i,
(4)
4

---

The MIL-NCE loss function is configured to maximize the sum of all positive pair similarity scores Eper; Si,p and minimize the sum of all negative pair similarity scores ΣnEN; Si,, However, hard positive pairs cannot receive enough gradients from - when there are easy positive pairs whose similarity scores are sufficiently large to dominate the numerator and denominator, as the MIL-NCE loss compares negative pairs with the sum of positive scores ΣpEP, st.p only, not each positive pair Si,p individually, For some q E P.. the gradient from C CMIL-NCE Si,g
MLMILNNC
EnEN: 8i,m
(5)
Dsig
Epep, Sip)
Epep, Si,p + EnEN, $i,m
therefore the gradient will vanish to zero when EpEp, Si,p is already large because of easy positive pairs even if the positive pair's score $i.q is small. In other words, easy positive pairs hinder the training of hard positive pairs in MIL-NCE loss. This problem will be more pronounced in unified framework because hard positives and easy positives frequently coexist with supervisions from intra-domain and inter-domain.
SupCon Loss
SupCon loss [10] for the i-th embedding is described by
4. SupCon
EpeP:
log
Si.p Ep'ER $1.8/ + EnEN, 81,0
In this case, each positive score $t,p is compared with the negative pairs, but the sum of the positive scores in the denominator still causes an undesirable side effect. For an easy positive pair with large similarity score, it can be possible to decrease the loss by decreasing its score and so the denominator. For q E P..
177 (EpER. St.p + EnEN, $t,13
$1.9
Osig
85,9 [EEEp St.p + EmEN, 8t,n
so hard positives would be trained better than MIL-NCE loss because of relatively large update by the Si.g term in the denominator. However, if we assume the sum of positive scores is much greater than the sum of negative scores, then
OC, SupCon
81.0
1 P.P
Σ PEP.
St.
Σ nEN;
Sin
81,9
EpEP, [sc.p]
(8)
Usia
As gradient is not always negative, SSpCConn will try to decrease the similarity score of an easy positive pair (z,, 2g) since si.q will be larger than the average positive score, instead of increasing or least maintaining it. In other words, hard positive pairs hinder the convergence of easy positive scores in SupCon loss.
Multi-positive NCE Loss As the sum of the positive scores in the denominator causes easy and hard positive pairs to interfere with each other, can just use multi collii-postive version of InfoNCE loss to make each positive pair independently contribute to the loss as follows.
C, EpEP
log 8t,p + EnEN, Si,2
8i,
(9)
As can be seen in the gradient
OC:
ENEN, 8i,
(10)
gsigg
P. st.9 (si.g EmEN, si,))
hard positive samples be trained with sufficiently large update from the $i,q term in the denomina- tor, and the decreasing easy positive pair similarity problem does not occur as the gradient is always negative.
With this multi-positive version of InfoNCE loss, we reconsider excluding i from the positive set P. in Equation 1. If contrastive loss can handle multiple positive pairs, then there is no reason to
5

---

exclude the trivial pair (2i, 24) from the loss definition. Since 24 is most similar to 2; itself, the trivial pair must be also utilized as a strong positive pair. which will result in
Si.p
C ())
log
(11)
si.p + E Sim
Here, propose a multi-positive NCE loss for our unified contrastive learning framework called MP-NCE loss, which is a weighted version of Equation 11 defined as
D. M.P.NCE = EEEPPU
(iD(ip)) log
8i,
(12)
Si. Σ.EN, 8tn
where D(i,p) indicates the domain combination from which the i-th and p-th data were sampled, and (D)).p) is a domain-specific balancing hyperparameter which makes each inter-domain and intra-domain supervision equally contributes to the loss. For example, when we use three augmented views of an image and one corresponding text for each original image tegge-text text pair from dataset, there are a total of 9N image -mage positive pairs, 6.N image text positive pairs, and N text-text positive pairs in a batch, so ('((.p) is set to 1/9, 1/6, 1 (2i, 2p) an image image pair, image- text pair, text-text pair, respectively.
Although we have proposed MP-NCE loss in multi-positive setting, one should consider using MP-NCE loss even in single positive settings, such as image self-supervised contrastive learning, by treating a trivial pair (24, 24) positive as well since MP-NCE involves negligible computational overhead compared to backbone networks.
2.3 Comain-Dependent Similarity Score
In SimCLR [3] and CLIP [20], the similarity score 8i,j between the i-th embedding z; and j-th embedding 2y is defined by
2, Zj
,1 = exp
(13)
where is a positive real number, usually smaller than 1. As the cosine similarity of two embeddings cannot have a value outside the interval [-1, 1], the cosine similarity is divided by the temperature т be pre- pre-defined hyperparameter, or rather be a learnable parameter
to extend its range. allowing the model to choose an appropriate scale for the convergence of a contrastive loss.
To classify input pair 2z) as positive negative, we can define a threshold b and classify it as positive if the cosine similarity between 2; and zj is greater than b, and negative otherwise. We may absorb this threshold b into the similarity score as offset like
Si. exp ( 2, |= ((Ei))=))))
(14)
and expect that the optimal threshold will be learned by the model, as in the case of the temperature. Note that the temperature will amplify the score if the cosine similarity is greater than b otherwise reduce it, so Equation 14 a reasonable similarity measure with which the threshold can be treated as a decision boundary for the binary classification problem. However, unfortunately, the offset b does not contribute to InfoNCE loss (Equation 3) at all since b's the numerator and denominator cancel out as
exp(b/T)si.p
CIafaNCE
log
St.p
log exp(b/T)st.p. + EneN, exp(b/T)s..n
(15)
Si.p. + EnEN, Si,n
for any and which means @CInfNNCE/// /āb is always zero.
On the other hand, when data pairs are sampled from multiple domains as in our unified framework, the threshold can be different depending on whether the sampled data pair is an intra-domain pair or an inter-domain pair, as it would be easier to classify intra-domain positive pairs than inter-domain positive pairs in general. This motivates us to introduce domain-specifit temperature TD(3,) and offset (D(t))) and propose a domain-dependent similarity score
2,T 23
bD((.)) ))
$1.j = exp
(16)
(D((.))
|

---

Table 1: Zero-shot image classification performance and linear probing performance on 11 down-
stream datasets. (Results reported in the original paper.
CIBAR 100
Caltech-10 101
CIFAR
SUN399
Food- 101
Flowors
Airerall
ImageNet
Average
Pre-train dataset
Pets
Cars
DTD
Method
Zero-shot classilcatiin:
CLIP-VET -F3-32-B/32
YFCC15M
19.4
62.3
33.6
40.2
33.7
6.3
55.4
1.4
16.9
31.3
27.5
SLIP-VIT-B/32
YFCCISM
28.3
72.2
45.3
45.1
44.7
65.9
1.9
21.8
38.3
33,9
YFCCI5M
30.2
72.1
39.7
51.6
46.9
7.1
3,9
70.1
2.5
24.2
41.2
35.4
UniCLIP-VIT-BS32
YFCCI5M
32.5
78.6
47.2
50.4
48.7
8.1
3.4
73.0
2.8
23.3
42.8
37.3
DeCLIP-ResNet50 [13]
Open30M
49,3
UnICLIP-ViT-B/33
Open30M
61.1
84.0
4.7
Linear probing:
CLIP-VIT VVT-B/32
YFCCISM
71.2
89.2
72.1
70.1
71.4
93.2
34.9
84.3
29.7
60.9
61.1
67.1
SLIP-VIT-B/32
YFCC15M
75.4
90.5
75.3
73.5
77.1
96.1
43.0
87.2
34.1
71.1
68.1
71.9
DeCLIP- ViT-B/32
YFCCISM
76.5
88.6
71.6
75.9
79.3
96.7
42.6
88.0
32.6
69.1
69.2
71.8
UniCLIP-ViT-B/22
YFCC15M
83.1
92.5
78.2
77.0
81.3
97.1
49.8
88.9
36.2
72.8
70.8
75.2
UniCLIP-ViT-BIA.
Open30M
85.4
95.1
81.5
79.2
84.4
97.3
67.3
91.1
39.0
77.2
74.0
79.1
For image-text unified contrastive learning, we have three possible domain combinations, there will three different temperatures and three offsets respectively for image-image pairs, image-text pairs, and text-text pairs.
With the proposed Nomain-depennee similarity score (Equation 16) and MP-NCE loss (Equation 12). the offsets are no longer cancelled out as negative pairs are sampled from multiple different domains. Specifically, because any real number can added the cosine similarity term as Equation 15 without changing the loss function, the offsets lose only 1 intrinsic dimension and thus the model is able learn relative thresholds. In other words, is now possible learn the Nomain-sspecioe offsets so that can expect the offset of an easier domain combination to be greater than that of harder one.
3 Experiments
Datasets For reproducibility, we use publicly available datasets for training and evaluation in
our experiments, including CC3M [23]. CC12M [2], DeCLIP YFCCI5M [13, 25] for training and
Pets [18], CIFAR-IO, CIFAR- [12], SUN397 [29], Food-101 []], Flowers [17], Cars [11], Caltech-
101 [8], Aircraft [14], DTD [6], ImageNet-Ik [21]. Flickr30k [19]. COCO Captions [5] for evaluation.
We define the union of CC3M, CC12M, and YFCCI5M as Open30M dataset.
Settings For each original image and corresponding caption, one weakly augmented image, two strongly augmented images, and one text form a positive group in our experiments. Detailed augmentation and optimization configurations can be found in Appendix.
3.1
Main Results
We evaluate the transferability of our model in modal and multi--odall downstream tasks. Lin- ear probing and fine-tuning on image classification tasks are performed for single-modal benchmarks, and imate-text retrieval tasks and zero-shot image classification tasks evaluated for multi-modal benchmarks.
Linear Probing Fine-Tuning For single-modal experiments, we remove the image projection head 91 and augmentation encoder fA, and use only the im- age encoder f1. Table reports linear classification performances on downstream datasets. We report ImageNet fine-tuning accuracy in Table 2. UniCLIP consistently outperforms other methods on all down- stream datasets in the single-modal experiments.
Table 2: ImageNet- Ik fine-tuning accuracy
for the models pre-trained on YFCCISM.
Method
Accuracy
72.27
SLIP-ViT-B/32
75.64
DeCLIP-VIT-B/3
74.34
UNCCLIP-VTT-BB32
76.54

---

Table 3: Zero-shot image-test text retrieval on the test splits of Flickr30k and COCO Captions with models pre- wre-trained on YFCCI5M. "Pre-trained on Open30M.
Image-to-text retrieval
Text-to-image retrieval
Flickr30k R@5
COCO Captions
Flickr30k R@5
COCO Captions
Method
R@10
R@1
R
R@1
R@
R@l
R@s
R
CLIP-VIT-B/32
34.9
63.9
75.9
20.8
43.9
55.7
23.4
47.2
58.9
13.0
31.7
42.7
LLIP-VIT-B/32
47.8
85.9
27.7
63.9
32.3
58.7
18.2
39.2
51.0
51.4
80.2
88.9
28.3
53.2
64.5
34.3
60.3
70.7
18.4
39.6
51.4
UniCLIP-VTT-BAX
52.3
81.6
89.0
32.0
57.7
69.2
34.8
62.0
72.0
20.2
43.2
54.4
UniCLIP-VIT-B/32
75.6
94.2
97.3
46.1
74.0
83.0
61.4
85.2
91.5
35.2
61.3
71.7
Zero-Shot Classification & Image-Text Retrieval Table shows zero-shot classification perfor- mances 11 downstream datasets. We perform prompt ensembling for each class with the same prompt templates as [16, 20J. Table 3 shows the results of zero-shot image-text retrieval on Flickr30k and COCO Captions benchmarks.
3.2 Ablation Studies
In this section, report experiments inspect how each component of UniCLIP contributes the final performance. We pre-train all variants of UniCLIP with a ViT-B/16 backbone on the CC3M dataset for 50 epochs, and compare their ImageNet- Ik zero-shot evaluation performances.
Image Projection Head Types We tried several different architectures for the image projection head including linear layer, MLP layers, and residual blocks, when the head takes augmentation embeddings as input or not, as Table 4a. turns out that MLP shows strong tendency to overfit, even performs worse than linear layer. By adding skip connections to the head, it can fully utilize augmentation information with increased capabilities while avoiding overfitting. Making the image projection head augmentation-awee improves the performance for all head types the head can handle the inter-domain misalignments.
Augmentation Configurations Table 4b studies the effect of augmentation encoding vari- ous augmentation configurations. Using only strongly augmented images severely degrades the performance without augmentation embedding, because strong augmentations will generate more image-text misalignments. Since it is observed that including one weakly augmented image works better than using only strongly augmented ones, choose to keep weakly augmented image in the positive set as a stable reference sample.
Table 4: ImageNet- Ik zero-shot accuracy with varying image projection head types and augmentation configurations.
(a) Image projection head types. One weak and strong image augmentations used.
(b) Augmentation configurations. I-ResBlock
head is used for augmentation embedding fig 3-ResBlock head is used with augmentation embedding.
Augmentation embedding
Head type
Accuracy
Augmentation embedding
MI. 3 layers
24.01
Augmentation
Accuracy
MLP layers
23.62
X
ResBlock
24.76
3 weak weak, strong
24.49
3 ResBlocks
24.46
X
24.76
strong
22.60
Linear layer
24.68
3 weak weak, strong
23.40
MLP layers
24.54
27.84
MI.P 6 layers
24.15
strong
26.43
ResBlock
27.67
3 ResBlocks
27.84

---

Domain-dependent Similarity Score and Unified Supervision In Table 5, we can see how the performance changes depending on whether the shared similarity score (Equation 14) or the domain- dependent score (Equation 16) is used. We also run experiments where positive and negative sets are formed separately with respect to the domain combination as in SLIP [16] and DeCLIP [13]. The best performance comes out from the domain-dependent similarity measure with unified supervisions, as expected.
Table 5: ImageNet- Ik zero-shot accuracy with domain-dependency of similarity score and supervi- sion.
Temperature and offset
Supervision
Accuracy
Shared across domains
Unified
25.51 26.59
Nomain-dependeen Domain-dependent
Separated
Unified
27.84
Loss Functions As analyzed in Section 2.2, SupCon loss [10] outperforms MIL-NCE loss [15], but performs worse than the pulti-positive version of InfoNCE loss (Equation 9), in Table 6. The balancing weight (i.ppp) can boost the performance, and surprisingly, we can significantly improve performance with negligible additional computations by simply adding a trivial pair (2i, zi) to the positive set P..
Table 6: ImageNet-Ik Ik zero-shot accuracy with different loss functions.
Loss function
Accuracy
MIL-NCE
22.23
SupCon
23.04
MP-NCE w/o trivial pair (zi, zi) and Wp((.) (Eq. 9)
24.60
MP-NCE w/o WD(..) (Eq. 11)
26.41
MP-NCE
27.84
4 Conclusion
We have proposed UniCLIP, a unified framework for visual-language pre-training that improves data- efficiency by integrating contrastive losses defined across multiple domains into single universal space. In this paper, image-text datasets were used to validate our method since vision and language among the most actively studied fields in deep learning. Although have experimented with Aision-languae multimodal datasets only, the proposed UniCLIP framework can be easily extended to other types multimodal datasets because designed in a aggnstt way except for the augmentation encoding part. All poddllty-specciff knowledge required to apply UniCLIP to different types of modality to describe each modality-specific augmentation as real vector, as in Section 2.1, which is quite straightforward. We leave it for future work to see how well the UniCLIP framework works with various types of multimodal datasets.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea (oVeMMenn(MSITT) (No. 2022-0-00184 Development and Study of Al Technologies to Inexpensively Conform to Evolving Policy on Ethics)
References
L. Bossard, M. Guillaumin, and L. V. Gool. Food-101-mint discriminative components with random forests. European conference on computer vision, pages 446 461. Springer. 2014. 7
[2] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558 -568,, 2021."

---

[3] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 2, 4. 6
[4] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021. 2
[5] X. Chen, H. Fang. T.Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 150.000325, 2015. 7
[6] Cimpoi, S. Maji, Kokkinos, Mohamed, and A. Vedaldi. Describing textures the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3613,, 2014.
[7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner. M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 4
[8] I.. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and puttern recognition workshop, pages 178-178. 178. IEEE, 2004. 7
[9] C. Jía, Y. Yang. Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Le,, visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages
Le, Y-H. Sung. Z. Li, and T. Duerig, Scaling up
PMLR, 2021." 1
[10] P. Khosla, P. Teterwak, C. Wang. A. Sarna, Y. Tian, Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 2020. 2, 5,9
[11] Krause, M. Stark, Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization, In Proceedings of the JEEE international conference on computer vision workshops, pages 354-561, 2013. 7
[12] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[13] Y. Li, F. Liang. L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations. 2022. 2,2,7,7 7,9
[14] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint /wx.xxv:333../// 2013.7
[15] A. Micch, J. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman. End-to-end lcarning of visual representations from uncurated Instructional vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879-9889, 2020. 2, 4,
[16] N. Mu, A. Kirillov, D. Wagner, and S. Xie, Slip: Self-supervision meets language-image pre- pre-training. arXiv preprint arXiv:2112.1 /2750, 2021. 1, 8,9
[17] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, IEEE, 2008.
[18] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012. 7
[19] B. A. Plummer, L. Wang, C. M. Cervantes, C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-lo-phee to-phrase correspondences for richer image-to-sentencc models. In Proceedings of the IEEE international conference computer vision, pages 2641-2649, 2015. 7
[20] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021. 1, 8
[21] O. Russakovsky, J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bersstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25) 2015. 7
[22] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unitied embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015, 4
10

---

[23] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed. image alt- text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2556-2565, 2018. 7
[24] K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances neural information processing systems, 29, 2016.
[25] B. Thomee, D. A. Shamma, Friedland, Elizalde, Ni, D. Poland, D. Borth, and L.-J. Li. 1fcm The new data in multimedia research. Communications of the ACM, 59(2):64 73, 2016.
[26] A. Van den Oord, Y. and O. Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints, pages arXiv-1807, 2018. 4
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[28] J. Wei and K. Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNI) pages 6382-6388, 2019.
[29] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva. Sun database: Exploring large collection of scene categories, International Journal of Computer Vision, 119(1):3 22, 2016.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions nceded to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "nQGv7DU1wBH", "reviewer": "Reviewer_q238", "summary": "This paper propose a framework called UniCLIP to integrate the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space for Vision-Language Contrastive Pre-training. This work is more like a fusion of three optimizations than other work: augmentation\u000212 aware feature embedding, (2) MP-NCE loss, and (3) domain dependent similarity measure.\nThe results of UniCLIP performs better than previous work.  ", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nGood performance and simple\nThis paper is will written and organized.\nWeaknesses:\nLack of innovation, this paper is more like a combinatorial work, (2) MP-NCE loss, and (3) domain dependent similarity measure These two are not original to this paper, and (1) data augmentation is also a very common technique.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "dAMNu5Hg9Kf", "reviewer": "Reviewer_rSQy", "summary": "The paper proposes UniCLIP (Unified Framework for Contrastive Language–Image Pre-training) to unify inter- and intra-modality contrastive learning under a joint pretraining objective. The main difference of UniCLIP from prior work is that UniCLIP considers all inter- and intra-modal positives/negatives in the contrastive loss. Specifically, as an example, when computing loss for a text-image pair, image-image and text-text negative pairs are also considered. A dedicated multi-positive NCE loss is used to highlight the gradient of hard positives, which is in theory and empirically better than other options such as MIL-NCE loss and Supervised Contrastive Loss. Another special design is having a separate augmentation encoder inserting augmentation information in between the image encoder and image projection head so that the image encoder is agnostic to data augmentations while augmentation’s change to image semantics is also taken into consideration in the projected embedding space.\n", "questions": "* For the baselines such as SLIP and DeCLIP, did you implement your own versions and train your own models? Sorry if I missed this detail.\n\n* Since the pretraining dataset is different from the original configuration of DeCLIP, the numbers reported in the original paper are not directly comparable to this paper. I find it a bit puzzling that in some recent reports (https://arxiv.org/pdf/2203.05796.pdf), DeCLIP outperforms SLIP by large margins while in this paper, they perform similarly. Any comments on this?\n", "limitations": "I do not see any major issues.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrenghts\n\n+ The paper has done a comprehensive and rigorous discussion of available contrastive loss functions. The explanations are clear in pointing out each specific configuration’s impact in the context of image-text contrastive learning.\n\n+ Zero-shot classification performance seems to be strong and improvement is consistent across most of the 11 tasks.\n\n\nWeaknesses\n\n- Why not compare finetuning results on image-text retrieval? Zero-shot performance is not entirely reliable for comparing cross-modal retrieval capability. In my experience, sometimes finetuning even with few-shot examples can result in a huge difference.\n\n- I find the idea of learning discriminative features by leveraging positives/negatives both within and across modalities not drastically different from prior works. Beyond vision-language pretraining, the choice of positives/negatives, and what to include/exclude is comprehensively explored in metric learning or earlier data mining literature. It is good to know that such design works well for vision-language pretraining, but it is not an exceptionally novel contribution.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O5hqPzGBvm1", "reviewer": "Reviewer_wcmt", "summary": "In this work, the authors propose three components for contrastive learning across modalities which aim to remove discrepancies associated with learning representations with data augmentations, \"hard\" and \"easy\" positive pairs, and differences in similarity scoring across domains. \n\nThe first contribution is augmentation-aware feature embedding which concatenates an augmentation feature vector to image encodings before being fed through a projection head. This augmentation feature vector is used in order to constrain the image encoder to output representations that are \"augmentation-agnostic\" while allowing the projection head to make use of augmentation information. \n\nThe second contribution is a modified NCE loss (MP-NCE) which includes multiple positives and also uses a \"strong positive\" pair of an item with itself. Additionally, a rebalancing parameter is utilized to ensure that losses are equally weighted across all combinations of domains. \n\nThe third contribution is a domain dependent similarity measure which reweights similarly scores by introducing separate thresholds $b_{i,j}$  and temperature parameters $\\tau_{i,j}$ for each pair of domains $i,j$.", "questions": "Please see Weakness 1 and 2", "limitations": "The authors have not adequately addressed the limitations and potential negative societal impact of their work and should do so in a final version of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and provides clear descriptions of each of the contributions as well as experimental results on well-studied datasets/benchmark models. \n\n- Each of the contributions is well motivated by shortcomings in existing contrastive models/losses.\n\n- Experimental results show clear improvements over recent works (CLIP, SLIP, DeCLIP)\n\n- The experiment settings and hyperparameters are clearly outlined in Appendix Section C\n\nWeaknesses:\n\n- Exposition and presentation of each of the contributions ability to solve the problems they address is somewhat limited although proposed components are shown to be beneficial through ablation studies.. Some examples of the augmentation aware feature embedding is provided in Appendix Section E but it is unclear how the MP-NCE and the domain dependent similarity measure change the representations learned and their similarities. It would be helpful to understand how distribution of similarities changes with these mechanisms. \n\n- Ethical considerations and limitations is not discusses, please consider including information on this material\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GBLQvRu10y-", "reviewer": "Reviewer_xbYy", "summary": "The paper proposes learning a CLIP model in a single representation space to overcome the limitations of previous work on independent learning of each self-supervised objectives. The model, which they call UniCLIP, integrates the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space. The authors utilize three methods to address the discrepancy between different domains: augmentatio-aware feature embedding, an MP-NCE loss, and a domain-dependent similarity measure. \n\n", "questions": "1) A detailed discussion of limitations is needed.\n2) Need experimental validation of the claim that UniCLIP can be used in various types of multimodal datasets. How can your claim be justified with such experiments?\n", "limitations": "The authors do not properly address the limitations of their work. I encourage the author to directly address that in their rebuttal.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) The paper is well-written.\n(2) Experimental settings are described in detail, aiding reproducibility.\n(3) The proposed method leads to significant improvements on zero-shot, linear probing and retrieval test.\n(4) Strong ablation studies.\n\nWeaknesses:\n(1) This work builds incrementally on previous works. \n(2) Experiments on a wider range of tasks are encouraged, e.g., image captioning and visual question answering, as the method claims to learn representations in a unified space, improvements on visual-language tasks are expected.\n(3) Lack of proper discussion of limitations.\n(4) Some of the claims are not backed by experiments, for instance, the claim that UniCLIP can be used in various types of multimodal datasets.\n(5) The paper has several typos (e.g., \"datsets\")", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept