Analyzing the Impact of Learnable Softmax Temperature in Contrastive Visual-Textual Alignment Systems: Benefits, Drawbacks, and Alternative Approaches
Authors: Zhun Sun
Chao Li
Venue: Accepted by TMLR ()

Abstract: This work does NOT read like “fabricate motivation - propose something - obtain sota results”. Instead, we provide an in-depth analysis of the learnable softmax temperature parameter in the practical training of contrastive visual-textual alignment models, commonly known as CLIP models. This parameter is critical for optimal system performance, yet its mechanism and potential drawbacks have been largely overlooked. Our study addresses this gap and proposes a novel solution by utilizing the architecture of Vision Transformers (ViTs). We focus on the crucial role of the softmax temperature in managing noisy training data. We demonstrate that there is a balance in the gradient of the contrastive loss, with the temperature parameter acting as a distance scaling factor. If not properly calibrated, the model struggles to align positive pairs due to numerical issues in the loss term. Conversely, a high temperature can lead to unstable learning dynamics. We explore alternative approaches to mitigate this problem from a topological perspective of the contrastive loss. Ultimately, we leverage multiple class tokens embedded within the transformer architecture to present a concise solution. This configuration significantly enhances zero-shot classification performance, improving baseline CLIP models pretrained on large-scale datasets by an average of 6.1%.

Published in Transactions on Machine Learning Research (10/2024)
Analyzing the Impact of Learnable Softmax Temperature in Contrastive TixtullTextual Alignment Systems: Benefits, Drawbacks, and Alternative Approaches
Zhun Sun Independent Rescorcher
zhunsun@gmail.com
Chao Li
chao. listrike..jjpp
RIKEN Center for Advanced Intelligence Project (RIKEN-AIE AIP),
Tokyo, Japan
Reviewed OpenReview: https: openrevieww net/ forum? id=ro1(NhsNst
https:://ppenreviaaw
Abstract
This work does NOT read like "fabricate motivation propose something obtain sota results". Instead, we provide an in-depth analysis of the learnable softmax lemperature pa- rameter in the practical training of contrastive visual-textual aligmment models, commonly known as CLIP models. This parameter is critical for optimal system performance, yet its mechanism and potential drawbacks have been largely overlooked. Our study addresses this gap and proposes a novel solution by utilizing the architecture of Vision Transform- ers (ViTss). We focus the crucial role of the softmax temperature in managing noisy training data. We demonstrate that there balance in the gradient of the contrastive loss, with the temperature parameter acting erly calibrated, the model struggles to align positive pairs due to numerical issues in the loss term. Conversely, a high lemperature can lead to unstable learning dynamics. We explore alternative approaches to mitigate this problem from a topological perspective of the contrastive loss. Ultimately, we leverage multiple class tokens embedded within the transformer architecture present a concise solution. This configuration significantly en- hances zero-shot classification performance, improving baseline CLIP models pretrained large-scale datasets by average of 6.1%. The codes and learned weights are provided in attps://githur cow/ninogame/ mllpam/me//lppooobbb
distance scaling factor. If not prop-
1 Introduction
Learning visual and textual feature representations that are semantically aligned in their embedding space is an ordinary problem in the vision-language cross-modal tasks (Frome et al. 2013 Karpathy & Fei-Feil 2015 Romera-Paredes & Torr 2015: Wang et al. 2016 Faghri et al. 2017 Xian et al. 2016). In early works that employ feature representations from deep neural networks, e.g. Frome al. (2013), the alignment, is often achieved by fundamental metric learning approach with the hinge rank loss. That is, the similarity between visual feature vector and textual feature vector is calculated w"WWo, where W are the learnable weight parameters. Thanks to significant advances in computational power,
now more effective and practical approach called contrastive learning. This method aligns positive sample pairs and simultaneously pushes negative samples apart in large mini-batches using the InfoNCE loss [Rafford
et al. 2021 Singh et al. 2022 Jia el. al. 2021 Pham et al. 2021 Yuan et al. 2021)
We first review the contrastive visual-textual alignment system. Given set of semantically related image- pairs {UU, V)1 (U, V)2...., (U, V)k), where (U, V) is pair of an image with tokenized text that considered to be semantic related. The goal is to learn a pair of encoders, simultaneously:

---

Published in Transactions on Machine Learning Research (10/2024)
U g V + v to map the image and text into embedding space, are called embedding
vectors of samples. Following the definition in Oord et al. (2018); Wang & Isola (2020) ; Chen et al. (2011a)
Radford et al. (2021), the contrastive loss could be formulated as
-d((()).(V))))
Lo(f.gTT,S) =
log
(1)
U.V~S
97
U, #U
V *V
where I is the temperature term. -(:,-) is the distance function between two embedding vectors, and
92 :
+ Σ --((((()
JE[M]
EE|M
is the negative term, with M E z+ denotes a fixed number of negative samples. The subscript i, means the image or text chosen from the j-h pairs. Intuitively, optimizing this loss term minimizes the distance between positive imago-text pairs and maximizes the distance between negative image-text pairs. It is worth mentioning that, in recent studies (Radford et al. 2021 Chen al.) 2021b), the contrastive loss is commonly implemented as the cross-entropy between one-hot labels and the class probability obtained by softmax within mini-batch SM. We also employ this implementation in this work shown in Section
In Equation [ the standard choice of the distance measure between an image-text pair for the contrastive learning algorithm is the Cosine Similarity, in both uni-modal (Chen et al. 2020a Caron et. al.) 2020
Chen et al. 2020b) and cross-modal (Radford et al. 2021 Jia et al.) 2021 Singh et al., 2022) scenarios. While this choice provides a solid foundation for feature representation vectors from different sources, is widely acknowledged that. training such configuration is challenging in the absence of learnable softmax temperature. This learnable temperature is prepended and continuously updated through gradient descent,
along with the training progress in practice (Wu et al. 2018 Radford et al. 2021
In this work, we conduct a comprehensive analysis of this learnable lemperature parameter based on the following steps. Our primary contribution is highlighted at the beginning each step.
1. We show that the temperature is learned achieve the numerical "equilibrium" of the con- trastive loss. In practice, we could not perfectly minimize ainimize/mmximitni maximizee the distance between positive negative imagc-text pairs due to the ambiguity (noise) that exists in S. However, the contrastive alignment system finds numerical 'equilibrium' where the loss from noisy samples is balanced, minimizing the positive dis- tance (alignment loss) and maximizing the negative distance (uniformity loss). In Section 2.1 we give a visualization of the ratio of the two losses under different temperature conditions. We find that both a larger batch size and more noise data require a large temperature to reach "cquilibrium".
2. We suggest. three conditions for contrastive alignment. to achieve "equilibrium", and the temperature acts as scaling factor for the numerical value of the distance. In Section [.2] we analyze the properties of the contrastive loss from a topological perspective and outline three prerequisites: 1. A proper definition of uniformity of the embedding space for samples; 2. A "relaxed" triangular inequality between sample distances; 3. An expansive numerical value for the distance. We show that the Cosine Similarity calculates the inner product value between normalized feature representation vectors, resulting in a similarity ranging from -1 to 1. Hence, the system requires a parameter to expand this range. Next in Section 2.3 we conduct a toy experiment to demonstrate the necessity of these conditions by varying topology and distance combinations.
3. We show that temperature parameter with biased initialization or large learned value can degrade training. Subsequently, in Section 2.4, we visualize the temperature learning curve, revealing that the learned temperature indicates the dataset's noise level. A biased initialization that doesn't match the dataset can impair performance. We also visualize the model's loss landscape concerning temperature changes, showing that high temperatures lead to inferior learning dynamics.
4. We develop a practical system to mitigate the effects of the temperature parameter us- ing the vision transformer's innate structure. In Section 2.3 we propose using product spherical
2

---

Published in Transactions on Machine Learning Research (10/2024)
Shuffle 0%
Shuffle = 10%
Shuffle = 20%
Ratio of Alignment to Unifomity
batch sire 64
batch size 236
batch size 4096
Equilbrium
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
25
50
75
100
12s
150
175
200
Temperature
Temperature
Temperature
Figure 1: The ratio of aligmment loss to uniformity loss under different temperatures. The "equilibrium" is presented as the ratio equals 1.0, when the log loss in (1. becomes 0.0. Shuffle P% standards for percent of the labels are shufiled.
embedding space with inner product distance an alternative. We also explore leveraging the vision trans- former's structure, introducing multiple class tokens to create product spherical embedding space, which substantially boosts system performance (Section 3). In a comprchensive large-scale experiment, we trained ViT-B/16 (TT-BB16-basse 16-based CLIP model that surpassed the baseline model in classification and retrieval tasks (Sec- tion 4
2 Rethinking the Contrastive Alignment
In this section, we delve into our motivation in detail. First, provide a visual representation of the equilibrium. Then, we discuss the conditions (inherent properties) necessary for the contrastive loss to reach this equilibrium. Next, we design a toy experiment with controlled configurations to demonstrate the effects of these properties. To generalize the problem, we conceptualize the embodding vectors as points within specified lopologies, using the term "distance between the points" instead of similarity. We assert that contrastively " "well-aptimized" visual-textual model should yield shorter distances between semantically related image-text pairs compared to unrelated pairs, regardless of how the pairs labeled. Finally, discuss the potential drawbacks of the temperature parameter.
2.1 The visualized equilibrium
In Figure 1 we give an intuitionistic presentation of the equilibrium with our learned reference model Section As explained by Wang Isola (2020), contrastive loss is combination of two objects: a) alignment of features from positive sample pairs and b) the distribution of the features encouraged to match.
To demonstrate this, we randomly select mini-batch of samples from our training dataset in varying sizes. We then manually introduce noise by shuffling a small percentage of the images' labels (paired texts). Finally, we compute the ratio of aligmment loss to uniformity loss (numerator and denominator in Equation 4 under different temperature conditions, averaging across all positive pairs in the mini-batch. Our findings indicate that a noisier mini-batch or larger mini-batch size requires a higher temperature to achieve loss equilibrium. Additionally, when the system operates at a lower temperature, tends to push negative pairs away rather than pull positive pairs closer. Without the temperature adjustment, the alignment loss becomes numerically too small for the system to effectively align positive pairs. In the next subsection, discuss the mechanism behind this observation, focusing on three often overlooked conditions.
2.2 Conditions to achieve the equilibrium in practical system
In this subsection, we outline three critical conditions necessary for achieving equilibrium in contrastive learning. These conditions are partly inspired by the properties of measure spaces, but more importantly,
3

---

Published in Transactions on Machine Learning Research (10/2024)
False Negative Pair
ui
122
True Negaliva, Pair
Figure 2: Illustration of the bounded negative distance between true negative pair of samples.
they address the question: we build a system without temperature, what is essential for ensuring that the system generalizes well? (See the settings in Section 2.3 The conditions focus on i) Embedding space (the set of embedding vectors); ii) Properties of the map (the characteristics of the function that maps pairs of embeddings to similarity values); iii) Range of similarity values (the similarity values produced by the map should be useful for learning). While missing one or more of these conditions does not prevent the system from being trained, it may load to system that is far from equilibrium, potentially resulting in poor generalization performance. Below, we discuss these conditions in detail.
2.2.1 Proper definition of uniformity of the embedding space
In this subsection, explain why cannot use le loss in the Euclidean space. Naturally, with the loss form defined in Equation [ the distribution object will result in uniform distribution on the sphere. Although the distribution of samples doesn't have to be exactly "uniform" " as discovered by Chen et al. (2021a), it is necessary to define a proper prior distribution for samples to match via optimal transport algorithms (e-g- sliced Wasserstein distance), which undoubtedly a computational burden. Consequently, the sphorical embodding space is doemed the most suitable topology for contrastive alignment, as it exhibits proper uniform distribution defined by the surface area measure. In contrast, the commonly adopted unbounded Euclidean space lacks this property.
2.2.2 "Relaxed" triangular inequality
In this subsection, we explain why we are encouraged to use a non-measure distance. Since the model is being optimized by the contrastive loss, is proposed to decrease the distance between labeled positive pairs and increase that between labeled negative pairs. To examine how the noisy (false) positive pairs influence the model (even using human-labeled datasots, see Chun et al. (2022), we assume that we have model that is "well-ottimzeed" using noisy dataset. For this model, we have the following properties: For positive pair (U,Ve V)+, their distance d(u", v") is upper-bounded by small et, and the distance for negative pairs (U, v)- is lower-bounded by large E".
Now, let us consider a set of two pairs of its training samples S = (U1, V1), (U2, V2)). Accidentally, the pair (U1, V2)- also semantically correlated, despite being recognized as a negative sample (Tt is pervasive have negative pairs of image and text that match each other equally well as the positive ones in the noisy dataset). If the distance function d is metric, then according to the triangle inequality axiom of metric, we have the following inequality (see Fig. 2 for an intuitive illustration),
é S d(u vi) < d(ui, vi) + d(ug. vz)) d(uj, vi) s 3c*
From this simple derivation, even in this "well-optimized"" model, still it will predict a distance apper-bounder by tt for this pair instead of a larger value than c Here, we have contradiction: the system would not learn numerical separated distance ranges for positive and negative pairs that minimize the contrastive loss, hence cannot achieve "well-optimized" model.
4

---

Published in Transactions on Machine Learning Research (10/2024)
Topology
Sphere Sd-1
Euclidean Rdd
PS(d/m, m)
PS(d/m,m)
Distance
u
u v undefined
Geo(u, v)
- ttr(u²v)
Uniformity
surface measure
surface measure
surface measure
Inequality
relaxed
restricted
restricted
relaxed
Distance Range
[-1, 1]
[0, +00)
[0, m0,
m, m
Table 1: Summary of different topologies endowed with different distances. The total dimension of the embedding vector is denoted as d. The mini-batch size is denoted as b. Green box stands for the prop- erties that are favored for contrastive learning. Red box stands for the properties that are unfavored for contrastive learning.
Because the models are not "well-optimized", we have the following observations (See Figure in the sup- plementary materials). First, the positive pairs of samples (Green bins) usually have a much larger distance than that in the perfect alignment condition. For instance, the distance between the positive pairs of a learned Euelidean-22 is around -7, while the perfect alignment condition means the distance should be O. Second, although the contrastive loss term does not regularize the distance between negative pairs of samples in the same modality (Gray/Yellow Yellow bins), this distance implicitly minimized due the strict triangular inequality. For instance, the distance negative text pairs distribute closer to the negative image-text pairs. Finally, in the inner product distance configuration, it only obeys "relaxed" triangular inequality yielded by ArcCos (the geodesic, see (Scuberrt (2021) ) for more information.) on the sphere. This makes the distance between positive/me iualaiiv/ pairs slightly more separated, especially in the out-domain scenario (Green/Rod bins in the lower part). From these observations, we find one solution is to have a "relaxed" triangular incquality to alleviate the ambiguity of the between negative pairs not strictly upper-bounded by that of positive pairs.
pairs. This property could make the distance
2.2.3 Broad distance range
In this subsection, we discuss the mechanism behind the learnable temperature trick given the above two constraints. As discussed in Section 2.1 the contrastive loss is minimized through the gradients of both alignment and uniformily losses. In noisy dalaset inducing "semantic ambiguity"; the faase negative samples are pushed away from each other (repulsion), while the false positive samples are pulled together (attraction). Idcally, the system could gradually find numerical "cquilibrium" (the stabilized state) when the noisy samples' gradients for attraction and repulsion are equal. For instance, if there is reasonable amount of false negative samples, the model would learn a smaller distance between the pairs labeled as "negative", such that the uniformity loss won't 't be too high when encountering false negative samples in mini- batches. On the contrary, the model would learn a larger distance between the pairs labeled as "positive" when the amount of false positive samples is inneglectable. In other words, the model reaches the equilibrium by learning compromised positive and negative distances under semantic ambiguity.
However, the required numerical value of the compromised distances is out of the range that the cosine similarity could provide. This is generally because the InfoNCE loss computes the sum of exponential functions. To reach this equilibrium, we need to expand the distance range to [-T,7], and since we don't 't know the noisy level of the dataset, we set the temperature learnable through gradients. For instance, the officially released CLIP model (Radford et al.) 2021) has a glancing similarity of 0.3 ~ 0.5 and 0.1 0.3 for positive and negative pairs of samples, respectively. The learned temperature approaching 100.0, indicating the value of distances for equilibrium are 30- 50 and 10 30 for positive and negative pairs of samples, respectively.
5

---

Published in Transactions on Machine Learning Research (10/2024)
Zero-Shot 12T R@1
Zero-Shot T21 R@1
Zero-Shot: Cls. Acc.
Linear PrPSe
Topology
Distance
Cls. Acc.
Temperature:
init=1.0, gradient True
Sphere Eue PS(64, 8) PS(64, 8)
-u'r
49.0
30.33
28.59
59.56
u - u112
47.4
30.71
29.85
60.09
Geo(u, v)
49.9
32.49
30.21
60.61
-tr("" v)
50.9
32.71
30.50
60.66
Temperature:
mit==1.0, gradient- False (No temperature)
Sphere Fuc PS(64, 8)
- u v
5.1
3.461
4.04
45.37
u - 11/2
47.6
30.43
29.51
59.20
Gco v)
4.1
2.921
3.10
21.67
PS(64, 8) -tr(u"
30.3
18.48
21.20
57.93
Table 2: The retrieval and classification performance. "gradient=(TrueFFal FFalse)" donates if the temperature is learnable. The configurations are chosen intendedly to support the conclusions in Table 1.
The results shown in this table do not claim some configurations are better than others.
2.3 The toy experiment
Experiment settings: We design a toy experiment to demonstrate how the properties mentioned above influence the performance of contrastive learning. We employ the 15M subset (Cui al. 2022) of the YFCC100M dataset (Thomee et al. 2016) the training dataset, which contains roughly 15.3 million internet collected weakly related image-lext pairs. We evaluate the learned models with the zero-shot retrieval performance on Flickr30K, and Zero-Shot/Linear Probe classification performance on ImageNet for reference. We employ the original ViT-S/16 architocture for our image encoders (Dosovitskiy et al. 2020), with an input image resolution of 224, resulting in 196 image lokens.
We evaluate four types of configurations for comparison, each featuring a distinct combination of topology and distance function. The configurations are summarized in Table [ Specifically, we consider:
(i). The sphere gd-1 with the inner product as distance, which is the commonly used cosine similarity (Le the standard CLIP configuration).
(ii). The Euclidean space Rd with le distance, which is the most commonly used loss function in machine learning algorithms with an unbounded distance range.
(iii). A product spherical embedding space PS(n, m) with the minimizing geodesic as distance, which is denoted as Geo(u, v) = tr (arccos"("""")) We use this configuration to show the importance of properly defined uniformity. Although its distance range is still bounded, is sufficient for our system to reach equilibrium using the proposed dataset (See Figure 3).
(iv). The same product spherical embedding space PS(n, m) with the inner product as distance. We use this configuration to show the importance of the relaxed tri-angular incquality.
Here, we explain how we implement the product sphere embedding space. The product sphere (PS(n, m)) 5n-1 ST-1 where gn-1 is the sphere embedded in R", Intuitively, it is a vector
can be defined as ga-1 x..
copies
composed of m chunks of n nn-dimeniional normalized sub-parts. Our implementation follows this intuition: reshape the original feature vector into matrix of shape x n, then & norrmallize the columns. It is worth mentioning that, this study primarily investigates the characteristies of contrastive lcarning loss, proposing three distinct properties that the vanilla approach (cosine similarity) fails to satisfy concurrently. Therefore, we explore an approach that satisfies these three properties simultaneously. While the product sphere is not the only solution in this scenario, it could facilitate the analysis because I) It has the same definition of Uniformity: II) We can control the distance range by the number of sub-spheres design ablation experiments; TIT) It is straightforward to implement.

---

Published in Transactions on Machine Learning Research (10/2024)
We provide further clarification on the distance functions employed in these configurations. As previously discussed, the cosine similarity calculates the (negative) inner product for vectors on the unit sphere as their distance. Accordingly, we adopt this design for the product sphere. Hence, the distance could be computed as the negative value of the trace of the matrix product, i.e. d(u, v) = -tr(u" v). This distance is clearly not a restricted metric. Therefore, for reference, we also consider the minimiiing geodesic as distance, which is a restricted metric (obeys the triangular inequality).
Hyperparameters and results: For the product spherical embedding space, we employed structure of = 64, m 8, denoted as PS(64,8). Concerning the temperature parameter, we initialize it with exp(0.0) (oquivalent to 1.0) and conduct two sets of experiments, one with gradients (Icarnable) and the other without gradients (not learnable). It's important to note that the parameter is initialized to 1.0 and is not learnable, it is essentially equivalent to having no such parameter. All the results are presented within Table 2 and subsequent sections will elucidate the reader on how to interpret this tabular data. Results using other initialization (Figure are given in the supplementary materials.
Effects of the uniformity: To examine the effects of uniformity, we compare the performance between the Euclidean with &2 distance and the product sphere with geodesic distance. These configurations have restricted triangular incquality in common. Mcanwhile, if the temperature is learnable, the product sphere also owns a practically unbounded distance range similar to the Euclidean. Then, the only difference is that uniformity can be defined on the product sphere (surface area measure). We observe that the performance of the product sphere with geodesic configuration performs better than the Euclidean with &g configuration. This result indicates the importance of properly defined uniformity.
Next, we examine the effects of the triangular incquality using the product sphere topologies endowed with different distance functions, that is, the geodesic distance and the inner product distance. The results are visually depicted in Table specifically between the 3rd and 4th lines of each data block presented in the table. Upon observation, it becomes evident that the inner product distance outperforms the geodesic distance on average in both retrieval and linear probe tasks. Moreover, when the temperature is unlearnable, the inner product distance still provides the model trainability, showing the advantage of removing the restriction of tri-angular inequality.
Effects of the Tri-angular Incquality:
Effects of the Distance Range: Finally, we present the effects of the distance range by comparing the proposed product sphere topology with inner product distance and the baseline spherical topology. Notably, when the temperature is learnable, the product sphere demonstrates a reasonable improvement in the top-1 recall and classification accuracy. It is worth mentioning that this implementation docs not bring extra computational complexity, and the only difference is the shape of the embedding space. This suggests that a larger distance range helps the alignment of the features from different modalities. There one more piece of evidence that lies in the last block: the Euclidean with l2 distance configuration obtains consistent performance regardless of the temperature parameter, it operates with an unrestricted distance range.
2.4 Drawbacks in the learning dynamics
Biased initialization may degrade training: We now examine how the initialization of the temperature impacts the training progress. Specifically, we additionally run experiments with the temperature initialized at e2.6).))) which is the default value in the official CLIP implementation, and notably higher value of exp(5.3) (~200).. The changes in the temperature during the training process are illustrated in Figure || The figures confirm that: The temperature converges to an equilibrium value irrespective of initialization when the uniformity is properly defined. This value reflects the noise level present in the datasets, configurations with "relaxed" triangular inequality and a broader distance range have lower equilibrium values. These findings suggest that, if the training insufficient or the dataset too noisy, then with a biased initialization of the temperature parameter, the training progress might be degraded.
Higher temperature hampers the optimization: We next examine the impact of temperature on the loss landscape and the norm of gradient, using the approach proposed in Li et al. (0188). We employ the
The detailed retrieval and classification results are provided in the supplementary material. The final performance is not largely impacted by the initialization ion though.
7

---

Published in Transactions on Machine Learning Research (10/2024)
20000
40000
60000
80000
100000
20000
40000
60000
80000
100000
sphere + ,,,>
euclidean + 12
4-
0-
20000
40000
60000
80000
100000
20000
40000
60000
80000
100000
PS geodesics
PS + <,
Figure 3: The learning curve of the temperature. -,,>, 12 and geodesics denote the negative inner product, l2, and geodesic distance, respectively. The orange, blue, and green curves denote the initialization of exp(5.3), exp(2.64), and 1.0, respectively.
Loss Landscape temperature
Loss Landscape at temperature
Gradient Norm
Weight Norm
000
Weight
Thmppo
-2.0
-25
-00
Norm
Weightt Norm
1000
-15
-30
à We gint Norm
-33
0.8
23
5.
75
(b)
(c)
Figure 4: The impact of temperature on the loss landscape. (a) and (b) depict the contours with two orthogonal weights A in random directions at temperatures of 1.0 and 100.0, respectively. (c) presents the contour gradient norm with weight A in random direction across various temperatures (dark color stands for larger norm).
learned reference model in Section to draw the figures, where the final learned lemperature is 100.0 (due to the "clamp" function). We randomly select a mini-batch of size 1024 from the training dataset. The results are shown Figure 4 From figures (a) and (b) we can observe that the model with a lower temperature has much smoother loss landscape. Furthermore, in Figure (c) we find that the norm of the gradient might become greater the temperature grows. Therefore, in practice, we need to employ techniques such gradient norm clipping stabilize the training.
3 The Multiple Class Tokens Solution
We first review the importance of the class token. In the design of both the textual transformer (BERT, Ken- (on & Toutanova (2019)) and the visual transformer (VIT, Dosovitskiy et al. (2020)), one learnable embed- ding is used to represent global information, termed as class token. Different from the sequence (patch) tokens, the class token is key component of the transformer encoder. It is randomly initialized and up- dated through gradient descent during the optimization. Furthermore, the class token holds a fixed position embedding, avoiding the influence of the positional information. Therefore, the class token is considered to participate the computation global attention.
Motivated by the properties of the class token, we propose employ multiple class tokens to build the product spherical embedding space, with each class token being sub-sphere of 5=-1, For the visual encoder,
8

---

Published in Transactions on Machine Learning Research (10/2024)
Text
Image
Image
Encoder
Encoder
Encoder
Encoder
[CLS]
[CLS] Token
Dot Product
[CLS]
[CLS] Tokens
Flatten Dot Product
Token
Tokens
(u, v)
d(u, v)
Sequence Tokens
Sequence Tokens
Projection
Projection
(a) Vanilla CLIP
(b) CLIP with Multiple Class Tokens
Figure 5: A sketch of the contrastive visual-textual alignment (CLIP) system.
IN zs cls.
INV2
IN-A
IN-R
Flickr30K Zero-shot
MSCOCO Zero-shot
Method
ZS cls.
ZS cls.
ZS cls.
12T
T21
Mean
12T
T21
Mean
baseline kmpl.]
Acc@1
Aco@1
Accã1
Acc@1
R@l
R@1
R@1/5/10
R@1
R@1
R@1/5/10
VTT-B/16-22/ visual bone.
68.7
61.9
50.1
77.7
81.9
62.1
86.1
55.4
38.4
66.3
CLIP [openCLirt]
67.0
59.6
33.2
77.9
83.2
65.5
87.6
52.4
38.4
62.4
CLIP [ou-ip.
69.5
61.4
49.5
70.6
84.2
61.7
86.4
64.1
43.9
72.4
CLIP(Mull((32.16) 76.4
68.0
55.8
75.2
85.2
66.3
88.3
63.8
42.9
72.4
VIT-L/14-224 visual bone for reference.
CLIP|openAI"] 75.5
69.7
70.7
87.9
85.0
65.2
87.7
56.3
36.5
65.2
CLIP(openCLiPt] 72.7 65.6
46.6
84.8
87.6
70.3
90.1
59.7
43.0
70.0
Table 3: Comparsion of large scale contrastive visual-textual pre-train model on benchmark datasets. I and denote the implementation from Radford et al. 2021 and Ilharco et al. (2021), respectively. The metric Mean stands for the average value of R@T 1 T2T /T2 retrieval performance. denotes the Karpathy test split Karpathyy & Fei-Fei 2015).
these class tokens are randomly initialized to break symmetry, while for the textual encoder, we use different absolute positional embeddings for each class token. We present sketch of the system in Figure Ei and pseudo-code in Section g to facilitate better understanding. Given the fact that the dimension for the
embedding space could be critical factor to the performance of the system (Gu et al. 2021), we select the dimension n with a conservative strategy. Specifically, we anchor the dimension of Euclidean spaces to be the same as the reference model, then vary the value n, m such that n X m = l. We denote this implementation as Multi(n, m). The sub-spheres could benefit from the global attention operation and provide more representative feature embeddings. On the contrary, the multi-token implementation requires more computational resources in the backbone since the class tokens are involved in the computation of global attention.
4 Large-Scale Experimental Results
4.1 Experimental Settings
Please note that our objective is not to produce the best publicly available model; rather, we solely conduct experiments under controlled and restricted conditions. We compare the performance of the proposed method using the configuration, which matches the publicly released ones in teams of dataset samples, model sizes, and training progress. We also re-implement the naive CLIP model as the reference, which holds similar performance the publicly released ones.
PIt needed clarify that in the multiple class tokens system, also employ the learnable temperature practically. Since the finally learned temperature significantly smaller than that the vanilla CLIP system (~ 4.0 versus 100.0), omit. it. highlight the difference between the systems.

---

Published in Transactions on Machine Learning Research (10/2024)
dimensions of the hidden embedding
U, mini-batch of images/texta token, [n, P. d] [n, 1,
PS the dimension each sub-sphere
PS_n the number of [CLS] tokens attached
cls_ cls_V class tokens for images/texts, [n, PS PS_n, d] / [n, PS_n, d]
learned temperature parameter
concatenate cls_tokens and extract features
U_, V_ concatenate [[cls_, U, U], axis=1), concatenate ([clla,,ttt(c__,,,, v], axis=1)
u_ bar disual_transforrtt (U_) # #[n, PS_n P, d] v bar textual transformer (V_) #[n, PS_n 1,
map features onto (S(n,m) and calculate distance projection__uu_barr_
[n, PS_n, PS_m]
nl) .12_ ft___oomaaizeeeeeeeee (axis==1)) [n, PS_n, PS_m] [n, PS_n, PS_m], , [n, PS_n, PS_m] -> [n, n]
neg_ distances = einsum (' ('inm, jnm->ij', , u, v) ex)xp()
symmetric loss function
labels = (n) 0, 1, n-1 loss (CE, (negg labels, axis=0) + [B_loss (neg_ disgances,nees, labels, axis=1)) /
Figure 6: Python-like pseudo-code of the proposed approach.
Datasets: For the experimental analysis in Section 4.2
we collect data from publicly available datasets.
The majority of the imago-text comes from the 4AION400M(OPen Clip) datasct (Schuhmann et al. 2021 However, since the availability of this dataset decreases through Limee]] we also add the training splits of some
other well-known large-scale datasets, they are CC3M Changpinyo et al. (021), CC12M (Sharma et al.
2018
MSCOCO (Chen et al.] 2015), Visual Genome (Krishna et al., 20177). Flickr30k (Plummer et al.
2015 ImageNet (Russakovsky et al.) 2015), OpenImage (Kuznetsova et al. 2020 and WebVision (Li ct al.
2017 This dataset contains total of 420 million individual images and roughly 500 million image-text
pairs. This dataset is comparable to the one employed in the official CLIP paper (Radford et al. 2021 and
another open source re-implementation (Ilharco et al. 2021).
Models: Specifically, we employ the ViT-B/16 as our image encoders. For our text encoders, we employ Ernie-2.0cen-base (Sun et al. 2020), which Bert model (Devlin et al., 2018) of 12 layers and 512 hidden neuron sizes with customized vocabulary of 30,522 tokens, and the maximum context. length is set to be 77. We project the feature representation (class token) from the top layer of transformers to (sum of) 512-dimensional embedding space. All the parameters except the temperature are optimized from random initialization. The default initialization of the project matrix employs the Gaussian initializer of zero mean, and standard deviation equal reversed square root of the input size (a.k.a. Kaiming initialization). The details the hyperparameters are provided in supplementary materials.
Evaluation: We first evaluate the proposed methods with two types of vision tasks: i) Zero Shot image- to-text and text-to-image retrieval on Flickr30k (Plummer al. 2015) and MSCOCO (Lin et al. 2014) ii) Zero-Shot classification ImageNet-R (Hendrycks et al. 2021a and ImageNet-A (Hendrycks et al. 2021b). For zero-shot retrieval on Flickr30K and MSCOCO, we employ the logits (distance) computed by the distance function and report the image-text pairs with the top-k shortest distance as the retrieval results. For zero-shot classification on
ImageNet-1K (Russakovsky et al. 2015), ImageNet- V2 (Recht et al. 2019
ImageNet. We employ multiple prompt templates described in Radford et al. (2021), while we first compute
The availability T.TTON400M is about 90%.
10

---

Published in Transactions on Machine Learning Research (10/2024)
COCO
1K
COCO 5K
CxC
ECCV Caption
Method basetine[impt]]
12T
T21
12T
T21
12T
T21
12T
T2I
R@1
R
R@1
R@1
R@1
R@1
mAP@R
R-P
RØ1
mAP@R
R-P
R@1
VGT-B/16-222 visual bone.
CLIP [TT]
71.7
52.5
52.5
33.1
54.0
34.7
23.7
34.0
68.8
34.8
44.0
73.4
CLIP [ppenCTIPP
74.0
57.6
55.4
38.3
57.3
40.0
26.2
36.6
70.3
36.9
46.4
77.5
CLIP [our-impl]]
80.8
63.0
64.2
43.1
65.3
44.9
30.5
41.0
78.6
40.5
49.9
81.2
CLIP (31)
81.1
63.1
63.8
42.9
65.3
44.8
30.9
41.7
76.3
41.7
50.5
84.1
ViT-L/14-224 visual bone for reference.
CLIP(openAI"]
74.3
55.4
56.4
36.6
58.0
38.3
24.0
33.8
71.3
32.0
41.8
73.0
CLIP (ope(ope
77.2
61.4
59.7
43.0
61.1
44.8
28.1
38.3
73.0
38.7
17.9
81.2
Table 4: Comparsion of large scale contrastive visual-textual pre-train model on benchmark datasets.
the distances between image and text embeddings, then average the distances. For linear probe classification ImageNet, we remove the learned projection head (no topological structure is preserved). then attach random initialized linear projector to map the feature representation to the 1,000 class logits.
Besides the tasks mentioned above, we provide more results using the ECCV dataset (Chun et al. 2022 The dataset is proposed for eliminating the false negative samples in the validation set of the original MSCOCO dataset. Instead of the commonly used Recall (R@K) metric, the datasets provide a new ranking- based metric, mAP@R. The authors of the ECCV dataset have shown that the mAP@R metric more aligned to humans than Recall@k. Therefore, the performance of model evaluated by mAP@R would be less occasional than the R@1. This metric is deemed more precise for evaluating the performance of models in the presence of noise.
4.2 Experimental results
The evaluations of the learned models the commonly employed image classification and image-text re- trieval tasks are reported in Table 3 with the ViT-B/16 as the visual backbone. It can be seen that the re-implemented CLIP holds a similar performance as the publicly released ones in most cases. On the other hand, our proposed model with the multi-token implementation of (32,16) significantly outperforms the other ViT-B/16 models in general, with less than 8% more computational costs. The only exception is the top-1 retrieval performance on the MSCOCO datasets. The reason could be two-fold. Firstly, we observe a mild "semantic decoupling" between the embedding of tokens through the visualization (see Section 4.3) that is, some of the individual class tokens foeus on specified objects and provide high alignment confidence. This may confuse in understanding the given scene as whole; hence, the recall@top-1 performance is de- graded. Secondly, the most suitable temperature during training for aligning object-level and scene-level concopts might differ. In our experiment, we decrease the upper limitation of the temperature to 6.25 (100 / 16 [tokens]) since the product spherical topology owns the border distance range. The scene-level concept alignment might require larger temperature for "ambiguity" achieve better retrieval performance.
For the ECCV caption datasct, utilize the officially released evaluation tool and present summary of models' performance in Table Our proposed multi-token product sphere topology outperforms others in terms of mAP@R, signifying the model's enhanced robustness in handling "semantic ambiguity" samples.
4.3 Visualization Tokens Attention Regions
In this section, we provide commonly adopted neural network explanation method to visualize the influence of inputs on the final outcome. Specifically, we employ the Grad-CAM (Selvaraju et al.) (017) algorithm to highlight the interested parts by the model of both images and their corresponding texls. Notably, the original design of the Grad-CAM algorithm precludes its direct application to textual data. Therefore, we enhance its capabilities such that also highlights the contributing parts of texts in token-wise style. We employ examples from the evaluation set of the Flickr30K and MSCOCO datasets for visualization. The results shown in Figure Through observations, we have noted that certain pairs of class
11

---

Published in Transactions on Machine Learning Research (10/2024)
littl girl is jumping in green landscape. a litll jumplng green larescape.
little girl Limping ina green lindsapee little gill jumpinn in green landscape
Image- Text Pair
Full Tokens
Token id=01
Token id=15
Figure Visualization of the importance map using the Grad-CAM algorithm. The columns from left right stand for: the input image-texi pair; the importance map computed based on the final matching score; and the importance maps based on the matching scores of two individual tokens and the involved token IDs, Additational results are provided in the supplementary material (including the failed cases).
tokens exhibit independent alignment, thereby reflecting distinct concept idea embedded within the image. We attribute the improved classification performance to this phenomenon, it enables more effective representation and understanding of the underlying content by leveraging the distinctive alignment of the class token pairs. It is noteworthy that the intrinsic decoupling phenomenon observed within the embeddings of class tokens is NOT universally present across all image-text pairs or within every token an imago-text pair. This is due the inherent challenges facod by visualization algorithms in achicving procise correspondence in the importance maps for intricate semantic representations.
5 Related Works
Momentum distillation: In recent works such as Cheng et al. (2021): Li al. (2021a), the momentum delfl)dditilllation is introduced to mitigate the semantic noise in the sample pairs. That is, a momentum version of the model is updated by the moving average of the model's historical parameters. Then, the cross entropy between the softmax logils computed by the model and its momentum version is used as an additional loss for supervision. The authors claim that the pseudo-targes of the momentum selff)ddistillatoo will not penalize the model for matching negative samples that are reasonably similar. Here, we consider that the pseudo-targets do "relax" the triangular inequality restriction implicitly by letting the distance of alignment be reasonably large. Hence, it could be much easier for the optimizer to find the equilibrium discussed in Section
Other implemention of non-metric distance: In Yao et al. 2021) the authors proposed so-called fine-grained contrastive learning scheme that matches all the visual and textual tokens using a maximum- average operator. Concretely, for each visual token, it finds the textual token with maximum similarity, then takes the average over the visual tokens as the similarity of the image to a text and vice versa. Using our framework, this work can be explained as embedding samples onto the product manifold gd-1 ... gd-1 endowed with the maximum-avorage distance, which is a non-metric distance. At the same time, the authors employ the sub-manifold gd-1 to represent. local information.
The effects of softmax temperature: In Wang & Liu 2021). the authors draw the uniformity of the embedding distribution and the tolerance to semantically similar samples of learned models under different lemperatures. From the observations, the authors claim that "a good choice of lemperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualitics and the downstream performances". Unlike our work, this work done under uni-modal contrastive learning, where the semantic correlation of the negative samples not a property of the datasets but rather drawback of the larger mini-batch size.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Uni-modal side tasks: In works such as Mu et al. (2021): ): Li et al. (2021b); Yang et al 2022), authors combine cross-modal contrastive loss with other uni-modal tasks, for instance, visual/textua sell-supervise contrastive learning, masked image/language modeling, These combined methods empirically demonstrate superior performance downstream tasks such as zero-shot classification. Although these works do not over- lap with this one, we find that the uni-modal tasks provide reasonable uniformity within the visual/textun feature embedding, contrary to the cross-modal contrastive shown in Section 2 obtain more "numerically relaxed" triangular inequality when dealing with noisy pairs of samples.
Therefore, the model could
6 Conclusion
Summary: This work discusses the essential properties of the feature embedding space for contrastive alignment. We show that the most commonly adopted cosine similarity has disadvantages in dealing with noisy data and training stability. Therefore, we propose combine the product sphere with the negative inner product distance to tackle these problems. We employ multiple class tokens to implement the approach, which performs better in various zero-shot classification and image-text retrieval tasks practically.
First, given significantly constrained computational resources (and time), we acknowledge our inability to conduct experiments on a larger scale regarding batch size, training data, and neural network parameters. However, the reported results are robust enough to substantiate our claims. Second, in re- cent studies, besides the contrastive alignment, more pre-training tasks are appended to the head of the model using the non-oormalized full token embedding. Such as image-text matching (Li et al.] [021a Yang et al. 2022), image captioning YYu al., 2022), masked modeling that do not employ the contrastive alignment (Wang et al. 2022). The performance improvement resulting from better contrastive alignment could be marginal in these configurations. Hence leave future work on designing the model of the full token embedding.
Limitation:
References
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In Computer vision-ECCI 2014: 18th European conference, zurich, Switzerland, Seplember 6-12, 2014, proceedings, port VT 13, pp. 446-461. Springer, 2014.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual fcatures by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 9924, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEFE/CVE Conference Computer Vision and Pattern Recognition, pp. 3558 3568, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607, PMLR, 2020a.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Infor- mation Processing Systems, 34:11834 11845, 2021a.
Xinlei Chen, Hao Fang, Tsung-Y- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. aix preprinl SrXiv:1504.00333 2015.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. Xiv preprint arXiv:2003.0,20 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the TEEE/CV /CF International Conference on Computer Vision, pp. 9640-96499 9649, 2021b.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Ruizhe Cheng, Bichen Wu, Peizhao Zhang, Peter Vajda, and Joseph E Gonzalez. Data-efficient language- supervised zero-shot learning with self-distilaatin. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 3119 3124, 2021.
Sanghyuk Chun. Improved probabilistic image-text representations. arXiv preprint arXiv:2505.18171 2023.
Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang Chang, and Seong Joon Oh. Eeev caption: wwweeeanndd--a--veeeee
Correcting false negatives by collecting COCO. In European Conference on Computer Vision (ECCV), 2022.
image-caption associations for ms-
M. Cimpoi, S. Maji, 1. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive language- image pre-training: clip benchmark of data, model, and supervision, 2022.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint orXin:2111.11431, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint XrXi::1810.0480, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint Xrxiv:2010. 11929, 2020.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv: arXiv:1707.0500 707.05612, 2017.
Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-61) 611, 2006.
Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
Weiwei Gu, Adilya Tandon, Yoog-Yeol Ahn, and Filippo Radiechi. Principled approach to the selection of the embedding dimension of networks. Nature Communications, 12(1):3772, 2021.
Haochen Han, Kaiyao Miao, Qinghua Zheng, and Minnan Luo. Noisy correspondence learning with meta similarity correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognilion, pp. 7517 7517-7526, 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 2226, 2019.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial ex- amples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Processing Systems, 34: 29406 29419, 2021.
Gabriel Iharro,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. Zenodo, July 2021. doi: 2t.5281/zenodo.350 zenodo.5143777
URL https:///ok http::::///i.oo//1.....
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic aligmments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128 3137, 2015.
Jacob Devlin Ming- Wen Chang Kenton and Lee Kristina Toutanova. Bert: Pro-training of doep bidirectional transformers for language understanding. In Proceedings of naacI-HLT, pp. 4171 4186, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained catego- rization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554 561, 2013.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Tvan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Compuler Vision, (28(7)11956-198)) 2020.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Golmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation, Advances in neural information processing systems, 34:9694 9705, 2021a.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXio preprint orXiv: 028602862, 2017.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.
arXiv preprint arXiv:2110. 05208, 2021b.
Zheng Li, Caili Guo, Zerun Feng, Jenq-Neng Hwang, and Zhongtian Du. Integrating language guidance into image-text matching for correcting false negatives. IEEE Transactions on Multimedio, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arxie preprint 2017.
S. Maji, Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fino-grained visual classification of aircraft. Technical report, 2013.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-
image pre-training. arxiv preprint orXie:212.. 127500 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NTPS workshop deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sirth Indian conference on computer vision, graphies & image processing, pp. 722 729. IEEE, 2008
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive
coding. arXiv preprint urXiv:1807.0377 2018.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference compuler vision and pallern recognilion, pp. 3498 3505. IEEE, 2012.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le. Combined scaling for zero-shot transfer learning. arXiv preprint 1Xxx:2111.10000 2021.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebník. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641 2649, 2015.
Yang Qin, Dezhong Peng, Xi Peng, Xu Wang, and Peng Hu. Deep evidential learning with noisy correspon- dence for cross-modal retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 0948-4956, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Benjamin Recht,, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenct classifiers generalize to imagenet? In International conference on machine learning, pp. 5389 5400. PMLR, 2019.
Bernardino Pareres and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International conference on machine learning, pp. 2152 2161. PMLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (LJCV), 115(3):211 222:222,, 2015. doi: 10. 11.1007/s11263005--090
Erich Schubert. A triangle inequality for cosine similarity. In International Conference on Similarity Search and Applications, pp. 32-44. Springer, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaezmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of flitefittredd 400 million image-text pairs. arXiv preprint xrxi::2111.0211 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: cleanod, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lúnguisties (Volume 1: Long Papers), pp.
2018.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638 156500, 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical melhods in natural language processing, pp. 1631-1642, 2013.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German Traffic Sign Recognition Benchmark: multi-class classification competition. In IEEE International Joint Conference on Neural Networks, pp.
2011.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAT Conference Artificial Intellignnce, volume 34, pp. 8968 8975, 2020.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59 (2):64-73, 2016.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 2495 2504, 2021.
Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preservin image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5005-5013 2016.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pp. 19299-9939, PML.R 2020.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10, 10442, 2022.
Zhirong WI, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3733 3742, 2018.
Yonggin Xian, Zeynep Akala, Gauray Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 69-77, 2016.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Compuler Vision, 119:3 22, 2016.
Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2022.
Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Bicro: Noisy correspondence rectification for multi-modality data via bi-directional cross-modal similarity consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 19883-
19892, 2023.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training, aiXiv preprint arXiv:211107775 2021.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. ar preprint arXin:2205.01 015111111 917, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint ar xrx::21111. 11432 2021.
18

---

Published in Transactions on Machine Learning Research (10/2024)
A Detailed Training Manuscripts
Used in Experiments in the Main
Value for Naive CLIP
Value for CLIP-Multi(32,,
Value for Ablation Table
Value for Ablation Table 6
Hyperparameters
Batch size
32,768
32,768
2,048
2,048
Vocabulary size
30,522
30,522
30,522
30,522
Training epochs
32
32
15
15 2/8 2/8/33// 128
Number [CLS] Tokens
1/8
Projection dims
512
32
312/64
256 /
Maximum temperature
100.0
3.95
100.0
100.0
Weight decay
0.2
0.2
0.5
0.5
Warm-up iterations
2,000
2,000
5,000
5,000
Peak Learning Rate
0,0005
0.0005
0,0005
0,0005
Adam 31
0.9
0.9
0.9
0.9
Adam 82
0.998
0.998
0.98
0.98
Adam
10-3
10 8
10 8
10 8
Gradient global norm
1.0
1.0
1.0
1.0
GPUs
128xA100
A280
32xV100
32xV100
Train Time
~ days
5 days
~1 day
~1 day
Table 5: Detailed syper-parameters used for in the experimental analysis.
We provide the typer-parameters employed in the experiments in Table 5 hyper-parameters employed in the original CLIP (Radford et al., 2021) paper for both Naive CLIP re- implementation and our multi-token and single-token product sphere implementation. We provide the details experimereettteeeee experiments below.
We follow most of the
of the hyper-parameters for large-scale and ablation
Large-scale Experiments: We train with batch size of 32,768 and the AdamW optimizer (Loshchilov & Hutter 2017) in all the large-scale experiments. We apply the standard training scheme of the origi- nal CLIP model, which contains 32 epochs of training. We did not employ mixed precision to reduce the possible overflow introduced by randomness for a stable reproduction. We set the B1 = 0.9, B3 = 0.998,
c = 1e-8 in AdamW, and weight decay = 0.2 to further improve the stability. We use the cosine learning rate decay scheme of peak learning rate equal to 5e-4, combined with a warmup period of 2,000 iterations. For data augmentation, we only apply the RandomRes izedrrop with a scale range of [0.8, 1.0). Finally, our multi-token product sphere implementation, we reduced the maximun temperature 3.95 due to its border distance range. This is value obtained from the ablation study from Appendix E Ablation Experiments: We train with batch size of 2,048 and the AdamW optimizer (Loshchilov & Hutter 2017) in all the ablation experiments. We apply a compact training scheme that updates the model for 000 iterations, which is roughly equal to training the model for 15 epochs of the dataset. Since this is a fast training scheme, we set the 31 = 0.9, 32 = 0.98, c = 1e-8 in AdamW, and weight decay = 0.5, such that the training could converge faster a stable approach. We use the cosine learning rate decay scheme of peak learning rate equal 5e-4, combined with warmup period of 5,000 iterations. In the linear probe evaluation, the hyperparameters follow the setup of MoCo v3 (Chen et al. 2021b). Concretely, we use SGD without momentum and no weight decay. The learning rate is schemed by cosine decay with peak learning rate equal to 1.0, combined with a warmup period of 5 epochs. We train for 100 epochs and augment the image using the wandooResszzedCroo with a scale range of [0.75, 1.0] and AutoAugment with the code rand-m9-mas 1om00-mstd...0 teattdd.5-in...... 5-inc1. In the ablation experiments, we do not change the maximum temperature clip value, leaving it the same for all topology configurations.
B Choices on structures and multi-token implementation
In Table 6 we modify the structure of the product sphere manifold under fixed total dimensions. can be seen that a higher m value (i.e. the number of product sub-spheres) is more likely obtain better zero-shot classification accuracy and text-to-image retrieval recall. We, therefore, conjecture that the broader distance
19

---

Published in Transactions on Machine Learning Research (10/2024)
Zero-Shot 12T R@1
Zero-Shot
Zero-Shot Cls. Acc.
Linear PrPSe
Topology
Distance
T21 R@1
Cls. Acc.
Temperature, init=e" 2.64 gradient=
S5h
-u-
48.3
31.45
30.62
60.38
PS(256, 2)
-tr(u"v)
48.0
32.25
30.33
60.52
PS(64, 8)
vtr(u"v)
52.3
32.89
30.70
60.32
PS(16, 32)
-tr(u"v)
50.4
33.01
30.93
60.79
PS(4, 128)
-tr(""v)
48.2
32.91
30.57
59.99
M266 2)
-tr(uTv)
49.2
32.29
30.04
61.59
Mulci(64, 8)
-tr(u"v)
54.0
34.27
31.93
62.41
Multi(16, 32)
-tr(u"')
54.0
33.43
30.88
63.71
Table 6: The retrieval and classification performance of the proposed approach using different PSlique man- ifold structures and the multi-token implementation. "gradient= (rradeett(Trre/Falsss) Truu/False)] donates if the temperature is learnable.
range helps the system reach equilibrium faster. However, an over-complicated structure such as PS((4128) could ruin the performance. The possible reason is that each sub-sphere S that is embedded in Rd has one less effective dimension. Therefore, the product sphere structure with large numbers of sub-spheres may perform worse.
However, an over wer-cooplicated structure such as PS(4,128) could ruin the performance. We conjecture that, since the sphere has one redundant dimension, the larger number of product sub-spheres reduces the representation capacity of the topology. And because we employ a textual encoder that is gently larger than the visual one, therefore the moderate reduction of the capacity helps overcome the overfitting on the textual side.
We also provide the ablation results of different multi-tokens product sphere structure implementations in the table's lower half, denoted as Multi(,,-). We concatenate all the representations logether for the linear probe before projecting them to 1,000 class logit. can be seen that the multi-token product sphere implementations outperform their single-token versions most of the time. Notably, since the increased number of parameters for the class Lokens (n d) is negligible compared to that of the overall system, consider the participants of class tokens in global attention as the primary reason for the performance boost, compared to their single-token version.
In addition, we discuss the computational complexity of the multi-token implementation. First, we consider the cost the logit matrix of the product sphere. The logit matrix is used to record the distance between pairs of samples, which considered to be the major source of computational complexity when the batch size is large (e.g. 221). In our case, given a batch size the spherical configure uses b² float numbers while the Multi(n, m) structure uses 62m. Second, we provide information on the additional cost brought by additional class tokens. Since
the transformer structure, the flops (training/ bemiiiin//iffenn /nfferncee time) scale with the number of parameters lime numbers of tokens, plus the square of nunber of tokens we used. Concretely, given a model with (-layers, d-hidden dimension, and t-tokens, the total flops can be roughly estimated as x x (12 x d + +2x x d). The number of activations roughly t 1 x (36 x d d 6 x x d).
Overall, we should use a sufficient amount of sub-sphere to make the distance range able to cover the 'equilibrium' shown in Figure intuitively. However, we need additional computational resources that scale with the (square of) number of tokens. While there is no closed-form solution to the optimal number, we consider using 8 ~ 32 tokens to be practical choice.
C Detailed performance of configurations under different temperature initilization
In Table 7 we provide the detailed experimental results of Figure 2 in the main manuscripts. We further provide the final temperature at the end of training and at what step the temperature converges (changes
20

---

Published in Transactions on Machine Learning Research (10/2024)
Temp. Init.
Temp. Final
Converge Step
Zero-Shot 12T R@1
Zero-Shot T21 R@1
Zero-Shot Cls. Acc.
Linear Cls. Acc.
Topology:
Sphere,
Distance: -u"v
2.659
4.033
18k
48.3
31.45
30.62
60.38
5.310
4.021
39k
46.8
31.13
29.60
59.82
1.000
3.976
22k
49.0
30.33
28.59
59.56
1.000
1.000
Detach
5.1
3.461
4.04
45.37
Topology:
Euclidean,
Distance:
| v
2.659
2.067
21k
47.9
32.29
30.36
59.90
5.310
5.107
1k
48.5
32.69
30.68
59.74
1.000
1.668
25k
47.4
30.71
29.85
60.09
1.000
1.000
Detach
47.6
30.43
29.51
59.20
Topology:
PS(64, 8),
Distance: Geo( u, 1²)
2.659
3.135
20k
50.7
32.35
30.79
60.43
5.310
3.168
55k
50.7
31.59
30.34
59.60
1.000
3.024
42k
49.9
32.49
30.21
60.61
1.000
1.000
Detach
4.1
2.921
3.10
21.67
Topology:
PS(64, 8),
Distance: - -tr(2' ( (u'v)
2.659
2.231
24k
52.3
32.89
30.70
60.32
5.310
2.280
57k
50.3
33.37
30.23
59.76
1.000
2.174
36k
50.9
32.71
30.50
60.66
1.000
1.000
Detach
30.3
18.48
21.20
57.93
Table 7: The retrieval and classification performance of different configurations under different temperature initialization conditions. "Temp. Init." denotes the values for initializing temperature; "Temp. Final" denotes the final temperature at the end of training: "Converge Step" denotes the number of steps for temperature starts to converge (changes less than 2% for an epoch.)
less than 2% for epoch). It can be seen that the performance of the Euclidean topology is only slightly affected by the initialization of the temperatures, and even though the temperature is detached from learning, it still performs reasonably well because of the unlimited distance range. At the same time, the spherical and product sphere topologies are affected by how the temperature initialized. However, a rough trend can be seen that the faster the temperature converges, the better performance the model achieves, which means the learnable temperature delays the learning of the methods. The model needs first to find a proper temperature and then begin to learn representations well.
Distribution of Learned Distance
We depict the distribution of distance for pairs of samples in Figure 8 We further employ the RedCaps (Desal et al. 2021] dataset as the out-domain data for visualizing the distributions of sample distances. As argued in Section 3.2 of the main manuscripts, since the cross-mode rross-modall contrastive loss does not handle the uni-modal data distributions, the distance between negative pairs of images and texts could be much smaller than that of positive image-lext. pair, resulting in tighter distance bound. Also, we can this phenomenon is much more severe in out-domain data, which could reduce the transferability of the feature embeddings to downstream tasks. It is also notable that, the product sphere with the negative inner product the distance function learns similar distributions compared to the sphere reference, while the numerical values of distances between samples are inherently larger without. having multiplied with temperature.
E
Additional Ablation on product sphere Structure
We provide more ablation results regarding the structure of the product sphere manifold under fixed total dimensions in Table We observe that the PS(32, 32) configuration performs the best in general, while
21

---

Published in Transactions on Machine Learning Research (10/2024)
logits_Pos
logits_neg
img_ney fig__neg
text_neg
150
100
100
50
-0.4
-0.2
0.0
0.2
0.6
0.8
1.0
-12
-10
-6
sphere + <,,>
euclidean 12
-5
-1
PS + geodesics
PS <
(a)
logits_pac
logits_neg
img_neg
sext_neg
150
100
50
0.0
0.2
0.4
0.6
0.8
1.0
-8
-2
sphere + <<,>
euclidean
100
100
-2 PS geodesics
PS
<,
(b)
Figure 8: Visualization of the distribution distances between samples. The logits_pos and logits_nes neg denote the distances between positive and negative image-lext pairs, respectively. The img_nep neg and text text_neg denote the distances between negative image-image and text-text pairs, respectively. The models are trained using the Yfce datasets, (a) and (b) depict the distribution of in-domain data (Yfce) and out- domain data (RedCaps), respectively.
the sphere with more 1024-dimensional embedding has slightly better linear probe performance. We also notice that a more complicated structure provides better text-to-image retrieval results.
F
Additional Results Using the TCL Framework
We combine our proposed method with the TCL model (Yang al. 2022). which is one of the state-of- the-art vision-language retrieval models that employ contrastive visual-textual alignment in its earlier stage. During the pre-training, the TCL induces mixture of in-modal and cross cross-modal modal contrastive losses, while conducting the masked language modeling (MLM) and imagc-text matching tasks simultancously. During the testing, the cross-modal contrastive alignment head first. lists sample pairs with high similarity scores, and then these pairs are fed into the matching head to obtain the final matching scores. We alternate the topologies of all the embedding spaces with PS(128,2). For the experimental analysis in this subscction, we follow the configurations of the reference models, employ collection of CC3M (Sharma et al.] 2018). MSCOCO Captions (Chen et. al. 2015), Visual genome (Krishna al. 2017 and SBU (Ordonez et. al.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Zero-Shot 12T R@1
Zero-Shot
Zero-Shot Cls. Acc.
Linear Probe
Topology
Distance
T21 R@1
Cls. Acc.
Temperature, init=e" e 2.64, gradient= True
Sphere(512)
u
48.3
31.45
30.62
60.38
Sphere(1024) (1024)
u"
50.7
32.05
29.60
60.53
PS(128, (1228, 8)
-truu
49.4
32.85
30.55
60.12
PS(64, 16)
- -truu
50.3
33.25
30.34
60.16
P32, 32)
- tr(uTv)
52.3
33.47
30.62
60.32
Table 8: The retrieval and classification performance of the proposed approach using different oblique man- ifold structures and the multi-token implementation. (arddenn=(TTr//Faasss) donates if the temperature is learnable.
2011] as the pre-training dataset, which contains roughly 4 million annotated image-text pairs. The models are then evaluated using Flickr30k (Plummer et al. 2015) and MSCOCO Captions (Chen et al.) 2015
The results are shown Table g Since our method does not affect the matching head, we also report the performance of the contrastive alignment head. In general, our method improves the average recall performance, but the improvement is not significant. We consider the reasons as The method (or recent similar methods) employs pre-trained vision and language models, as well as a matching head and an ML. head; hence it is less sensitive to the gradients from the contrastive alignment; ii) The datasets employed for training contain less noise, while the training is scheduled with an overlength scheme (the zero-shot performance does not increase in the last epochs).
Additional Notes on TCL We also provide the comparison results with officially relcasod chockpoints. It can be seen that our implementation performs 0.5-1.0% worse than the official checkpoints. On the other hand, our implementation has better alignment head performance. Since in the official repository, the reason might be the following: i) Datascts difference, that we have ~3000 fewer images in the SBU dataset while owning 5000 more images in the CC3M dataset; ii) We resize the CC3M dataset to short edge 500 pixels, while the official repository does not clearly provide the pre-processing approach; iii) We implicitly have a short training time or smaller matching loss weight than the official checkpoints due to the difference in the framework.
employing the codes released
G Additional Results with more datasets
In Table 10 evaluate our models with more out-domain datasets, which means, the training splits of these datasets are not included in our train dataset. We employ the following datasets: Food101 (Bossard
let al.
2014), Cars (Krause et al., 2013), Cifar10/100 (Krizhevsky et al. 2009). Country211 Thomee et. al.
2016
(a sub-set of the YFCC100M), FGVCAircraft (Maji et al. 2013). GTSRB (Stallkamp et al. 2011
397
(Xiao et al., (016)01)), RenderedSST2 (Socher et al. 2013). SVHN (Netzer et al. 2011), STL10 (Coates)
el al.
2011h, DTD (Cimpoi el al. 2014), OxfordllIPet (Parkhi el al.) 2012), Flowers102 (Nilsback Zisser-
man
2008),
EuroSAT (Helber et al. 2019 and Caltech101 (Fei-Fei Fei et al.] 2006).
H
Test of Mixture-of-Expert Hypothesis:
We investigate the mixture-offexperr hypothesis of the proposed method. Since the class token is considered to encode the global representation of the sample, and all the class tokens are identical, iee. use the same positional embedding, the employment of multiple class tokens may function in mixture-of-expert style. That is, after training, each sub-sphere (or a subset of sub-spheres) in the product sphere structure is capable of alignment. Then, the system functions as a mixture of weak alignment models (experts). To test this hypothesis, we calculate the zero-shot classification performance of the CLIP(Multi(32116)) model with randomly selected subsets of sub-spheres. From Table I1 we find that the drop performance is reasonably small (~1)%) with half of the alignment tokens. This result reveals possible mechanism of the product
23

---

Published in Transactions on Machine Learning Research (10/2024)
Flickr T21
Coco
Method
12T
Recall
12T
T21
Recall
baselinelimpll
R@1
R@1
mean
R@L
R@1
mcan
Zero-shot performance.
93.00
79.60
93.97
71.40
53.50
79.49
[CL|officia]
(84.20)
(67.10)
(88.45)
(55.40)
(40.80)
(69.92)
91.00
78.28
93.25
70.16
53.05
79.07
TCI[ou-imppl]
(83.30)
(68.40)
(88.73)
(57.34)
(43.21)
(71.31)
91.20
78.14
93.29
70.14
53.35
79.14
TCL(PS(128.2)
(84.80)
(67.86)
(88.84)
(57.10)
(43.13)
(71.32)
Fine-tumed performance.
94.90
84.00
95.57
75.60
59.00
82.87
[CL[officia]
(87.90)
(71.38)
(90.92)
(65.34)
(48.94)
(76.53)
93.80
83.06
95.17
73.56
57.74
82.06
TCL|our-impl]
(88.30)
(72.94)
(91.27)
(66.98)
(50.34)
(77.43)
93.80
82.90
95.18
74.78
57.72
82.13
TCL(PS((18:2)
(88.60)
(73.26)
(91.39)
(65.60)
(49.83)
(76.86)
Table 9: Retrieval performance on Flickr30K and MSCOCO of our implemented TCL model and the variant using our proposed method. The numbers in brackets are the performance obtained using the rastivstive alignment head.
Food101
Cars
Cifar10 ZS cls.
Cifar100 ZS cls.
Country211 ZS cls. AccE1
FGVCAireraft ZS cls. Acc@1
GTSRB ZS cls. A
SUN397
Method (impl)
ZS cls.
zs cls. AccG1
ZS cls.
Acc01
Acc@1
Acc01
Accãl
WTTB//6-222 as visual done.
CLIP (openAIt)
88.7
64.8
90.8
66.9
22.8
24.2
43.4
64.4
CL.II 86.1
83.7
91.7
71.2
18.1
17.7
43.5
69.6
CIJP/our-impl]
80.8
67.2
94.8
74.0
16.3
10.7
48.5
50.6
CLIP (LII(M)
75.3
95.2
17.2
39.5
48.9
VIT-L/14-224 visual bone for reference.
93.1
77.9
95.6
75.8
31.9
31.8
50.6
67.6
CLiPC
89.6
94.6
77.4
23.0
24.9
49.9
72.6
RenderedSST2
SVHN
STLIO
DTD
OxfordllIPet ZS cls. Accãl
Flowers ZS cls. Acc@1
EuroSAT ZS cls.
Caltech 101
Method
ZS cls.
ZS cls. Aco@1
ZS cls.
ZS cls.
Zs cls.
baseline[impll
Acc@1
Acc@1
Acc@1
Accâ1
Acc@1
WT-B/16-22/ as visual bone.
CLIPfopenAIt)
60.7
51.9
98.3
45.0
88.9
69.1
55.9
89.0
CLLPppp 54.4
34.1
97.0
51.3
66.9
50.2
91.3
CLIP(our-impl]
31.1
98.5
51.0
85.3
77.4
36.5
85.3
CLIP (Multi((2211))
49.4
33.1
99.1
54.0
80.7
73.4
40.3
86.2
ViT-L/14-22/ visual bone for reference.
CLIP(openAI")
58.4
99.4
55.4
93.2
79.2
62.6
92.5
CLIP (LIP(openCLiP 56.0
49.6
98.1
60.5
91.7
73.1
62.3
92.7
Table 10: Additional zero-shot classification results. training dataset.
None of the datasets listed used as part of the
sphere structure during optimization, where subset of sub-spheres is priorly aligned. Therefore, some tokens are learned to be more sensitive than others specific objects (See the GradCAM visualization).
24

---

Published in Transactions on Machine Learning Research (10/2024)
elephants
Boldd green
eiephants
eleshants
Image-Tees Text Pair
Full Tokens
Token id=03
Token id=10
the angle of tower clock. the angle with clock ang
clock
view clock
Image-Tees Text Pair
Full Tokens
Token id 00
Token id-15 15
there are different citrus fruits the bowl. there are different ctrus fu in the boef.. there are differert cirus fruits the bowl
there are different citrus fruits the bosl
Image-Texx Text Pair
Full Tokens
Token id=08
Token id=12
simple office with chair table simple office red weodes bible simple offoc with chair woodentalle, office chalr Inbll
Image-Tecs Text Pair
Full Tokens
Token id=04
Token id=10
Figure 9: More visualization of the importance map using the Grad-CAM algorithm.
25

---

Published in Transactions on Machine Learning Research (10/2024)
elephants
fllld
expphnts fleld eleshants fiflld
Image-Tees Text Pair
Token id=05
Token id=06
Token id=07
the angle view of tower the angle Image-Tees Text Pair
clock.
clock.
view of tower clock
Token id-08
Token id 09
Token id-13 13
there are different citrus fruits the bowl. there are different cirustuuts is in the boof there are different chrus frilts the boell
there are cifferent citrus froits the bowl,
Image-Texx Text Pair
Token id=03
Token id=07
Token id=10
simple office with chair table
dimole office and toble, smple offos with
office chalt
Image-Texx Text Pair
Token id=03
Token id=09
Token id=13
Figure 10: Failure cases of the importance map using the Grad-CAM algorithm.
26

---

Published in Transactions on Machine Learning Research (10/2024)
# #Tokens
8
Top1 Acc. 13.0+9.7
21.5+14.3
27.7+:20.4
62.1 #4.4
Table 11:
ImageNet zero-shot classification performance of CLIP)Multi(22.16) model using randomly selected subsct of [CLS] tokens.
More Visualization using GradCAM
In Figure we provide more visualization results using GradCAM. In Figure 10 show some failure cases when the attention in the textual mode is focused on non-object words.
J brief introduction to the product sphere (oblique) manifold
The oblique manifold is a mathematical structure used in various optimization problems, particularly in the field of machine learning and signal processing. is type of manifold that arises when considering the set of matrices with columns constrained to have unit norm.
Definition: The product sphere manifold SP(n, p) is defined as the set of n p matrices X, where each column of X is unit vector in R". Mathematically, it is expressed as:
SP(n.p) X "xx xiag(XXX))
where I is the identity matrix.
Geometry: The product sphere manifold a product of p spheres 5"-1, each of dimension n 1, Therefore, it can be thought of (p
Tangent Space: The tangent space a point X the product sphere manifold is given by the set of matrices Z of the same size as X, such that each column of Z is orthogonal to the corresponding column of X. Mathematically, this is expressed as:
TxSP(n.p) 2 Rexp diag (X"
=)=0 =0
Exponential Maps: Due to the product structure of spheres, the exponential map at a point X in the direction of a tangent vector can be computed column-wise on each sphere.
K A note on the recent advance in noisy image-text matching
Recently, many pieces of research have been made to tackle the noisy image-text matching problem in con- trast learning. Below, we provide concise survey of these works, for the readers who want to know more about this topic. Although these works may not be comparable with our proposed method, they still support that noisy visual-textual correspondences are an important research topic in this field.
Chun et al. (2022): This paper argues that existing ITM benchmarks have a significant limitation of many missing correspondences. Then, it proposes new dataset, ECCV Caption, to correct. the massive false negatives and a new metric, mAP@R, to evaluate VL models.
Li et al. (2023): This paper proposes method to correct false negatives by integrating language guidance into the ITM framework. This framework corrects the locations of false negatives in the embedding space. Chun (2023): : This paper also argues that the image-text matching task suffers from ambiguity due to multiplicily and imperfect annotations. Then, this paper proposes an improved probabilistic ITM approach that introduces a new probabilistic distance with a closed-fore solution.
Huang al. (2021): This paper points out that the training data may contain mismatched pairs. To learn the noisy correspondence, the authors divide the data into clean and noisy the correspondence via adaptive prediction model.
partitions and then rectify
27

---

Published in Transactions on Machine Learning Research (10/2024)
Qin et al. (2022): This paper considers the major challenge in cross-modal retrieval is the noisy correspon- dence in training data. This refers to the fact that some of the training pairs may not be correctly aligned, t.e,, the image and text do not actually correspond to each other. They propose a framework to address this challenge by integrating two novel techniques: Cross-modal Evidential Learning and Robust Dynamic Hinge
Yang al. (2023): : This paper proposes a general framework for cross-modal matching that can be easily integrated into existing models and improve their robustness against noisy data. This framework estimates soft labels for noisy data pairs by exploiting the consistency of cross-modal similarities.
Han et al. (2023): The paper proposes Meta Similarity Correction Network to provide reliable similarity scores for cross-modal retrieval. The method learns to distinguish between positive and negative pairs of data using meta-data, and can be used to remove noisy samples from the training dataset.
28


Reviews:
[{"review_id": "UyGsloEssw", "reviewer": "Reviewer_EqWU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**  \n1. The paper is clearly written and well organized. I especially like their narrative to tell the story.  \n2. In Section 2, the author comprehensively discuss the properties of the parameter and its impact with the presence of noisy data due to semantic ambiguity. The conclusions are supported by empirical validations.  \n3. The authors conduct extensive experiments to support the effectiveness of the proposed multiple class tokens solution. The visualization of cross-modal attention further demonstrates the strength of the solution.  \n4. This work include sufficient comparisons to the prior works on visual-textual contrastive learning.\n\n**Weaknesses:**  \n1. While the multiple class tokens approach is one major contribution of this work, the motivation and implementation details are unclear. How are the multiple class tokens connected to the temperature parameter? In the pseudo code, why is only the [CLS] token embeddings are used to compute the distance? There is no discussion about these technical details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work discusses the properties of the temperature parameter in the training of visual-textual contrastive learning. Firstly, the authors show that larger temperature parameter is required to achieve an equilibrium between minimizing positive distance and maximizing negative distance for both a larger batch size and more noisy data. Then, the authors identify three prerequisites that must be satisfied to achieve an equilibrium (the contrastive loss converges). When there is semantic ambiguity, the temperature parameter can be utilized to re-scale the range distance range corrupted by noisy data. In turn, a larger temperature at equilibrium may imply higher noise level, leading to performance degradation. In response, the authors propose a multiple class tokens solution to mitigate the impact of temperature parameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors are supposed to add more detailed introduction and discussion on the proposed multiple class tokens approach. More specifically, I would recommend the authors to write down the detailed math formulas so that the readers can better understand the embedding space of [CLS] tokens. In addition, the author should clarify how the proposed approach is connected to the properties of the temperature parameter.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "szgjrBRDs5", "reviewer": "Reviewer_72pd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n\n1. [**Thorough theoretical analysis and insightful examination**] The paper provides a comprehensive analysis of the temperature parameter's role in contrastive visual-textual alignment models. This analysis is well-supported by visualizations and empirical results, offering valuable insights into the workings of these models. The authors identify three key conditions for effective contrastive alignment: proper uniformity of the embedding space, a \"relaxed\" triangular inequality, and a broad distance range. This thoughtful examination of an important component in contrastive learning models contributes significantly to the field's understanding.\n2. [**Novel and innovative approach**] The proposed method using product spherical embedding space with multiple class tokens is an innovative solution that addresses the identified issues without introducing significant computational overhead. This approach expands the distance range without requiring an explicit temperature parameter, potentially offering a more robust alternative to existing methods. The use of multiple class tokens within vision transformers is particularly noteworthy, as it allows for capturing different aspects of visual-textual alignment.\n3. [**Extensive experimentation and broader implications**] The paper presents comprehensive experiments on various benchmarks, demonstrating consistent improvements over baseline CLIP models. The average improvement of 6.1% across different tasks is substantial and well-supported by the presented results. The authors provide insightful Grad-CAM visualizations that illustrate how different class tokens capture various aspects of visual-textual alignment, adding depth to the empirical findings. Furthermore, the work has broader implications for other contrastive learning scenarios and multimodal models, potentially influencing future research directions in the field.\n# Weaknesses:\n\n1. [Lack of clear motivation and explanation for the choice of product spherical embeddings] The paper doesn't adequately justify why this particular approach was chosen over other potential solutions.\n2. [Insufficient explanation of the geodesic distance mentioned on page 5] The paper doesn't clearly define the distance metric or explain why the given equation minimizes the path distance.\n3. [Limited analysis of computational trade-offs and hyperparameter sensitivity] While the paper mentions that the multi-token approach requires more computational resources, it doesn't provide a detailed comparison of training time, inference time, and memory requirements. Additionally, the process for selecting optimal numbers and dimensionality of sub-spheres in the product space is not well-explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides an in-depth analysis of the learnable softmax temperature parameter in contrastive visual-textual alignment models like CLIP. The authors identify three key conditions for effective contrastive alignment: proper uniformity of the embedding space, a \"relaxed\" triangular inequality, and a broad distance range. To address limitations of the temperature parameter, they propose a novel approach using a product spherical embedding space with multiple class tokens, implemented for ViT. Extensive experiments demonstrate that this approach improves zero-shot classification and retrieval performance compared to standard CLIP models, with an average improvement of 6.1% on various benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide a clear motivation and justification for the choice of product spherical embeddings. Explain why this approach was selected and how it addresses the identified issues with the temperature parameter. Clarify the explanation of the geodesic distance on page 5. Define the distance metric explicitly and explain the reasoning behind the equation used to minimize path distance.\n2. Include a more detailed analysis of computational trade-offs, comparing training time, inference time, and memory requirements of the proposed approach to the standard CLIP model. Offer guidelines or heuristics for selecting the optimal number and dimensionality of sub-spheres in the product space, based on experimental findings.\n3. [Optional, Enhancemenet] If possible, provide more intuition on why multiple class tokens capture different aspects of visual-textual alignment (Consider exploring the connection between the proposed approach and mixture-of-experts models). Maybe discuss potential impacts on the interpretability of learned representations. I think these discussions could further enhance this paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dbq65YNNeI", "reviewer": "Reviewer_hAis", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths:***\n- The motivation of this paper is well stated and the paper is well structured.  \n\n- The analysis angle is new and interesting.\n\n- The proposed method seems effective.\n\n***Weakness:***\n- In the Section 2.2: 'Relaxed' triangular inequality, this paragraph is less supported:\n      \n    1. The claim: \"For this model, we have the following properties... a large $\\epsilon^-$\". Does this claim have any prior study or experimental results to support?\n\n    2. Three observations are not supported by experiments or reference papers either.\n\n    3. For \"minimize the contrastive loss (See supplementary materials)\", which section(s) in Supplementary it refers to?\n\n    4. \"Consequently ... instead of a larger value than $\\epsilon^-$ is not straightforward to me either, and it can be further explained.\n\n    5. \"For this simple deviation, .... contrastive loss\" can also be explained more.\n\n - When the temperature is untrainable, uniformity becomes an undesirable property in Table 2. Is there any comment or insight on this finding? \n\n- All experiments are conducted on one single visual backbone. It would be beneficial to see whether the observations generalize to other backbones and the proposed method remains effective. It is also unknown whether multi-token solution can be applied to CNN-based backbones.\n\n- CLIP[Multi(32, 16)] seems not to consistently outperform other CLIP implementations. Additionally, it is interesting to know the performance beyond ImageNet variants. Given that CLIP can be evaluated in a zero-shot way, it is recommended to compare Multi(m, n) with baseline CLIPs as in Radford et al. (2021).\n\n- To my best knowledge, the training dataset of OpenCLIP has been made public available. As suggested in [a], the dataset distribution determines the model robustness. To validate that it is the architectural design that benefits the performance instead of training set distribution, training a model on the dataset provided by OpenCLIP is more convincing to show the superiority of the design.\n\n[a] Data Determines Distributional Robustness in Contrastive Language-Image Pre-training (CLIP), ICML 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides analysis on the learnable softmax temperature parameter in the training of CLIP. It points out the significant role of the temperature parameter in handling the noisy training data. To mitigate the problem, an alternative approach is proposed from the topological view by capitalising on multiple class tokens embedded within the transformer architecture. The experiments show the efficacy of the method compared to baseline CLIPs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the paper is interesting and appears novel. However, there are concerns regarding the ambiguity of Section 2.2: 'Relaxed' Triangular Inequality. Additional experiments would better support the claims and make the paper more convincing. Please refer to the weaknesses highlighted to improve the submission.", "broader_impact_concerns": "No clear negative societal impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes that certain criteria are important for a contrastive loss to learn good representations and studies the effectiveness of several losses which meet these criteria to varying degrees. It then suggests a new approach to contrastive learning in Transformer-based architectures that L2-normalizes several CLS tokens, concatenates them, and takes their dot product, instead of using a single CLS token as in a standard model. This approach substantially improves accuracy on ImageNet and several other datasets.

This paper is borderline, with two reviewers weakly recommending acceptance and one weakly recommending rejection. On one hand, the proposed product sphere embedding method appears to be quite effective, improving ImageNet accuracy from 69.5% to 76.4%. On the other hand, the importance of the "Three conditions to achieve the equilibrium" is not well-supported by experiments. I was AE for a [previous version](https://openreview.net/forum?id=zNRGSv7WKm) of this paper and had similar concerns, but the revised paper is slightly better in this regard. Beyond these concerns, the clarity of the writing leaves something to be desired.

After considering the reviews both here and for the previous version of this paper, I think it should be accepted with the following changes:
- At the start of the subsection on "Three conditions to achieve the equilibrium", it should be made clear that the importance of these factors is speculative, and although Table 2 provides some evidence by investigating loss functions that meet only some of the conditions, these loss functions also differ along other axes, and thus these experiments provide only weak evidence that the factors are important.
- In several places in the work (such as at the end of the first paragraph), there are in-text citations such as "Radford et al. (2021)" where there should be parenthetical citations such as "(Radford et al., 2021)". Perhaps some `\cite` commands need to be replaced with `\citep`.

In addition, in many places, the writing is grammatically valid but awkward, e.g. the way that the phrases "visualize", "On the contrary", "we figured out", "we capitalize on" are used in the abstract are not incomprehensible, but also not quite idiomatic English. I don't think this should be a reason for rejection, but if the authors have any facilities at their disposal to improve the quality of the writing, I'm sure readers would appreciate it.
Decision: Accept (poster)