Design of the topology for contrastive visual-textual alignment
Authors: Zhun Sun
Venue: Submitted to ICLR 2023 (2023)

Abstract: Pre-training weakly related image-text pairs in the contrastive style shows great power in learning semantic aligning cross-modal models. The common choice to measure the distance between the feature representations of the image-text pairs is the cosine similarity, which can be considered as the negative inner product distance of features embedded on a sphere, mathematically. However, empirically, aligning image-text pairs on the spherical topology is vulnerable to the semantic ambiguity phenomenon resulting from the noise in the pre-training datasets. Specifically, under the noisy training data, instead of the optimal alignment-uniformity solution, the system would achieve an equilibrium (a gap between distances of positive and negative pairs), when the gradients for attraction and repulsion are neutralized.  Although intuitively, the model should always find this equilibrium given a sufficiently long training scheme, its numerical values might be out of the distance range (e.g. [-1, 1] for the cosine similarity). In the practice of former studies, this problem is partly tackled by introducing a learnable softmax temperature parameter, in other words, by explicitly scaling the range of the distance function.  In this work, we alternatively design the topology of embedding space and its endowed distance function. Motivated by studies that make use of Riemannian geometry for visual tasks, we propose a rather simple solution to address the aforementioned equilibrium problem. That is, we map the feature representations onto the oblique manifold endowed with the negative inner product as the distance function. In the experimental analysis, we show that we can improve the baseline performance by a large margin (e.g. 4\% in the zero-shot image to text retrieval task) by changing only two lines of the training codes.

Under review as a conference paper at ICLR 2023
DESIGN OF THE TOPOLOGY FOR CONTRASTIVE VISUAL-TEXTUAL ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-training weakly related image-text pairs in the contrastive style shows great power in learning semantic aligning moos- models. The common choice measure the distance between the feature representations of the image-text pairs is the cosine similarity, which can be considered as the negative inner product dis- tance of features embedded aligning image-text pairs on the spherical topology is vulnerable to the seman- tic ambiguity phenomenon resulting from the noise in the pre-training datasets. Specifically, under the noisy training data, instead of the optimal alignment- uniformity solution, the system would achieve an equilibrium distances of positive and negative pairs), when the gradients for attraction and repulsion are neutralized. Although intuitively, the model should always find this equilibrium given a sufficiently long training scheme. However, since the con- trastive loss implemented using softmax and cross-entropy loss, which makes the required numerical values for equilibrium much larger, where the equilibrium might be out of the distance range (e.g. [-1, 1] for the cosine similarity). the practice of former studies, this problem is partly tackled by introducing learnable softmax temperature parameter, in other words, by explicitly scaling the range the distance function. In this work, we alternatively design the topology of embed- ding space and its endowed distance function. Motivated by studies that make use of Riemannian geometry for visual tasks, propose rather simple solution to address the aforementioned equilibrium problem. That is, we map the feature rep- resentations onto the oblique manifold endowed with the negative inner product as the distance function. Furthermore, we propose a multi-token implementation of the oblique manifold. With this configuration, we outperform the officially re- leased CLIP-ViT/L-1 model using ViT/B-16 visual backbone on the zero-shot image classification and image-to-text retrieval tasks.
sphere, mathematically. However, empirically,
gap between
INTRODUCTION
Learning visual and textual feature representations that are semantically aligned in their embedding space is ordinary problem in the rision-language mooss-modal tasks (Frome et al. 2013 Karpathy
& Fei-Fei 2015) Romera-Paredae & Torr 2015 Wang et al., 2016 Faghri et al.] 2017] Xian et al., 2016). In early works that employ feature representations from deep neural networks, e.g. Frome ee al. (2033) the alignment is often achieved by a fundamental metric learning approach with the hinge rank loss. That is, the similarity between visual feature vector u and textual feature vector is calculated as u Wo, where W are the learnable weight parameters. Thanks to the revolu- tionary advances in computational power, we can now achieve this in a more effective and practical approach termed contrastive learning, where we align quantities of positive samples and push their negative samples away simultaneously
large mini-batch (Radford et al.) [202T] (Singh al., 2022, Jia et al.) 2021] Pham et al. 2021 Yuan et al.) 2021). The standard choice of the distance measure between an image-text pair for the contrastive learning algorithm is the Cosine Similari ty (in both uni-modal Chen et al. (2020a); Caron et al. (2020); Chen et al.) (2020b) and cross-modal Rad- ford et al. (2021); Jia et al. (2021) Singh et al. (2022) scenarios). Mathematically. the Cos ine Similarite ty computes the negative inner product value between feature representation vectors mapped onto the unit sphere embedding space. The spherical embedding space is advantagcous in aligning visual and textual feature representations in the following aspects. First, calculat-

---

Under review as a conference paper at ICLR 2023
ing the inner product consumes low computational resources during both forward and backward propagation. Second, we have a proper definition of uniformity on the sphere. The uniformity is a demanded property in optimizing the contrastive loss, by which the feature representations would preserve maximal information of the data.
However, since the data for large-scale contrastive alignment are internet-collected noisy image-text pairs, we often find pairs of semantically related images and texts labeled as "negative" and verse visa, which we term "semantic ambiguity". Because of the ambiguity, it is impossible the perfect alignment and uniformity conditions of sample embeddings for the system. More specif- ically, during the training, the false negative samples are pushed away from each other (repulsion), while the false positive samples are pulled together (attraction). As a consequence, the system will gradually find an equilibrium when the noisy samples' gradients for attraction and repulsion are neu- tralized. In other words, we say the training progress is converged under the given To be more concrete, owing to the fact that the gradient is eventually back-propagated from the dif- ference between the positive and negative distances. Given sufficient model capacity, the numerical values between the distances of positive and negative pairs of samples will be optimized to fit the noisy level of the dataset. For instance, if there is reasonable amount of false negative samples, then the model would learn a smaller positive distance for not being punished too hard, when en- countering false negative samples in another mini-batch. Furthermore, the triangular inequality (or a "relaxed" version) of the distance within a group of semantically similar samples will pull the positive pairs of samples away from each other (See Section 3.2) Finally, the model reaches the cquilibrium of compromised positive and negative distances, which minimizes the contrastive loss regardless of the semantic ambiguity. Therefore, we say the equilibrium is essentially property of the dataset, referring to its noise level, under a certain embedding space and its distance function.
achieve
Here, the problem is that, since we implement the contrastive loss with the combination of softmax and cross-entropy, which makes the required numerical values for equilibrium exponentially larger. The numerical values of distances required by softmax at equilibrium might be out of the distance range (e.g. [-1, 1] for the cosine similarity). Therefore, in the practice of former studies (Wu et al.) 2018 Radford et al. 202I), researchers have to scale the range with a learnable softmax tempera- ture parameter. Although the learnable temperature somehow fixes the range problem, it still has two drawbacks that need to be addressed. Firstly, the learnable temperature delays the learning progress. Empirically, we observe that the model trends to acquire proper scaling for the distance range ear- lier than achieving a good alignment. Secondly, a large temperature is numerically unstable for the pocpppropagation, especially for the low-bit precision computation. In this work, we alternatively design the topology for embedding vectors and its endowed distance function. Motivated by the utilization of Riemannian geometry for visual tasks, we propose relatively simple solution to ad- dress the aforementioned out-of-range equilibrium problem. Our contributions can be summarized as follows:
1. We reveal that the learnable softmax temperature is essentially scaling factor for the dis- tance range, which indicates the noise level of the dataset in the contrastive visual-textual alignment. We also observe that the model learns a suitable temperature before representa- tions, which degrades the performance.
2. We tackle the out-of-range equilibrium problem resulting from the softmax cross-entropy loss, by designing the topology of embedding space. That is, we employ the oblique mani- fold endowed with the negative inner product distance functions.
3. We demonstrate that the proposed approach can be non- con-pinfuull implemented by changing only two lines of the training code, whilst improving the baseline performance in the zero- shot image-tex retrieval tasks. In the larger scale experiment, we have learned a ViT-B/16 model that outperforms the officially released ViT-L/14 model.
PRELIMINARY
We start with notation and review mathematical expressions of the basic building blocks used in our analysis. In this work, we denote scalars by italic letters, e.g., n, m, B, D E R, and denote vectors and oigher-order tensors by boldface letters, e.g., x [zo, Zi, 1n-1] T E R and Y e N We denote sets by calligraphic letters, e.g., 21 = (U, U2.... ). We also employ italic letters to define functions, with subscripts denote their parameters, e.g. fo(.). The operation lpp
Notations:
2

---

Under review as a conference paper at ICLR 2023
denotes the Ep norm of vector and denotes the absolute value of a scalar. For any integer K, we use [K] to denote the set of integers from 1 to K.
Textual Pre-trained Model: Given set of semantically related image-text pairs S {(U,, V1), (U2, V2), (UK, Vx)), where U is an image of size H x W x C. V is a tokenized text of length L. The goal is simultaneously learn pair of encoders fo U u, Ip V to map the image and text into an embedding space, u, v are called embedding vectors of samples. A well-optimized visual-textual pre-trained model aligns the embedding vectors across the visual and textual models. That is, the embedding vectors extracted semantically related image-text pairs earn higher similarity scores than the non-related ones. To generalize the problem, we view the embedding vectors as points on specified typologies, and the similarity score between embedding vectors an endowed distance function that evaluates the distance between the points. For instance, the commonly employed cosi i ne simí larity calculates the inner product of the normalized embedding vectors on the unit sphere as the (negative) distance between the sample pairs. To this end, we further consider the encoders as compositions of functions that i) map the inputs into the euclidean space and ii) map the input vectors in euclidean space on specified typologies. We denote this two-step mapping as: fo fo f, where fo U - ū is the encoder with learnable parameters, E Rd the output in the d-dimensional euclidean space. f ū + u, u E M is an specified op- erator (without learnable parameters) maps the representations onto topology M, which is usually considered as manifold embedded in the d-dimensional euclidean space.
Contrastive Learning: Following the definition in Oord al. (2018); Wang & Isola (2020): Chen
ct al. 2021 Ia Radford et al. (2021). we formulate the contrastive loss as
e (rd(f((U)ao(VV))
Co(fe.
E U.
log
ΣjE[M] 7dllo(W))oo Eie[Mn e F (0(
U, A VJ #V
where 7 is the temperature term, we write it as a multiplier for simplicity. d(.,,) is the distance function between two points. M Z+ is fixed number of negative samples. Briefly, optimizing this loss term minimizes the distance between positive image-text pairs and maximizes the distance between negative image-text pairs. It is worth mentioning that, in recent studies Radford et al. (202T) Chen et al. (2021b), the contrastive loss is usually implemented as the cross-entropy between one-hot labels and the class probability obtained by softmax within a mini-batch Sys. We also employ this implementation in this work, which can be formulated as
I1 (alo(())) + -H (alo(v)))
LolJes9oiTa Uoi = E U.V-SM iE[M]
(2)
where H(-I-) is the cross-entropy loss, U4 (-TdfaU)). V, (-rd(fo(U,) 90 (V)))) are the (negative) distance between an image/text to all the texts/images in the mini-batch. o is the softmax function, is the one-hot label vectors of
Oblique Manifold: Briefly, the oblique manifold Ob(n, n) the set of matrices of size n x m with unit-norm columns. We follow the definition in Absil et al. 2009), that is,
Ob(n, m) == (X E T== diag( X x) In).
where the diag(-) is the diagonal entries of the matrix, I, is the identity matrix of size n. The geometry of an oblique manifold exactly the same that of the product manifold of spheres ... x g 1n where gn-1 is the sphere manifold embedded in R
m coples
METHOD
In this section, we introduce how we implement the proposed approach. That is, employing the oblique manifold the embedding space, with the (negative) inner product its distance function.
3

---

Under review as a conference paper at ICLR 2023
image_encoder vision transforner
2 #
text_encode text transTormer
3
#
U
mini batch of aligned images, [ex hy My c]
4
#
V
mini-hatch of aligned texts, (n, 1]
5
#
of the obl ique monifold
Ob_n
of the oblique manifold
learned temperature parameter
extract features each modality w_bar (t) (U) f[a, d]
v_bar text_encoder (V) A[a, d]
13 Tinees 15-16 are the only codea that need to change compared to standard CTTP algorithm.
14 # map features onto ob (n,
15 u_bar.reshapa ('n -> db_m ub_n') _122__11maa_a_aee (axis=-1) ceeshape'">> n. d!)
repapere (*a
Ob_m Ob_n') .12_norma_izee (axis=-1) (axiss- rreshape (1-> 61)
sealed pairvisa inner product distance, [n, t.exp()
nag_dt distancas
symmstric funstion
labels azange (n) $ 0, 1, ... a
23 lass (cs_ (neg_distances, labels, axis=() ci_losoonee__daatanee labels, axis=l)) /
Figure 1: Python-like pseudo-code for the core of the proposed approach.
3.1 PROPOSED APPROACII
To map the feature vectors onto the oblique manifold Ob(n, m), we first reshape the feature vectors tł ò of size d to matrix of shape x then we -oromalliz the columns to obtain u, v. Finally, the distance defined as the negative value of the trace of the matrix product, i.e. d(u, v) -truu"v), We provide the python-style pseudo-code of the proposed approach in Figure 1. is worth mentioning that, in the implementation, we vectorize the embedding vectors back to their original size d, then apply the vector inner product to compute the distance. Moreover, we employ the term "neg distances" to avoid reduplicated calculation of the negative operation.
Basic Implementation:
Multi-Token Implementation: We provide another implementation of the proposed topology and distance functions by utilizing the [CLS] token (the token used to reperent global information in ViT, see the implementation details in Dosovitskiy el al. (2200) of the transformer architecture. Briefly, attach multiple randomly initialized [CLS] tokens in the first embedding layers. At the output layer, we consider each token embedding as a sub-sphere of the oblique manifold. We then compute the inner product of the vectorized visual and textual embeddings. Unlike the standard implemen- tation, the multi-token implementation requires more computational resources in the backbone. On the contrary, the sub-sphere could benefit from the global attention operation and provide more representative feature embeddings.
3.2 RETHINKING THE PROPERTIES OF TOPOLOGY
In this section, discuss the motivation for our proposed approach in detail. Besides the former discussed distance range, we further discuss the properties of the proposed approach by comparing it with three reference configurations of different topologies and distance functions. Specifically, we consider i) the sphere gd-1 endowed with the inner product as distance, ii) the euclidean space gd endowed with & distance; iii the oblique manifold Ob(d/ m, m) endowed with the minimizing geodesic as distance, which denoted as Geo(u, v) = tr (arccos v)). The overview of the comparison is summarized in Table The comparison summarizes the three important properties of the embedding topology in addition to the distance range. Keeping these properties favored for contrastive learning is important in the design of embedding topology.
4

---

Under review as a conference paper at ICLR 2023
Topology Distance
Sphere g -u
Euclidean R
Oblique d m, m)
Oblique d/m, m)
a 2ll2
Geo(u, v)
-tr(u"
Memory Resource
O(b²)
((b2d)
O(t2m)
0(12)
Uniformity
surface measure
undefined
surface measure
surface measure
Inequality
relaxed
restricted
restricted
relaxed
Distance Range
[-1, 1]
+00)
[0, mn]]
m, ml
Summary of different topologies endowed with different distances. The total dimension of the embedding vector is denoted as The mini-batch size is denoted as b. Green box stands for the properties that are favored for contrastive learning. Red box stands for the properties that are unfavvred for contrastive learning. Best view in color.
Table 1:
i. Low computational resource: In the contrastive learning algorithm, the logits (or dis- tance/similarity) matrix often costs the most computational resource in large-scale training. For instance, given a mini-batch of sample pairs of size b with di-dimensional output, the computa- tion of the inner product achieves complexity of Ob2²d) and storage usage of O(b²). However, since the pack-propagation of the f2-norm requires intermediate results that cannot be "inplace" calculated, the E norm (or any lp ---m based distance) in euclidean space requires a storage us- age of O()'d). On the contrary, the geodesic distance cache curve lengths and hence requires Obb2m) storage usage, while the proposed approach only needs one matrix multiplication after re- re-vectorization.
ii. Proper definition of uniformity: As explained by Wang & Isola (020). contrastive loss is a combination of two objects a) alignment of features from positive sample pairs and b) the distribution of the features encouraged to match. Naturally, with the loss form defined in Equation (2). the distribution object will result in uniform distribution the sphere. Although it is not essential for the distribution of samples to be uniform as discovered by Chen et al. (2021a), it is necessary define a proper prior distribution for samples to match via optimal transport algorithms (e.g. sliced Wasserstein distance), which is undoubtedly a computational burden. Both the sphere and oblique manifold have a proper uniform distribution defined the surface area measure, while the unbounded euclidean space does not. In practice, the & norm defined distance between the samples grows larger along training and eventually overflows.
iii. "Relaxed" triangular inequality: Assume that we have "well-optimized" model fā. 9h where for positive pair (U, V)+, their distance e- = d(uu, v²) d(fz (U))
(U)) (V)) able small, and the distance for negative pairs (U,V) is reasonable large. Let us consider the following scenario, in S {UU, V1), (U2, V2)}, the pair (2) is also semantically cor- related despite of being recognized as a negative sample. For the "well-optimized" model, will predict distance of e for this pair instead of e If the distance function d is metric, then according to the triangle inequality axiom of metric, we have the following inequality,
is reason-
6 = d(u vi)= S d(uuj, 15)) + d(uz; mi) + d(u;, of) = 3
(4)
We can see that the negative distance is bounded by three times the positive distance, despite the fact that the positive distance should be as small as possible. Given the fact that, in a mini-batch of sufficiently large size, it very likely to have negative pairs of image and text that match each other equally well as the positive ones. Therefore, we need to "relax" the triangular inequality to alleviate the effects of ambiguity of the positive/negative pairs. In practice, we have the following observations: i) The positive pairs of samples usually have a much larger distance than that in the perfect alignment condition, ii) d(u,; wz)) may become exceptionally small since none of the loss terms regularize it, resulting in further tightened bound of the negative distance; iii) Although the inner product distance is not a metric, it still obeys an "relaxed" triangular inequality because we can yield metric the sphere by ArcCos (the geodesic), see Schubert (2021) for more information.
In Table []] we characterize both the minimizing geodesic and the l norm distances are metrics and hence obey the restricted triangular inequality.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTAL ANALYSIS
4.1 EXPERIMENTAL SETTINGS
Datasets: For the experimental analysis in Section 4.2 we employ the 15M subset (Cui et al., 2022) of the YFCC100M dataset (Thomee et al. 2016) as the training dataset, which contains roughly 15.3 million internet collected weakly related image-text pairs. We evaluate the proposed meth- ods with two types of vision tasks: i) Zero-Shot image-to-text and text-to-image retrieval on Flickr30k (Plummer et al. 2015) and iï) Zero-Shot and Linear Probe object classification on ImageNet-1K (Russakovsky et al., 2015). Furthermore, we employ the RedCaps (Desai et al.) 2021 dataset as the out-domain data for visualizing the distributions of sample distances. For the experimental analysis in Section 4.2 we collect data from publicly available datasets (Schuhmann al. (02I];; Changpinyo et al. (202I); (Sharma et al. (2018); Chen et al. (2015); Krishna et al. [ Plummer et al. (2015] Chen et al. (2015); Russakovsky et al. (2015): Desai et al. (2021); Kuznetsova al. (2020); Li al. (2017). etc.). We also have clawed weakly related image-text pairs from the web, resulting a total of 420 million individual images and roughly 500 million image-text pairs. This dataset is comparable to the one employed in the official CLIP paper (Radford et al.) 2021 and another open source re-implementation (Ilharco et al. (021). To further remove the bias caused by datasets, we also re-implement the naive clip algorithm for reference.
Due to the limited computational resources, we adopt moderate scaling of the models. Specifically, For the ablation experiments, we employ the original ViT-S/16 architecture for our image encoders (Dosovitskiy et al. 2020), with an input image resolution of 224, resulting in 196 image tokens. For large-scale training, we employ the ViT-B/16 as our image encoders. For our text encoders, employ Ernie- meniie2.0-en-bee O-e-bbeee (Sun et al. 2020), which is literally Bert model (Devlin et al. 2018) of 12 layers and 512 hidden neuron sizes with a customized vocabulary of 30,522 to- kens, and the maximum context length is set to be 77. We project the feature representation ([CLS] token) from the top layer of transformers to a (sum of) 5 512-dimensional embedding space. All the parameters except the temperature are optimized from random initialization. The default initial- ization of the project matrix employs the Gaussian initializer of zero mean, and standard deviation equal reversed square root of the input size (a.k.a. Kaiming initialization). For the temperature, we initialize it with e' for & = e", ee,ee" e². Syyper-parameteyy employed for training are provided in the appendix. The details of the hyperparameters are provided in Table in the appendix.
Models:
For zero-shot retrieval on Flickr30K and Coco, we employ the logits (distance) com- puted by the distance function and report the image-text pairs with the top-k shortest distance as the retrieval results. For zero-shot classification
Evaluation:
ImageNet. We employ multiple prompt templates described in Radford et al.) (202I), while we first compute the distances between image and text embeddings, then average the distances. For linear probe classification the learned projection head (no topological structure is preserved), then attach a random initialized linear projector map the feature representation the 1,000 class logits.
ImageNet, we remove
4.2 EXPERIMENTS RESULTS
Comparison between different configurations: In Table 2 we report the performance of the con- figurations described in Table[] We employ the oblique manifold structure of 64, m 8 for our proposed approach, denoted as Ob(64,8) 88) in the table. The proposed approach provides 1.40%%/1.449% improvement over the baseline for the retrieval tasks. It is worth noting that the geodesic distance version also improves the baseline reasonably, despite the fact that it obeys the restricted triangular inequality. This suggests that a more elaborate topology structure helps the alignment of the fea- tures from different modalities. The Fuclidean configuration performs the worst, which could be the drawback of having no properly defined uniformity. For the zero-shot and linear probe classification accuracy on ImageNet, the oblique configurations provide better performance when the temperature initialization is low. We suggest the reason is that the equilibrium temperature for the oblique is lower than that for the sphere; hence the model could focus more on learning the representations of the visual modality.
Effects of temperature initialization: In Table 2 also demonstrate the results using different temperature initialization. Although it can be seen that the final performance is not largely impacted by the initialization, large small temperature at the start of the training still cause difficulties

---

Under review as a conference paper at ICLR 2023
Lero-Shot 12T R@1
Zero-Shot T21 R@1
Zero-Shot Cls. Acc.
Linear Probe Cls. Acc.
Topology
Distance
Temperature, init=e', gradient= True
Sphere Fuc
48.3
31.45
30.62
60.38
u m/12
47.9
32.29
30.36
59.90
Ob(64, 8)
Geo(u, m) -tr(u"
50.7
32.35
30.79
60.43
Ob(64, 8)
$2.3
32,89
30.70
60.32
Temperature, init=e", gradient= True
Sphere Euc
- -² Iu 2/12
46.8
31.13
29.60
59.82
48.5
32.69
30.68
59,74
0b(64, Ob(64.
Geo(u, v) 50.7
31.59
30.34
59.60
-tr(w" v)
33.37
30.23
59.76
Temperature, init=e", gradient= True
Sphere Euc
- -t
49.0
30.33
28.59
59.56
I 0//2
47.4
30.71
29.85
60.09
Ob(64, 8) 0b(64, 8)
Geo(u, v)
49,9
32,49
30.21
60.61
-tr(u" v)
50.9
32.71
30.50
60.66
Temperature, init=e", gradient-
Sphere Euc
3,461
4.04
45.37
Iu - 11l2
47.6
30.43
29.51
59.20
0b(64, 8)
Geo(u, 8)
2.921
3.10
21.67
0b(64, 8) -tr(u"
30.3
18.48
21.20
57.93
Table 2: The retrieval and classification performance of different configurations under different tem- perature initialization conditions. "gradient=
"gradient=(Trru/Falle) (TTru/Faasse) " donates if the temperature learnable.
2-
0-
20000
40000
60000
50000
100000
20000
40000
60000
80000
100000
sphere <
euclidean
4-
0-
0
20000
40000
80000
80000
100000
20000
40000
60000
80000
100000
oblique geodesics
oblique
Figure 2: The learning curve of the temperature. -isd. 12 and geodesics denote the negative inner product, l2, and minimizing geodesic distance, respectively. The orange, blue and green curves denote the initialization of e², el, and e", respectively.
in optimizing. It is also interesting to see that when the temperature is fixed at 1.0, the sphere one (narrow distance range) and the geodesic (restrict triangular inequality) fail learn meaningful feature embeddings for contrastive alignment. The performance of the proposed approach is also largely reduced. However, since the euclidean topology does have an upper bound of the dis- tance, the optimizer can still reach the equilibrium. We further draw the trend of the temperature during the training progress in Figure From the figures, we can confirm that: Given bounded distance range, the temperature is an inherent property of the datasets, depicting the noise level of datasets; ii) The temperature will first converge to an equilibrium regardless of initialization, then raise gradually along the optimization progress; iii) It takes longer training iterations for the temper- ature to converge on the oblique embedding space, while the final temperature is smaller than that in the sphere embedding space. The results suggest that border distance range and topology structure help the model focus more on aligning images and texts rather than finding the cquilibrium.
Retrieval performance Flickr and Coco of our implemented TCL model and the variant using our proposed method. The numbers in brackets are the performance obtained using the contrastive alignment head.
Ablation on oblique structures and multi-token implementation: In Table 3 we modify the structure of the oblique manifold under fixed total dimensions. It be seen that higher m

---

Under review as a conference paper at ICLR 2023
Zero-Shol 12T R@1
Zero-Shot T21 R@1
Zero-Shot Cls. Ace.
Linear Probe
Topology
Distance
Cls. Acc.
Temperature, init=e".
gradient= True
Sphere(512)
-u'"
48.3
31.45
30.62
60.38
Ob(256, 2)
-tr(u"v)
48.0
32,25
30.33
60.52
Ob(64, 8)
-tr(u"=)
52.3
32.89
30.70
60.32
Ob(16, 32)
-tr(u"")
50.4
33.01
30.93
60.79
Ob(4, 128)
-tr(u"vv
48.2
32.91
30.57
59.99
Multi(25,, 2)
-tr(u"')
49.2
32.29
30.04
61.59
Multi(64, 8)
-tr(u"
54.0
34.27
31.93
62.41
Multi(16, 32)
-tr(u"
54.0
33.43
30.88
63.71
Table 3: The retrieval and classification performance of the proposed approach using different oblique manifold structures and the multi-token implementation. ggradient=(Tru/Fulkse donates the temperature is learnable.
ImageNet Zero-shot Acc@
Flickr Zero-shot
Coco Zero-shot
Method [ispline(impl.]]
12T
T21
Recall
12T
T2I
Recall
R@1
R@1
mean
R@! 1
R@
mean
VTTR//66-224 visual bone.
CLIP[openAT']
68.7
81.9
86.1
55.4
38.4
66.3
CLIP(openCLIP1) 67.0
83.2
65.5
87.6
52.4
38.4
62.4
CLIP(our-impL.] 69.5
84.2
61.7
86.4
64.1
43,9
72.4
CLIP(Multi(16.32)
76.4
85.2
66.3
88.3
63.8
42.9
72.4
ViT-L/14-224 visual bone for reference,
CIJP||opeAAI|]
75.5
85.0
65.2
87.7
56.3
36.5
65.2
CL.P||penCILIP1] 72.7
87.6
70.3
90.1
59.7
43.0
70.0
Table 4: Comparsion of large scale contrastive visual-textual pre-train model on benchmark datasets. and denote the implementation from Radford al. (2021) and Ilharco al. (2021). respectively.
value (i.e. the number of product sub-spheres) is likely obtain better zero-shot text-to- image retrieval recall. However, an over-complicated structure such as Ob(4, 128) could ruin the performance. We conjecture that, since the sphere has one redundant dimension, the larger number of product sub-spheres reduces the representation capacity of the topology. And because we employ textual encoder that is gently larger than the visual one, therefore the moderate reduction of the capacity helps overcome the overfitting on the textual side. We also provide the results using the multi-tokens implementation discussed in Section 3. denoted as Multi(-.... For the linear probe, concatenate all the representations together betore projecting them to 1,000 class logit. This implementation outperforms other configurations by a large margin at the cost computational resources. Training the largest Multi(16, 32) models takes approximately 14% more time.
4.3 RESULTS USING LARGE-SCALE DATASET
In T we compare the performance of the proposed method using larger scale configuration. can seen that using the ViT-B/16 as the visual bone, our implementation of the naive model performance is similar to the publicly released ones. On the other hand, the model equipped with the multi-token implementation of (16,32) significantly outperforms the other ViT-B/16 models, with only less than 8% more computational costs.
5 RELATED WORKS
: Li et al. (2021a). the mo- mentum (ssti-)distillaation is introduced to mitigate the semantic noise in the sample pairs. That is, momentum version of the model is updated by the moving average of the model's historical parame- ters. Then, the cross entropy between the softmax logits computed by the model and its momentum version is used as an additional loss for supervision. The authors claim that the pseudo-targed of the momentum (self- Jdistillation will not penalize the model for matching negative samples that are reasonably similar. Here, consider that the pseudo-targets do "relax" the triangular inequality
Momentum distillation: In recent works such as Cheng et al. (202T):
8

---

Under review as a conference paper at ICLR 2023
restriction implicitly by letting the distance of alignment be reasonably large. Hence, it could be much easier for the optimizer to find the equilibrium discussed in Section 3.2
Other implemention of non-metric distance: In Yao et al. (2021), the authors proposed called fine-grained contrastive learning scheme that matches all the visual and textual tokens using maximum-aveed aveerage operator. Concretely, for each visual token, it finds the textual token with maximum similarity, then takes the average over the visual tokens as the similarity of the image to a text and vice versa. Using our framework, this work can be explained as embedding samples onto the product manifold gd-1 x ... x gd-1 endowed with the maximum-averagg distance, which is a non non-metric distance. At the same time, the authors employ the sub-manifold g represent local information.
The effects of softmax temperature: In Wang & Liu (202I). the authors draw the uniformity the embedding distribution and the tolerance to semantically similar samples of learned models under different temperatures. From the observations, the authors claim that "a good choice of tempera- ture can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances". Unlike our work, this work is done under uni-modal contrastive learning, where the semantic cor- relation of the negative samples is not a property of the datasets but rather a drawback of the larger mini-batch size.
Uni-modal side tasks: In works such as Mu et al. (202)): Li et al. (222bb;; Yang et al. (2022), au- thors combine cross-modal contrastive loss with other uni-modal tasks, for instance, visual/textual self-supervised contrastive learning, masked image/language modeling. These combined methods demonstrate superior performance in downstream tasks such zero-shot classification empirically. Although these works do not overlap with this one, we find that the uni-modal tasks provide reason- able uniformity within the visual/textual feature embedding, contrary to the cross- cross-modal contrastive shown in Section 3.2 Therefore, the model could obtain a more "numerically relaxed" triangular inequality when dealing with noisy pairs of samples.
Other works that employ oblique manifold: It is notable that learning representations embedded on the oblique manifold for computer vision tasks have been explored by former studies. For in- stance, in Qi et al. (202T), the authors employ the oblique topology for few-shot learning. However, different from these works, our paper mainly tackles the noisy database problem in the contrastive image-text alignment task. We employ the oblique topology with non-metric distance function to tackle the out-of-range equilibrium problem.
The hyperbolic topology is another popular choice for hicrarchical representation embedding space, in both NLP (Nickel & Kiela 2017). CV (Zhang et al. 2022 Liu et al. 2020 Khrulkov et al. 2020), and cross-modal (Guo et al. [021] tasks. However, the hyper- bolic topology has unfavored properties similar to Euclidean space. That is, the resource required for computing distance is high, difficult to define/implement uniformity in terms of numerical sta- bility, and the geodesic distance has restricted triangular inequality. Therefore, we do not consider this topology in this study.
Hyperbolic embedding space:
CONCLUSION
Summary: In this work, discuss the important properties of the feature embedding space for contrastive alignment. We show that the most commonly adopted cosine similarity has disadvan- tages in dealing with noisy data and training stability. Therefore, we propose to combine the oblique manifold with the negative inner product distance to tackle these problems. The proposed approach demonstrates better performance in the image-text retrieval tasks practically.
Limiation: First, due to remarkably limited computational resources, we cannot conduct experi- ments on a larger scale in terms of batch size, training data, and parameters in the neural network (and time). Second, in recent studies, besides the contrastive alignment, more pre-training tasks are appended to the head of the model using the non-normalized full token embedding. Such as image- text matching (Li et al. 202 Yang al.) 2022), image captioning (Yu et al.) (022), or masked modeling that do not employ the contrastive alignment (Wang et al. 2022). these configurations, the performance improvement resulting from better contrastive alignment could be marginal. And hence leave future work on designing the topology of the full token embedding.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 14:11834--1844 2021a.
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:/504.000 00325, 2015.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self supervisd vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021b..
Ruizhe Cheng, Bichen Wu, Peizhao Zhang, Peter Vajda, and Joseph Gonzalez. Data-efficient language-supervice zero-shot learning with self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3119-3124, 2021.
Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive
language-image pre-training: clip benchmark data, model, and supervision, 2022.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11433 2021.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint 1eXivv1707.0560 2017.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in neural infor- mation processing systems, 26, 2013.
Hao Guo, Jiuyang Tang, Weixin Zeng, Xiang Zhao, and Li Liu. Multi-modal entity alignment in hyperbolic space. Neurocomputing, 461:598-607, 2021.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori,
Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openolip. Zenodo, July 2021. doi: 10558//2enodo.51337777 5143777
URL|httpst / 0ww.:/0///2221.21111112
10

---

Under review as a conference paper at ICLR 2023
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Sung, Zhen Li, and Tom Duerig. Scaling up visual and Asionnlanguage representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3128-3137, 2015.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6418 2-282, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International of computer vision, 123(())32-73, 2017.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Tustt, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, (8(7):1956-196) 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafig Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021a.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint /wt.:ii:000.0288000 2017.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image image pre- training paradigm. arXiv preprint arXiv:2110005200, 2021b.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-War Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 9273-9281. 2020.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, arXiv preprint arXiv:2112.1275:0 2021.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. Advances in neural information processing systems, 30, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint /
2018
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances neural information processing systems, 24, 2011.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingx- ing Tan, and Quoe Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:21 2I1.:2/111.1050.. 2021.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region to-phhase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp- 2641-2649, 2015.
Guodong Qi, Huimin Yu. Zhaohui Lu, and Shuzhao Li. Transductive few-shot classification the oblique manifold. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8412-8422 2021.
11

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Bernardino Comera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learn- ing. In International conference on machine learning, pp. 2152-2161. PMLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), I 15(3):111-252 2015. doi: 10. 10.1007/s| 1263-015-0816-y.
Erich Schubert. A triangle inequality for cosine similarity. In International Conference on Similarity Search and Applications, pp. 32-44. Springer, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia litsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint (rXiv:2111.0211 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
15638-15650, 2022.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8968-8975. 2020.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec 100m: The new data in multimedia research. Communications of the ACM, 54(2)644-73, 2016.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2495-2504 2504, 2021.
Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep pruuctrr-preservingg image-text embed- embed dings. Proceedings of the IEEE conference on computer vision and puttern recognition, pp-
5005-5013, 2016.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp.
PMLR. 2020.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXi 10442, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision 2018.
and pattern recognition, pp.
Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zero-shot classification. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 69-77, 2016.
Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Dision-languas pre-training with triple contrastive learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2022.
12

---

Under review as a conference paper at ICLR 2023
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-ing imnae pre- treetrrining. arXiv preprint arXiv:2111.07883, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432. 2021.
Baoquan Zhang, Hao Jiang. Shanshan Feng, Xutao Li, Yunming Ye, and Rui Ye. Hyperbolic knowl- edge transfer with class hierarchy for few-shot learning. In Proceedings of the Thirty-Fire First Inter- national Joint Conference on Artificial Intelligence, 2022.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 TRAIN HYPER-PARAMERR PARAMETERS:
We train with batch size of 2,048 and the AdamW optimizer in all our experiments. Following the hyper-parameters that improve training stability in the original CLIP (Radford et al. (022I) paper, we set the 31 = 0.9, 32 0.98, 1e-8, and weight decay = 0.5. We also employ the gradient clip operation with global norm of 1.0. We did not employ mixed precision to reduce the possible variance introduced by random initialization. We apply compact training scheme that updates the model for 108,000 iterations, which is roughly equal to training the model for 15 epochs of the dataset, and takes approximately day to train one model using 32 Nvidia V100-16G GPUs. We use the cosine learning rate decay scheme of peak learning rate equal to 5e-4, combined with warmup period of 5,000 iterations. For data augmentation, we only apply the RandomRes Sorrddomeeizedrroo zedCrop with scale range of [0.5, 1.0].
In the linear probe evaluation, the hyperparameters follow the setup of MoCo v3 (Chen et al. 2021b) Concretely, we use SGD without momentum and no weight decay. The learning rate is schemed by cosine decay with peak learning rate equal to 1.0, combined with a warmup period of 5 epochs. We train for 100 epochs and augment the image using the RandomRes sizerrroppppop with a scale range of (0.75, 1.0) and Augouut with the code rand-m9-ms- -0--mm-mmmt..--inn.... 5-inc1.
Temp. Init.
Temp. Final
Converge Step
Zero-Shot
Zero-Shot
Zero-Shot Cls. Acc.
Linear Cls. Acc.
12T R@1
T21 R@1
Topology:
Sphere,
Distance: -u
2.659
4.033
18k
48.3
31.45
30.62
60.38
5.310
4.021
39k
46.8
31.13
29.60
59.82
1.000
3.976
22k
49.0
30.33
28.59
59.56
1.000
1.000
Detach
5.1
3.461
4.04
45.37
Topology: Euclidean,
Distance: w
2.659
2.067
21k
47.9
32.29
30.36
59.90
5.310
5.107
1k
48.5
32.69
30.68
59.74
1.000
1.668
25k
47.4
30.71
29.85
60.09
1.000
1.000
Detach
47.6
30.43
29.51
59.20
Topology:
Ob(64, 8),
Distance: Geo u, v)
2.659
3.135
20k
50.7
32.35
30.79
60.43
5.310
3.168
55k
50.7
31.59
30.34
59.60
1,000
3,024
42k
49.9
32.49
30.21
60.61
1.000
1.000
Detach
4.1
2,921
3.10
21.67
Topology: Ob(64, 8),
Distance: Lr uu"
v)
2.659
2.231
24k
52.3
32.89
30.70
60.32
5.310
2.280
57k
50.3
33.37
30.23
59.76
1,000
2.174
36k
50.9
32.71
30.50
60.66
1.000
1.000
Detach 30.3
18.48
21.20
57.93
Table 5: The retrieval and classification performance of different configurations under different temperature initialization conditions. The performance report in this table is the same as Table but is aggregated by topologies. "Temp. Init." denotes the values for initializing temperature; "Temp. Final" denotes the final temperature at the end of training; "Converge Step" denotes the number of steps for tempearture starts converge (changes less than 2% for an epoch.)
A.2 TABLE 2 FROM THE VIEW OF TOPOLOGIES
In Table we review Table 2 by the topologies. We further provide the final temperature at the end of training and at what step the temperature converges (changes less than 2% for epoch, also see Figure 2] It can be seen that the performance of the Euclidean topology only slightly affected by the iniiialization of the temperatures, and even though the temperature is detached from learning, still performs reasonably well because of the unlimited distance range. While the spherical and oblique topologies affected by how the temperature is initialized. However, rough trend can
14

---

Under review as a conference paper at ICLR 2023
logits_pos
logits_nes __ng
Img_neg
negg_ne
30
0.8
1.0
-10
sphere -
euclidean 12
100
50
50
oblique geodesics (a)
oblique + -
logits_pos
logits_ne __ng
img_neg
next_neg
150
100
50
50
0.4
1.0
-2
sphere <<,>
euclidean
150
50
50
oblique geodesics
oblique
Figure 3: Visualization of the distribution of distances between samples. The logits pos and logits neg denote the distances between positive and negative image-text pairs, respectively. The img_neg and text_neg denote the distances between negative image-image and text-text pairs, respectively. The models are trained using the Yfce datasets, (a) and (b) depict the distribution of in-domain data (Yfec) and out-domain data (RedCaps), respectively.
be seen that the faster the temperature converges, the better performance the model achieves, which means the learnable temperature delays the learning of the methods. The model needs first to find proper temperature and then begin to learn representations well.
A.3 DISTRIBUTION OF LEARNED DISTANCE
We depict the distribution of distance for pairs of samples in Figure 3 the main manuscripts, since the cross-modal contrastive loss does not handle the uni-modal data dis- tributions, the distance between negative pairs of images and texts could be much smaller than that of positive image-text text pair, resulting tighter distance bound. Also, we can see this phenomenon is much more severe in out-domain data, which could reduce the transferability of the feature em- beddings to downstream tasks.
As argued in Section 3.2 of
also notable that, the oblique endowed with the negative inner product the distance function learns similar distributions compared to the sphere reference, while the numerical values distances between samples are inherently larger without having multiplied with temperature.
A.4 ADDITIONAL ABLATION ON OBLIQUE STRUCTURE
We provide more ablation results regarding the structure of the oblique manifold under fixed total di- mensions Ta We can observe that the Ob(32, 32) configuration performs the best in general, while the sphere with 1024-dimensional embedding has slightly better linear probe perfor-
15

---

Under review as a conference paper at ICLR 2023
Zero-Shot 12T R@1
Zero-Shot T21 R@1
Zero-Shot Cls. Acc.
Linear Probe Cls. Acc.
Topology
Distance
Temperature, imit=e', gradient= True
Sphere(512)
uuv
48.3
31.45
30.62
60.38
Sphere(1024)
v
50.7
32.05
29.60
60.53
Ob(128, 8)
-tr(u
49.4
32.85
30.55
60.12
Ob(64, 16)
-tr -tr(u v)
50.3
33.25
30.34
60.16
Ob(32, 32)
- -tr( 2
52.3
33.47
30.62
60.32
Table 6:
The retrieval and classification performance of the proposed approach using different oblique manifold structures and the multi-token implementation. "gradient=(True/Fakes donates the temperature is learnable.
mance. We also notice that more complicated structure provides better text-to-image retrieval results.
A.5
DETAILED MODEL HYPERPARAMETERS OF OUR CLIP MODEL
Hyperparameters
Value for Naive CLIP
Value for CLIP-Mult(11,32)
Batch size
32768
32768
Vocabulary size
30522
30522
Training epochs
32
32
Number [CLS] Tokens
16
Projection dims
512
32
Maximum temperature
100.0
3.95
Weight decay
0,2
0.2
Warm-up up iterations
2000
2000
Adam 31
0.9
0.9
Adam 32
0.998
0.998
Adam
10
10-8
Table 7: Hyperparameters used for training.
A.6 ADDITIONAL RESULTS
We combine our proposed method with the TCL model Yang et al. (2022), which is one of the state-of-the-art vision-language retrieval models that employ contrastive visual-textual alignment in its earlier stage. During the pre-training, the TCL induces mixture of in-modal and cross- modal contrastive losses, while conducting the masked language modeling (MLM) and image-text matching tasks simultaneously. During the testing, the modsl contrastive alignment head first lists sample pairs with high similarity scores, then these pairs are fed into the matching head to obtain the final matching scores. We alternate the topologies of all the embedding spaces with Ob(128,2) more precisely, we change the normalization function as shown in Section 3.1 For the experimental analysis in this subsection, we follow the configurations of the reference models, employ collection of CC3M (Sharma et al./ 2018). Coco captions (Chen et al. 2015), Visual genome (Krishna et al., 2017) and SBU (Ordonez al. [01T) as the pre-training dataset, which contains roughly million annotated image-text pairs. The models are then evaluated using Flickr30k (Plummer et al. 2015)
and Coco captions (Chen et al. 2015).
The results are shown in Table 8. Since our method does not affect the matching head, we also report the performance of the contrastive alignment head. In general, our method improves the average recall performance, but the improvement is not significant. We consider the reasons as i) The method (or recent similar methods) employs pre-trained vision and language models, as well as matching head and an MLM head, hence it is less sensitive the gradients from the contrastive alignment; ii) The datasets employed for training contain less noise, while the training is scheduled with overlength scheme (the zero-shot performance does not increase in the last epochs).
16

---

Under review as a conference paper at ICLR 2023
Flickr T21
Coco T2I
Method baseline[impl.]
12T
Recall
12T
Recall
R@I
@@
mean
R@1
R
mean
Zero-shot performance.
93.00
79.60
93.97
71.40
53.50
79.49
TCL[official]
(84,20)
(67.10)
(88.45)
(55.40)
(40.80)
(69.92)
91.00
78.28
93.25
70.16
53.05
79.07
TCL[our-impl.1
(83.30)
(68.40)
(88.73)
(57.34)
(43.21)
(71.31)
91.20
78. 14
93.29
70.14
53.35
79.14
TCL(06(128,2)
(84.80)
(67.86)
(88.84)
(57.10)
(43.13)
(71.32)
Fine-tuned performance.
94.90
84.00
95.57
75.60
59.00
82.87
TCL[officia]]
(87.90
(71.38)
(90.92)
(65.34)
(48.94)
(76.53)
93.80
83.06
95.17
73.56
57.74
82.06
TCL[our-impl.]
(88.30)
(72.94)
(91.27)
(66.98)
(50.34)
(77.43)
93.80
82.90
95.18
74.78
57.72
82.13
TCL(Ob(122,221
(88.60)
(73.26)
(91.39)
(65.60)
(49.83)
(76.86)
Table 8: Retrieval performance on Flickr and Coco our implemented TCL model and the variant using proposed method. The numbers in brackets are the performance obtained using the con- trastive alignment head.
Additional Notes on TCL We also provide the comparison results with officially released check- points. It can be seen that our implementation performs 0.5-1.0% worse than the official check- points. On the other hand, our implementation has better alignment head performance. Since we employing the codes released in the official repository, the reason might be the following: i) Datasets difference, that we have ~3000 fewer images in the SBU dataset while owning 5000 more images in the CC3M dataset; ii) We resize the CC3M dataset to short edge 500 pixels, while the of- ficial repository does not clearly provide the pre-processing approach; iii) We implicitly have short training time smaller matching loss weight than the official checkpoints due to the difference in the framework.
17


Reviews:
[{"review_id": "pnWtjW67WWW", "reviewer": "Reviewer_bLt8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work strives to learn a visual-textual embedding space from weakly label image-text pairs from the internet. \nIn a typical contrastive learning with cosine-similarity, the noise in training data prevents the deep network to produce a solution with alignment-uniformity, and the network achieves a suboptimal system of equilibrium instead. This work resorts to oblique manifold as embedding space that tackles the equilibrium problem. Empirical improvement on zero-shot image to text retrieval task is observed.", "summary_of_the_review": "This paper proposes to learn visual-textual embedding space on oblique manifold. The idea is interesting and novel, and the paper is well written. The reviewer feels pleasant to read the paper. Waiting to see the clarification of motivations and some statements.", "strength_and_weaknesses": "+ In general, this paper is easy to read. Formulation of the problem and description of the method are clear and rigorous.\n+ The idea of using oblique manifold to learn a visual-textual embedding space is novel.\n+ The finding on learnable softmax temperature is a distance scaling factor of contrastive learning on noisy dataset is interesting.\n\n- Missing some related literatures such as hyperbolic embedding space.\n\nThe motivation in choosing oblique manifold is a bit weak to some extent. The main arguments are 1) its geometry is spherical, where proper definition of uniformity is available, and 2) inner product saves computational cost.\n- Why the uniformity in embedding space is a valid goal to embed text, given the structure of text can be hierarchical?\n- While it is true that unbounded Euclidean space does not have proper uniform distribution defined (Sec 3.2 ii), no neural network is able to learn in unbounded space. We can simply clip to bounded range where uniform distribution is available.\n- In terms of computational cost, do authors take into consideration l2 normalization steps in their algorithm?  \n- Also, is the computation of distance function a bottleneck for optimizing underlying deep learning models? It sounds like a these difference in computational cost are marginal.\n- The statement of \"numerical values of distances at equilibrium might be out of the distance range (e.g. [-1,1] for the cosine similarity)\" is somehow unclear for me. Any two vectors naturally have a cosine similarity value constrained in [-1,1]. Why it might be out of range?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pIcmH-fepPU", "reviewer": "Reviewer_YGrh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a strategy to make the temperature learnable. ", "summary_of_the_review": "This method is too simple and lacks novelty. The authors impose the proposed method related to manifold learning, but it is not well proven and discussed theoretically. The experiments cannot well demonstrate the effectiveness of the proposed method.", "strength_and_weaknesses": "Strength:\nThis paper proposed a simple strategy to make the temperature learnable. The proposed method simply changes two lines of the training codes to improve the performance.\n\nWeakness:\n1. This paper is badly written and hard to follow.\n2. The key idea of the paper is to make the temperature learnable, which is straight and lacks novelty.\n3. The proposed method is pretended too much. The authors impose the proposed method related to manifold learning, but it is not well proven and discussed theoretically.\n4. The experiments cannot comprehensively demonstrate the effectiveness of the proposed method. From the experimental results, one could see that the proposed method could only improve slightly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "okKSr0h6AXH", "reviewer": "Reviewer_W5p4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a new distance metric called Oblique(d/m, m) instead of the fashionable cosine similarity for the contrastive learning paradigm. To verify the effectiveness of the proposed simple method, the authors perform extensive experiments. ", "summary_of_the_review": "The paper is not well written and hard to follow. The experiments are not solid for comprehensively evaluating the proposed method.", "strength_and_weaknesses": "##Strength：\nThe proposed new distance metric is neat and could be easily used for the contrastive learning paradigm.\n\n##Weaknesses:\nAlthough the proposed method could be easily generalized to the existing multi-modal pretraining works using the contrastive learning paradigm, the effects cannot be verified well in the provided experiment results. As shown in Table 4, using the proposed metric only brings limited retrieval performance improvement compared to the vanilla baseline. The authors could provide more results to verify and defend the superiority of the proposed method.\n\nAlthough the technical implementation of the proposed method is clear, there are some statements and arguments needed to be further claimed. \n- “Here, the problem is that the numerical values of distances at equilibrium might be out of the distance range (e.g. [-1, 1] for the cosine similarity).” Why the numerical values of distances could be out of the range of [-1, 1] for the cosine similarity?\n- “For instance, if there is a reasonable amount of false negative samples, then the model would learn a smaller negative distance for not being punished too hard when encountering false negative samples.”\n\n- The reviewer finds that the font of the paper is different from other submission manuscripts. Does the paper use the proper template?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "gyx_Dc999FE", "reviewer": "Reviewer_NGQV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to employ the oblique manifold as the embedding space for contrastive visual-textual alignment learning, where the feature representations are mapped onto the oblique manifold endowed with the negative inner product as the distance function. Discussions and experiments on multiple different topologies endowed with different distances demonstrates the superiority and effectiveness of the proposed approach.", "summary_of_the_review": "Overall, this paper utilized a new distance function for contrastive visual-textual learning, although simple and effective, this paper does not provide new things up with Oblique Manifold, and it lacks deeper analysis/demonstration on how the method solves existing problems. ", "strength_and_weaknesses": "Strengths:\n- The proposed algorithm is easy to implement and proved effective with multiple datasets.\n- This paper provides detailed discussion on the properties of the embedding topology and make fair comparison with other topologies including Sphere, Euclidean, and Oblique.\n\nWeaknesses:\n- In Introduction, the authors discussed the noise problem in contrastive alignment learning, and it seems that the proposed embedding space would handle the problem better than others. However, it may lack the relevant discussions in the body part or experiments.\n- This paper claims that “the sphere and oblique manifold have a proper uniform distribution defined as the surface area measure”, which might be the reason why Sphere and Oblique performs better than Euclidean. However in Table 2, it seems that the conclusion would be largely affected by the Temperature init. \n- It may lack the discussion/visualization on the comparison of the learned distribution with Sphere and Oblique, which should give a more clear explanation on why Oblique performs better than Sphere.\n- Oblique Manifold has been utilized in other feature learning tasks, such as [i] and more discussions should be added.\n[i] Transductive Few-Shot Classification on the Oblique Manifold, ICCV 2021.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject