InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution
Authors: Xiangru Jian
Yimu Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-Modal Retrieval
Data Degeneration
Graph Convolution
TL;DR: In this research, we articulate the data degeneration issue within the gallery set found in cross-modal retrieval, introducing a novel post-processing method to mitigate it. Consequently, we witness a substantial enhancement in retrieval performance.

Abstract: Over recent decades, significant advancements in cross-modal retrieval is mainly driven by breakthroughs in visual and linguistic modeling. However, a recent study shows that multi-modal data representations tend to cluster within a limited convex cone (as representation degeneration problem), which hinders retrieval performance due to the inseparability of these representations. In our study, we first empirically validate the presence of the representation degeneration problem across multiple cross-modal benchmarks and methods. Next, to address it, we introduce a novel method, called InvGC, a post-processing technique inspired by graph convolution and average pooling. Specifically, InvGC defines the graph topology within the datasets and then applies graph convolution in a subtractive manner. This method effectively separates representations by increasing the distances between data points. To improve the efficiency and effectiveness of InvGC, we propose an advanced graph topology, LocalAdj, which only aims to increase the distances between each data point and its nearest neighbors. To understand why InvGC works, we present a detailed theoretical analysis, proving that the lower bound of recall will be improved after deploying InvGC. Extensive empirical results show that InvGC and InvGC w/LocalAdj significantly mitigate the representation degeneration problem, thereby enhancing retrieval performance.

INVGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution
Xiangru Jian and Yimu Wang" University of Waterloo
(xiangru. viian,yima wang)
Abstract
Over recent decades, significant advancements in cross-modal retrieval are mainly driven by breakthroughs visual and linguistic mod- eling. However, recent study shows that multi-modal data representations tend to clus- ter within limited convex cone (as represen- tation degeneration problem), which hinders retrieval performance due to the inseparability of these representations. In our study, we first empirically validate the presence of the repre- sentation degeneration problem across multiple cross-modal benchmarks and methods. Next, to address it, we introduce novel method, called INVGC, a post-processing technique in- spired by graph convolution and average pool- ing. Specifically, INVGC defines the graph topology within the datasets and then applies graph convolution in subtractive manner. This method effectively separates representations by increasing the distances between data points. To improve the efficiency and effectiveness of INVGC, we propose an advanced graph topol- ogy, LOCALADI, which only aims increase the distances between each data point and its nearest neighbors. understand why INVGC works, we present detailed theoretical analy- sis, proving that the lower bound of recall will be improved after deploying INvGC, Exten- sive empirical results show that INVGC and IN- vGC w/LoCALADI significantly mitigate the representation degeneration problem, thereby enhancing retrieval performance. Our code is available at link.
-252
621
(a) Original representations.
(b) Updated representations by INVGC
0.815
Suw 0.077
0.131
Counts
0.537
Coupts
mingge
0.4
0.6
0.8
1.0
0.4
0.S
0.5
1.0
Cosine Similaiitt
Cosine Similairty
(c) Histogram the similarity (d) Histogram of similar- hetween nearest neighhors.
ity between nearest neighbors after INVGC.
Figure I: A 2D visualization of the representation of text data uniformly sampled from the MSCOCO dataset gen- erated by CLIP (Radford al., 2021) and then reduced by PCA. As shown in Figure 1a, the representation de- generation problem can be observed as the represen- tations concentrated in small convex cone, while after employing INvGC, representations are better sep- arated. jeims Tsim. and mingim. are the mean, standard deviation, and minimum of the similarity. Adeg is the score of representation degeneration problem defined in Equation (2)..
map data from different modalities into a common space where they can be directly compared. For example, in text-to-video retrieval, the objective is to rank gallery videos based on the features of the query text. Recently, inspired by the success in self self-supervised learning (Radford et al., 2021), significant progress has been made in CMR, in- cluding image-text retrieval (Radford al., 2021; Li al., 2020; Wang et al., 2020a), video-text re- trieval (Chen et al., 2020; Cheng et al., 2021; Gao et al., 2021; Lei et al., 2021; Ma et al., 2022; Park et al., 2022; Wang et al., b022a,b; Zhao et al., 2022;
1 Introduction
Cross-modal retrieval (CMR) (Wang et al., 2021: Yu et al., 2023; Kim et al., 2023), which aims to enable flexible retrieval across different modalities, e.g., images, videos, audio, and text, has attracted significant research interest in the last few decades. The goal of CMR is to learn pair of encoders that
Corresponding author.

---

Wang and Shi, 2023; Wang al., 2023), and audio- text retrieval (Oncescu al., 2021), with satisfac- tory retrieval performances.
wal et al., 2023). might lead the emergence of representation degeneration problem. Drawing inspiration from the graph convolution and average pooling, we propose a novel method, INVGC, which separates representations by per- forming the graph convolution inversely to separate representations with bigger margin as shown in Figure la. Specifically, different from the vanilla graph convolution, considering one modality, IN- vGC separates representations by subtracting the representation of the neighboring nodes from each node, instcad of aggregating them as,
However, Liang et al. (2022) demonstrate that the current common learning paradigm CMR leads to the representation degeneration problem, which concentrates all the representations in a small (convex) cone (Gao et al., 2019; Zhang et al., 2020) in image-text retrieval and the cosine simi- larity between any two points is positive, as shown in Figure la. Consequently, retrieval performance will be significantly affected (Radovanovic al., 2010; Gao et al., 2019). Due to the limitation of space, detailed related works are presented in Ap- pendix A.
x = - Σ Vi,
(1)
In this paper, to step forward in addressing rep- resentation degeneration problem and further im- prove the retrieval performance, we first empir- ically test whether it is prevalent across various cross-modal retrieval models and datasets, includ- ing video, image, text, and audio. We found that the representations of MSCOCO generated by CLIP are gathered in a very narrow cone in the embed- ding space, proving the existence of representation degeneration problem, as shown in Figure la. This case does not stand alone since similar observations are observed across several cross retrieval models and datasets as shown in Appendix C.3.
where xi' and X₃ are the updated and the origi- nal representations, Sij the similarity between the i-th and j-th data, and r is a predefined hyper- parameter. As shown in Figure I, INvGC better scatter representations and alleviates representation degeneration problem. Moreover, the histogram of similarity between any two points is more balanced with INvGC, as shown in Figure 1. To boost the effectiveness and efficiency of IN INCGC, we propose an advanced adjacency matrix LOCALADI that di- rects INvGC w/LOCALADI to focus on the nearest neighbors of cach data point instcad of considering all the data points.
To evaluate the effectiveness of INVGC and IN- vGC w/LOCALADI, we conducted experiments on eight cross-modal benchmarks (Xu et al., 2016; Chen and Dolan, 2011; 1; Fabian Caba Heilbron and Niebles, 2015; Hendricks et al., 2017: Lin et al., 2014; Plummer et al., 2017; Kim et al., 2019; Drossos et al., 2020). Experimental results show that INVGC alleviates representation degeneration problem across different datasets and methods and improves retrieval performance as a by-product. In summary, our contributions are as follows
Next, model how severe the representation degeneration problem is and the relationship be- tween this problem and the retrieval performance, drawing inspiration from previous work (Gao et al., 2019; Zhang et al., 2020; Yu et al., 2022; Liang et al., 2022; Yang et al., 2022: Huang et al., 2021). we define it in CMR as the average similarity be- tween each point and its nearest neighbor in the gallery set, shown in Equation (2). We ob- serve that the scores are very high across differ- ent datasets and methods. They are able to model this problem as a high score always leads to more concentrated data distribution as shown in Ap- pendix C.3.
We are the first to formalize the definition of representation degeneration in pross-modal re- trieval and perform a theoretical analysis of the relationship between representation degen- eration problem and retrieval performance.
While CMR has suffered from this problem, on the other side, the graph convolution (Kipf and Welling, 2017) and average pooling (Boureau et al., 2010), which are widely employed in graph neural networks (Gilmer al., 2017: Kipf and Welling, 2017: Velickovic al., 2018) and deep neural net- works (He al., 2016; Yu al.,, 2023), respectively, are designed to move the representations closer to each other if they are semantically similar (Baran-
Inspired by the graph convolution, we pro- pose the first post-processing method in cross- modal retrieval, namely INVGC, to alleviate representation degeneration problem without any training process or additional data.
'The code is released at link.

---

We design an adjacency matrix, called Lo- CALADI, for the graph convolution, which leverages only the nearest neighbors of each data point instead of all the data. INvGC with LOCATAADI, namely INVGC w/LooAA.ADe It is shown to be more effective and efficient.
To understand the relationship between the de- generation score (Equation (2)) and the retrieval performance, we present the following theorems.
Theorem 1. Let x1 be any point in G, X2 be the nearest neighbor of x1 and n be the dimension of the representation. Given query point q that is semantically similar X1 and sampled from Q. which follows independent and identical uniform distribution in R", the probability of a query point q to successfully retrieve X1, denoted P(x1, b), is bounded by.
Extensive experiments show that INvGC and INvGC w/LOCALADI alleviate representa- tion degeneration problem and improve re- trieval performance as by-product.
2 Preliminaries
1 b 4
6" P(X1,
2.1 Task Definition
In this paper, we focus on the representation degen- eration problem in cross-modal retrieval (Wang et al., 2020b). Two modalities are denoted as X' and y. X' is the query modality, while is the gallery modality. The (test) gallery, denoted contains all the representa- tions of the (test) gallery data, where Ng is the size. The query set is Q = [qi,..., )N,),, where Ng is the number of queries. Usually, in cross- modal retrieval, the gallery data does not overlap with the training data. Additionally, as INVGC re- quires training (or validation) data to address the representation degeneration problem, we define the set of representations of training (or validation) query and gallery data as = (qi...., .INg))
where b = |x1 -x2112/2
Corollary The ratio between the probability of successful retrieval of any two different neighbor- hood radii, namely b1 and by, is
P(x1, b1) = P(x1, b2)
b1
n
= (g),...,
b2
Due space limitation, the proofs are deferred to Appendix D.I.
Remark 1. These theorems show that high simi- larity of the nearest neighbor, iee, smaller b, leads to an exponentially lower probability for successful retrieval. Therefore, a higher Adeg score leads to bad retrieval performance.
and Ō = (gi,.... @N respectively, where N and N& are the size of the training (or validation) query and gallery set, respectively. The similarity between two embeddings and b is defined as Sab = sa b), where sim(-,-) could be some measure of distance.
3 INVGC
To alleviate the representation degeneration prob- lem and further boost the retrieval performance, we design a post-processing method, called INVGC, which does not need any additional training.
Our idea is generally based on the mathematical principles laid out in Equation (1) and Figure 1, where the inverse version of graph convolution is able to decrease the similarity between data points and their neighbors. For the sake of clear represen- tation,
2.2
Representation Degeneration in Cross-modal Retrieval
Taking inspiration from Liang et al. (2022), we define the representation degeneration problem as the average similarity of pairs that are closest to each other,
the cosine similarity the similarity metric in this section following Luo et al. (2022), as it is the most common practice in cross-modal retrieval?. The formal definition of INVGC is presented in Section 3.3.
A Aeg Σ sim(x, x, y),
where is the closest data point to while m is the number of data points. higher score, as shown in Appendix C.3, indicates more severe representation degeneration issue, with each data point being closer to its nearest neighbors in the set.
3.1 Mathematical Intuition
INVGC originates from graph convolution, which is widely employed in graph neural networks (Kipf
'NNVGC can be casily migrated to other similarity metrics adopted by different retrieval methods (Croitoru et al., 2021; Liu et al., 2019).

---

and Welling, 2017: Gilmer et al., 2017: Velickovic al., 2018).
basic assumption in machine learning (Bogolin et al., 2022). A detailed explanation of the claims is included Appendix B.3. Moreover, as CMR needs to contrast the data points from both modali- ties, we utilize the (train or validation) gallery set G and the (train or validation) query set Q to bet- ter estimate the hidden distribution as shown in Equation (4),
Specifically, graph convolution will concentrate all the embeddings of similar nodes which might lead to the concentration of similarity (de la Pena and Montgomery-SSme SSmthh, 1995) and the data degen- eration problem (Baranwal et al., 2023). On the other side, a similar operation, average pooling, has been employed in computer vision (He al., 2016; Wang and Shi, 2023). Average pooling will aggre- gate the features that are location-based similar'.
x
x; 10-7 rg
Σ XJEG
S
r4 Σ Sixs
(4)
As a result, graph convolution and average pool- ing concentrate the representation of all the similar nodes and force them to become very similar to each other, potentially leading to representation de- generation problem. This observation inspires us to pose the following research question:
where rg and rg are two hyperparameters, S E IR NNN and S E N is the adjacency ma- trices between every pair of embedding from G and G and that from Q and Q, respectively.
To the best of our knowledge, INVGC is the first to utilize the (inverse) graph convolution for separating the representation of data. Instead of the commonly used capability of aggregation and message passing, we introduce an inverse variant of convolution that separates data representations compared to the vanilla graph convolution.
Can the issue of representation degeneration prohlem in cross-modal retrieval be alleviated by conducting inverse graph convolution (INVGC)?
To answer this question, we first give an inverse variant of graph convolution as follows,
x; =
Σ Anzxj
3.2 Constructing Adjacency matrix
Next, we need to establish the graph structure, ie.. build the adjacency matrix S and 59, since there is no natural graph structure in the dataset. The simplest idea will be that the edge weight between and j equals 1, i.c., S.j = 1, if the cosine similarity between X; and Xj. i.e. sim(Xi. Xj). is larger than O (or some thresholds). INVGC with this form of the adjacency matrix is an inverse variant of average pooling. It serves as a baseline in this study, denoted as AvGPOOL.
where A is the adjacency matrix for the data in the gallery set. Since the adjacency matrix is not avail- able, to encourage separating the nearest neighbor in terms of the similarity score of each node, we choose Aig = si == Sij with detailed dis- cussion in Section 3.2. Therefore, based on the inverse graph convolution, we propose the basic setting of INVGC shown in Equation (1), which only considers one modality. We notice that it is able to alleviate the representation degeneration problem as shown in Figure 1.
However, this scheme is not capable to reflect the degree of the similarity between x; and Xj since it cannot precisely depict the relation between dif- ferent data points.
Note that the ultimate goal INvGC is to re- duce Adog score of the distribution of the represen- tation of the gallery instead of merely the gallery set G, which can be regarded only as sampled subset of the distribution with very limited size in practice. Therefore, the best approximation of the distribution is the training (or validation) gallery set & since it the largest one we can obtain'.
As the magnitude of Sij directly controls how far will x; go in the opposite direction of xj. greedy design will be using the similarity score between them as the edge weight, i.c., S.y sim xj).
Therefore, we can calculate the adjacency matrix S", which contains the similarity score between ev- ery pair of embedding from G and G. respectively. Specifically, the (i.j)-entry of S follows,
Similarly, the distribution of the query set Q is theoretically expected to be similar to that of G as
'The details of graph convolution and average pooling discussed in this study are deferred to the Appendices B. and B.2, respectively.
S = sim gi 81)
(5)
"In practice, the test queries are invisible to each other as the queries do not come at the same time. So the size of the query set N2 is equal to 1.
Similarly. the matrix S containing the similarity score between every pair of embedding from G and

---

Q is calculated as follows,
In summary, INvGC, as shown in Equation (9), brand new type of graph operation performed on the specifically designed graph structures (ad- jacency matrix) of data points in the cross-modal dataset. helps alleviate the representation degen- cration problem by separating data points located close to each other in the representation space.
5 sim qj)
(6)
Now, with the well-defined S and 8 we can finally perform INVGC to alleviate representation degeneration problem.
As shown in Theorem 1, given any data point, the similarity of the nearest neighbor in the repre- sentation space is critical retrieval performance. Inspired by this, when performing the inverse con- volution on a node, we force INvGC to pay atten- tion to those most similar nodes to it. This can be achieved by assigning edge weight only to the nodes having the top k percent of the largest simi- larity scores relative to the given node. Specifically, each entry of 8 and S9 in this case is calculated follows,
After obtaining G', the similarity between the i- th gallery points g' and query point q is calculated as sim(q. g').
Experiments
We conduct series of experiments to demonstrate that INvGC can efficiently alleviate the represen- tation degeneration problem in a post-hoc manner. We compare INVGC and INVGC w/LOCALADI with the baseline performance produced by the rep- resentation model adopted. We also introduce the inverse version of average pooling, namely AvG- POOL., as another baseline. A series of ablation studies indicate that INvGC addresses the repre- sentation degeneration problem by reducing the similarity between points in the gallery set and is not sensitive to hyperparameters and the amount of training data.
(i. sim(g Ei) if sim Ei) 2 k))
clse
(i, sim(q: qj)
iifiim(q: qz) 2 k))
clse
(8)
4.1 Experimental and implementation settings
where k)) is the value of k-percentage largest similarity between node i and all the nodes in G. The same approach applies P( k) as well. We denote this refined adjacency matrix as LOCALADJ since focuses on the local neighbor- hoods of each node.
The implementation of INvGC exactly follows Equation (9) with the adjacency matrix defined in Equations (5) and (6) and separate pair of tuned hyperparameters rg and r.g for each retrieval model of each dataset. To balance the edge weight in Sa and S and make sure the scale of weights is stable, they subtract their average edge weights, respectively.
3.3 Formal Definition of INVGC
With the well-formed adjacency matrices, we for- mally define INvGC to obtain the updated embed- ding of G, denoted G', in a matrix form as,
The only difference between INVGC w/LoCaLADI and INvGC is the adjacency matrix applied. Instead of the ones from Equa- tions (5) and (6), apply St and S defined in Equations (7) and (8). We choose the value of be (i.e., top 1% nearest neighbors) throughout the study while the proposed method very robust to the value of the k. There reasonable values that can boost the performance, proved by the ablation study in Section 4.3.
I norm - GG) + norm - (G.n,,sa))) r.) G' -'=5
(9) where uorm(-) normalizes matrix with respect to rows, which is employed for uniformly distributing the intensity of convolution when cosine similar- ity is used and should be removed when adopting other similarity metrics and the adjacency matri- ces S9 and S can be calculated as Equations (5) (7). Note that, compared Equation (4), we sep- arate the convolution on & and Q pursue more robust results, for avoiding the distribution shifts between & and Q due to the imperfection of the representation method.
large range of
AvGPooL is the simplest variant of inverse graph convolution with binary adjacency matrix where the edge weight between pair of nodes is if they are neighbors, or 0 otherwise. Then, fol- lowing the approach in LOCALADI, we update the embeddings using only the top p percent of near-

---

Table Similarity measures within gallery set (MeanSim @ equivalent to Adey on MSCOCO. "MeanSim" and "MeanSim@ee" @k" refer to the mean of the similarity between all data points and that between all data points and their top-k neighbors.
Text-to- -ii-to--iieoooo Retrieval
Video-to- Text Retrieval
MeanSim |
MeanSim@ 11
MeanSim@ 104
MeanSim
MeanSim
MeanSim 10L
CLIP
0.4717
0.8211
0,7803
0,5162
0.9029
0.8396
CLIP INvGC
0.4693
0.8142
0.7738
0.5137
0.8972
0.8542
Difference to the baseline (%)
0.51 4
0.84
0.83 4
0.48
0.63 4
0.63
CLIP w. INvGC w/OCCALDI
0.4646
0.8059
0.7647
0,5105
0.8924
0,8477
Difference the buseline (%)
1.51
1.85
2,00
1.09
1.16
1.38 J
Table 2: Similarity measures between test gallery and query set on MSCOCO. The nearest neighbor is considered with respect to gallery set.
Text-to- Video Retrieval
Video-to- Text Retrieval
MeanSim,
MeanSim@1 4
MeanSim @ 10.,
MeanSimi
MeanSim@
MeanSim @ 10.
CLIP
0.1516
0.3282
0.3138
0.1516
0,3213
0.3009
CLIP INvGC
0.1500
0.3245
0.3101
0.1511
0.3198
0.2994
Difference the haseline(%)
1.06..
1.13,
1.184
0.33 +
0.47.
0.50 +
CLIP w. INvGC w/LOOCLADI
0.0635
0.2214
0,2035
0.0921
0.2414
0.2208
Difference to the baseline(%)
58.11
32.54
35.15 +
39.25
24.87
26.62
est neighbors. To provide more comprehensive benchmark, we pick four values of p from 10% to 100%. Note that also independently tune Tg and ry for AvGPOOL make sure the evaluation is fair. The comparison with AvGPOOL validates only the effectiveness of INvGC but also the im- portance of the similarity- adjacency matrix. The detailed setting is included Appendix C.4.
4.3 INVGC and INVGC w/LOCALADI
In this section, to answer a series of questions relating to INVGC and INvGC w/LOCALADI, investigate its performance MSCOCO with CLIP given different settings. Due to space limi- tations, discussions of the sensitivity of INYGC w/LOCALADI to and the complexity of both methods are included in the Appendix.
RQI: Is the data degeneration problem leviated? This problem can be firstly explained by the change of the similarity measure within the test gallery set G. As presented in Table I, we collect the mean similarity of the gallery set of both tasks for three scenarios, the overall mean (MeanSim), the mean between the nearest neigh- b 1), and the mean between nearest 10 neighbors (MeanSim 10). Note that Mean- Sim @ is strictly equivalent to (G) is very clear that INvGC and INVGC w/LoCALADJ do reduce the similarity on all accounts, especially McanSim @ 1, indicating a targeted ability to allevi- ate the data degeneration issue.
4.2 Datasets and evaluation metries
While we mainly focus our experiments on stan- dard benchmarks for text-image retrieval, i.e.. MSCOCO (Lin et al., 2014) and Flickr30k (Plum- mer al., 2017), also explore the generaliza- tion of INvGC by selecting four text-video re- trieval benchmarks (MSR-VTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011), Didemo (Hen- dricks et al., 2017), and ActivityNet (Fabian Caba Heilbron and Niebles, 2015)) and two text- audio retrieval benchmark (AudioCaps (Kim et al., 2019) and CLOTHO (Drossos et al., 2020)). The details of the benchmarks deferred to Ap- pendix C.I.
Besides, given the assumption that the distribu- tion of the test query set Q is close that of G, the similarity score of the gallery both tasks is also worth exploring. Since any point in should theoretically share an identical repre- sentation with its corresponding query in Q and we try to maximize the similarity between them, we
To evaluate the retricval performance of INvGC, we use recall at Rank K (R @R@K. higher is better), median rank (MdR, lower better), and mean rank (MnR, lower is better) as retrieval metrics, which are widely used in previous retrieval works (Luo et al., 2022; Ma et al., 2022; Radford et al., 2021).
the query set for

---

and tune the other show the change on the R@) metrics. The results of the MSCOCO datasct are shown in Figure 2. Although subject to some varia- tion, INvGC constantly outperforms the baseline, which is presented as the red dashed line. This in- dicates that the proposed method can consistently improve performance with a very large range of parameters.
T21
ImGC f/- CUP (MSCOOO)
T2L
i 6,051 CLP (NSCOco)
5.00
0.52
0.06
5.06
6.50
0:50
(a) Image-to-Tedd R@1 w.rl (b) Image-to to--oo--oee Text R@ w.r.t
52.0
51.0
IZE R81
ImvG fr- (MSCOCD)
12T
50.5
Cyy
RQ3: How much data is needed for both pro- posed methods? Since we use the training query and gallery set as a sampled subset from the hidden distribution of representation, it is quite intuitive to ask if the performance of INvGC or INvGC w/LLoCALDD is sensitive to the size of this sam- pled subset. Therefore, we uniformly sample dif- ferent ratios of data from both the training query and gallery set at the same time and evaluate the performance of both methods with the identical hy- perparameters, as presented in Figure From the results, we conclude that both proposed methods perform stably and robustly regardless of the size of the training data.
(MSCOCC)
$8.0
0.000
0.025
8,050
8.075
0.0
0.5
1.0
15
25
(c) Text-to-Image R@1 wart (d) Text-to-lmage R@1 w.r. 19
Figure 2: Hyperpapameters sensitivity of INVGC on
MSCOCO.
52.5
33.1
33.0
127 12T881
52.0
32.9
51.5
32.7
Ratio training
training
(a) Image-to-
Text Text R@ w.r. (h) Text-to-Image R@ w.r. the size data we use. the size of data we use.
RQ4: Is INYGC w/L.OAALAJJ sensitive to the hyperparameter k?? To address the ques- tion, we evaluate the R@I 1 metries of INVGC w/LoCALADI with very large range of possi- ble values (even in logarithmiic scale) compared to the optimal choice adopted(i.e..k = 1). For each task, we fix everything except the A value.The re- sults of MSCOCO dataset are shown in Figure 4. Compared with the baseline, it is safe to claim that the proposed INVGC w/LOCALADI is very robust to the choice of k.
Figure 3: Image-text retrieval performance w.r.t the size on MSCOCO.
exclude this similarity score between them. Con- sequently, we want the similarity to be as small as possible since it reflects the margin of the retrieval task, as a gallery point is supposed to be as far from the irrelevant queries as possible to have a more robust result. We adopt the same metrics as Table 1, and the results are presented in Table 2. Again, we observe comprehensive decrease in the similarity score, especially between the nearest neighbors. Note that, compared to INVGC, INVGC w/LoCALADI can better address the representation degeneration problem, validating that the design of LoCALADI does help alleviate the problem by focusing more on the local information. Not for MSCOCO alone, we witness exactly similar results across all the datasets and methods, whose detail is included in Continuation on RQ1 Appendix C.7. RQ2: Is INVGC (or INVGC w/LLCAL.AD.) sensitive to the hyperparameter y and r7? To answer the question, we evaluate the R@ metrics of INC with a very large range of hyperparame- ters respective to the optimal choice adopted. We defer the analysis of INVGC w/LOCALADI to Ap- pendix C.7. For each task, we fix one of rg or rig
CLP//SCOCOO
CLIP(MSCOCO)
52.0
0 $1.5
32
2 50.5
51.
baseine
8 31
base
Ing6C WiLocalAd
IngaC WiLocalAd
50.0
10
10
10
20
k(%)
(a) Image-to-Ten R@ 1 w.r. (b) Text-to- Image R@1 wan the value of we use.
the value of k we use.
Figure 4: The value of k of INVGC w/LOCALADI MSCOCO.
4.4 Quantative results
In this section, we present the quantitative results of four cross-modal retrieval benchmarks. Across eight different benchmarks, INVGC and INVGC w/LoCALADs significantly improve upon the base- lines, demonstrating the superiority of our pro- posed methods.

---

Table 3: Retrieval performance on MSCOCO (5k split). Best in Bold and the second best is underlined.
Table 6: Retrieval performance on ActivityNet. Best in Bold and the second best is underlined.
Normalization
R@51
Text-to Image Retrieval
R@10
MoR
Normalization
Text-to-Video Retrivaal
R@10
MdR
MoR
CLIP
*ANGPooLiraso-DJ
30.37
30.34
54.74
54.77
66.08
66.14
4.0
4,0
25.36
25.39
CLINClip
AVGGPooL (ratice0 ()
41.85
41.83
74.47
74.44
84.84
84.84
2.0
2.0
6.84
6.84
+AvePoni (ms
30:37
54-77
66.
4.0
25.36
41.80
74.44
$4.84
2,0
6.84
.AvGPooLiiaab--0.5
30,38
54.77
66.10
4,0
25.38
-AVGPooL/ratice0..
41.85
74.44
84.84
2.0
6.84
-AXGPooL(ratsel)
51.77
66.11
*AYGPoOL(raisee)
41.88
74.44
$4.84
2,0
6.81
-INVEIC:
32,70
57.53
68.24
4.0
24.35
+INVCC:
41,90
74.40
84.86
2.0
+INVGC w/LoCALADI
33.11
57.49
68.19
4,0
28.95
+INVGC www.I ANM
43.25
75.58
X5.74
2.0
6.82
Ovear
52,52
52.50
80.04
80:01
87:95
87.96
1.0
1.0
10.70
10.68
X-CLIP
AvGPoot (catim0)1)
46.25
46.47
76.02
75.94
86.05
36,05
2.0
2.0
6.37
6.38
*AYGPooL/rais0022 25)
80:00
87.96
10.68
AVGGPoo (rf
46.35
75.98
$6.01
2.0
6.38
+AvoPont (mtiee))
+AvaPoni (mi
52.51
52.50
80100
80.02
87.96
N7.96
1,0
1.0
10:67
10.68
(eaiie0))
46.47
75.94
86.01
2,0
6.38
IINVCC
52.63
80,05
N7.96
1.01
10.72
-AVGPooL/ratioe))
46.43
75.89
86.01
2.0
6.38
-IXVGC w/LOCALAD/
$2,93
80.05
87.78
1.0
11.09
-INVOC
46.43
75.68
36.22
2.0
6.35
-INVOC w/LOCALAD
47.82
76.46
86.36
2.0
6.91
Table 4: Retrieval performance on Flickr30k. Best in Bold and the second best is underlined.
Table 7: Retrieval performance MSVD. Best in Bold and the second best is underlined.
Test-lo-lmng Retricval
Normulization
R61
RGi5
R@:10
MdR
MrR
Normalization
Text-to-
Video Retrieval
R58
R
Roe I11
Mcl
MnR
58,98
83:18
90.14
1.0
6.04
CLIP
K3.56
90..18
1.0)
6.144
CLIPSClip
44.64
74.66
2.0
10.32
+AvGPooLiszio-
59.10
83.56
90.18
1.0
6.04
-AVGPooL/ratioo..] I)
41.87
73.89
83:07
2.0
11.93
+AvePoni (mikee).)
59.10
83:54
90.18
1.0
6.05
AAvePom (malion 25)
45.06
74.04
83.49
11.29
-AvGPooLinsto-1)
59:10
90.18
1.0
PvvG (maine0))) 5)
45.12
74.32
83.66
2.0
10.76
+INVEIC:
60:18
85. 30.30
91.20
1.00
5.52
.VVoPooL.tatio-)
45.09
74.54
83.91
10,45
-INVGC w/LoCALADI
60.45
85:30
91.10
1,0
5.59
+INVGC
15:43
74.82
84:00
2.0
10:12
+NNVCC w/L.DAAAAAI
45.73
75.53
84.37
2.0
10.42
71.60
91:50
94.95
1.0
4.24
Owear
71.62
91.44
94.92
1.0
4.25
CLIP2Video
47.05
76.97
85.59
2.0
9.53
+AvGPooLimais022 25)
91.50
91.94
47.04
76.98
85.61
2.0
9.54
+AvePoni (miteel).) 5)
71.66
91.50
94.92
1.0
4.24
Ara 25)
47.07
*AVGPooLinzio-1>
71.62
94.96
1.0
Avc (miii-00.5)
47,06
76.99
85.62
2.0
9.53
+INVGC
71.68
91.16
95.06
1.0
1.25
AvecPor (mtio=1)
47,06
76.97
.INVEIC A/L
71.74
91.56
94.58
1.0
4.29
+INVGC
47,09
77.00
85.64
2.0
9.48
+INVGC W/LOCALAD/
47,47
77.46
85.84
2.0
9.53
46.31
76.84
85.31
2.0
9.59
Table 5: Retrieval performance on MSR-VTT (full split and Ik split). Best in Bold and the second best is underlined.
X-CLIP
AvG (tato=-00))
46.36
76.70
85.28
2.0
9.66
*AVGPoLL/aaii 25)
46.33
76.71
AvcPon (mlinn))))
46.35
76.71
85.25
2.0
9.62
AvGPooo (maioo))
46.41
INVICC
46.82
76.69
85.38
2.0
9.65
+INVGC W/LOCALAD/
16,19
76.82
65.29
2.0
9.63
Test-tt- Videa Retriccal
Normalization
R@1
RG5
R@:0
McR
MnR
CLIP
44.10
21.70
81.40
2.0
15.51
Table 8: Text-audio retrieval performance on CLOTHO.
AvGPonLiraticen
44.20
71.60
81.50
2.0
15:55
Best in Bold and the second best is underlined. *** refers to the results direct copied from (Koepke et al.,
AvcPoooLtatioo- 25)
44.20
71.60
81.50
2.0
15:55
Avc (5a
44.20
71.50
81.50
2.0
15.54
1NVGPooLiratios])
44.10
71.60
$1.50
2.0
2022). MoEE (Miech et al., 2020), MMT (Gabeur et al.,
-INVGC
44.40
71.90
81.60
2.0
15.36
INVGC WLLCAALADI
44.40
71.70
81.20
15.65
2020) two methods used for audio-tex text retrieval.
CLiP2Video
46.00
71.60
81.60
2.0
14.51
-VGPPoLiraiic0O.) 1)
45.90
71.70
81.50
2.0
14.52
Test-to Tett-to-Andio Recioval
(tatioo 25)
46.10
71. 71.80
81.50
2.0
14.53
Method Nommalization
R@1
R#51
R
MdR
MnR
-vcGPon (ration0)))
46,00
71. 71.76
81.50
2.0
14,53
]VVCPooLtratio-))
45.90
71.70
81.50
2.0
14.52
MoEE*
6:00
20.80
32.30
23.0
60.20
-INVGC
16.20
71.70
81.30
2.0
14.44
MMT
6.50
21.60
66.90
23.0
63.70
-INvG w/l.orAL.ADI
46.60
72.10
81.70
14.50
6:27
23.32
33.30
23,0
5k.96
AR-CE
+ ()
6.37
22.33
33.59
23.0
65.57
X-CLIP
AvoPooLLratiien .
25)
46,50
46.30
46.40
74:00
74.00
74.00
83.40
83.40
83.50
2.0
2.0
2.0
12,80
12.88
12.85
+AVGPooLiraiiee))
AvG (neiun0)5)
AvvGPoLLLnaio00225
6.47
6.28
23.34
22.12
22.32
33.30
33.58
33.30
22.0
22.0
23.0
59.84
50.14
58.95
16.30
74.00
83.30
2.0
12.63
4NNYCC
6.11
22.14
34.72
21.0
61.98
AvciPonn
46.20)
74.00
85.40
2.0
12.85
+INvCC A/
22.35
34.64
58.51
-INYGC
47,30
74.00
83.30
2.0
13.42
-INvGC WLLOCALADI
47.10
74.20
83.50
13.09
the baselines on both the MSCOCO and Flickr30k datasets, improving R@1 and R@5 by at least 2% compared to all baselines.
Text-Image Retrieval. Results are presented in Tables 3 and 4. We observe that one of our methods achieves the best performance on R@ 1, R 55, and R@ 10 by a large margin. When evaluated on the CLIP method, INVGC w/LoCALAD! outperforms
Text- Video Retrieval. Results are presented in Tables to 7. We can also conclude that one of our methods achieves the best performance on R @@ 1,

---

First, although INVGC has been validated through theoretical analysis and demonstrated its efficacy in separating representations, its performance may Table 9: Texi-audio retrieval performance on Audio- Caps. Best in Bold and the second best is underlined. "*' refers to the results direct copied from Koepke et al. (2022). MoEE (Miech et al., 2020), MMT (Gabeur al., 2020) are two methods used for audio-text retrieval.
ments on various cross-modal benchmarks showed that both of our methods successfully alleviated the problem of representation degeneration and, as a result, improved retrieval performance.
Limitations
Tedi-i-Aadio Retrioval
Methed
Normalization
RG
R85 1 R0:00
MdR
MaR
MoBE"
23:00
55,70
71.00
16.30
MMT
36.10
72.00
81.50
2.3
7.50
22.33
54,49
20.54
15.89
AR-CE
AVGPoor (nsiumt)
22.
15.89
AvG 25)
22.35
$1.49
70.50
5.0
15.89
vary actoss different datasets and modalities. The effectiveness of our method might be influenced by variations in dataset characteristics, such as data distribution, scale, and complexity. Further inves- tigation and experimentation on a wider range of datasets are needed to fully understand the gener- alizability of INVGC and its performance under diverse conditions.
22.35
51.69
70.51
15.89
+AvGPoor./msiie
22.33
54.49
15.89
+IKVGC
22.35
$4.46
70.56
5.0
15.89
+INVCI W|LDCALADI
24.117
R@5, and R@ 10. Specifically, the ActivityNet dataset, INvGC w/LoCALADJ shows excellent per- formance with both CLIP4CLIP and X-CLIP meth- ods, significantly outperforming all the baselines on R@1 and R @ roughly by 2%.
Second, while our method shows promising sults in alleviating the representation degeneration problem, it is worth noting that cross-modal re- trieval tasks can still pose challenges due inher- ent differences in modalities. Variations feature spaces, data modalities, and semantic gaps between modalities may limit the overall retrieval perfor- mance. Future research efforts should focus on exploring complementary techniques, such as mul- timodal fusion, attention mechanisms, domain adaptation, to further enhance the retrieval accuracy and alleviate representation degeneration problem. In the future, it would be interesting to explore the performance of INvGC on diverse datasets, the challenges associated with cross differences, and better definitions or metrics for measuring rep- resentation degeneration problem.
Text-Audio Retrieval. Results are presented in Tables 8 and 9. On the CLOTHO dataset, INvGC exhibits significantly better performance compared all the baselines while INVGC w/LoCALA achieves the best results on the AudioCaps dataset.
In summary, our experiments demonstrate that employing INVGC consistently improves retrieval performance across different datasets and retrieval tasks. The models with INVGC demonstrate better accuracy and ranking in retrieving relevant vidcos, images, and textual descriptions based on given queries. The complete results of retrieval perfor- mance can be found in Appendix C.6.
5 Conclusion
This paper addressed representation degeneration problem in cross-modal retrieval, which led to a decrease in retrieval performance. The represen- tation degeneration problem was validated across multiple benchmarks and methods. To alleviate this issue, we proposed a novel method called IN inspired by graph convolution and average pooling. The method established a graph topology structure within the datasets and applied graph convolution in an inverse form with subtraction over the neigh- borhood. Additionally, we designed the adjacency matrix, LOCALADI, that only leveraged the nearest neighbors of each data point rather than the entire dataset, resulting in a more effective and efficient method, INVGC w/LOCALADI. Both INVGC and INVGC w/LOCALAADI were validated through the- oretical analysis and demonstrated their ability to separate representations. Finally, extensive experi-
References
Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. 2023. Effects of graph convolutions in multi-layer networks. In The Eleventh International Conference on Learning Representations.
Richard Bellman. 2015, Adaptive Control Processes A Guided Tour (Reprint from 1961), volume 2045 of Princeton Legacy Library. Princeton University Press.
Simion- Limion-Vlad Bogolin, Ioana Croitoru, Hailin Jin. Yang Liu, and Samuel Albanie. 2022. Cross modal retrieval with querybank normalisation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans. LA, USA, June 18-24, 2022, pages
IEEE,
Y-Lan Boureau, Jean Ponce, and Yann LeCun. 2010.
A theoretical analysis of feature pooling in visual

---

recognition. In Proceedings of the 27th International Conference on International Conference on Machine Learningg ICMI' 10, page 111-118, Madison, WI, USA. Omnipress.
on Computer Vision and Pattern Recognition, pages 961-970.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK August 23-28, 2020. Proceedings, Part IV, volume 12349 of Lecture Notes in Computer Science, pages 214-229. Springer.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs.
David Chen and William Dolan, 2011. Collecting
highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association Computational Linguistics: Human Language Technologies, pages 190-200, Portland, Oregon, USA. Association for Computational Lin- guistics.
Zhe Gan, Yen-Chun Chen, Linjic Li, Chen Zhu, Yu Cheng. and Jingjing Liu. 2020. Large-scale ad- versarial training for vision-and-language representa- tion learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 6-12, 2020, virtual.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. 2020. Fine-grained video-text retrieval with hierarchical graph reasoning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10635-10644. Computer Vision Foundation / IEEE.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration prob- lem in training natural language generation mod- els. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA. May 6-9, 2019. OpenReview.net.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang. and Dong Shen. 2021. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. CoRR, 0ats/2199.0290.
Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. 2021. CLIP2TV: an empirical study on transformer-basee methods for video-text retrieval. CoRR, abs/2111.05610.
Ioana Croitoru, Simion- Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu, 2021. Teachtext: Cross- modal generalized distillation for text-video retrieval. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 11563-11573. IEEE.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Ri- ley, Oriol Vinyals, and George Dahl. 2017. Neu- ral message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 1263-1272. 163--1222JMMLR.o00 JMLR.org.
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. 2013. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 35( (5(11)):99162929
Victor de Pena and S. Montgomery- Smith. 1995, Decoupling Inequalities for the Tail Probabilities of Multivariate U-Statistics, The Annals of Probability. 23(2):806 816.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www deeplearningbook.orr
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Key- van Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022. X-pool: Croas language-video attention for text-video retrieval. IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans. LA, USA. June 18-24, 2022, pages 4996-5005. IEEE.
Konstantinos Drossos, Samuel Lipping, and Tuo- mas Virtanen. 2020. Clotho: an Audio Cap- tioning Dataset. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740. ISSN: 2379-190X.
Simple GraphConv. Pyg description simple graph convolution. http::://pytorcch-gometttt readthedocs. readthedocs..ooee/lllaa wwtpiii///eeeeeeee wwweee/laattttggeeeeeeeeeeeeeeee http///ttttttteettte///totttttt
geometric nn. conv. SimpleConv. html#torch_ geometr commetric.nn con.v............. Sen..SimpeeCon. 2023-06-17.
Accessed:
Bernard Ghanem Fabian Caha Heilbron, Victor Escor- cia and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity un- derstanding. In Proceedings of the IEEE Conference
William L. Hamilton. 2020. Graph representation leam- ing. Synthesis Lectures on Artificial Intelligence and Machine Learning, 14(3):1-159.

---

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 IEEE Conference Computer Vision and Pattern Recognition. CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
sparse sampling, In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual. June 19-25, 2021, pages 7331-7341, Computer Vi- sion Foundation IEEE.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre- training for vision-language tasks. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 121-137. Springer.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell, 2017. Localizing Moments in Video with Natural Lan- guage. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5804-5813, Venice. TEEE.
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. 2021. Learning with noisy correspondence for cross-modal matching. In Advances in Neural Information Processing Systems, volume 34, pages 29406-29419, Curran Associates, Inc.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: Un- derstanding the modality gap in multi-modal con- trastive representation learning. In Advances in neural information processing systems.
Eamonn Keogh and Abdullah Mueen. 2017. Curse of Dimensionality, pages 314-315. Springer US, Boston, MA.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014- 13th European Conference, Zurich, Swizzerland, September 6-12, 2014, Proceedings. Part V. volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. 2019. AudioCaps: Generating captions for audios the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 119-132, Minneapolis, Minnesota. Association for Computational Linguis- tics.
Venice Erin Liong, Jiwen Lu, Yap-Peng Tan, and Jie Zhou. 2017. Cross-modal deep variational hash- ing. In IEEE International Conference on Computer Vision, ICCV 2017, Venice. Italy, October 22-29, 2017, pages 4097-4105, IEEE Computer Society.
Tachoon Kim, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Mark Marsden, Alessandra Sala, Seung Hwan Kim, Bohyung Han, Kyoung Mu Lee, Honglak Lee, Kyounghoon Bae, Xiangyu Wu, Gao, Hailiang Zhang, Yang Yang, Weili Guo, Jianfeng Lu, Young- taek Oh, Jae Won Cho, Dong jin Kim, In So Kweon, Junmo Kim, Wooyoung Kang, Won Young Jhoo, Byungseok Roh, Jonghwan Mun. Solgil Oh, Ke- Emir Ak, Gwang-Gook Lee, Yan Xu, Mingwei Shen, Kyomin Hwang, Wonsik Shin, Kamin Lee, Wonhark Park, Dongkwan Lee. Nojun Kwak, Yujin Wang, Yimu Wang, Tiancheng Gu, Xingchang Lv, and Mingmao Sun. 2023. Nice: Cvpr 2023 challenge on shot image captioning.
Yang Liu, Samuel Albanie, Arsha Nagrani, and An- drew Zisserman. 2019. Use what you have: Video retrieval using representations from collaborative ex- perts. In 30th British Machine Vision Conference 2019, BMVC 2019. Cardiff, UK Septemher 9-12, 2019, page 279. BMVA Press.
Huaishao Luo, Lei Ji, Ming Zhong. Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293- 304.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In International Conference on Learning Representations.
Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022. X-CLIP: end- to-end multi-grained contrastive learning for video- text retrieval. In MM 222: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, pages 638-647 ACM.
A. Sophia Koepke, Andreea-Maria Oncescu, Joao Henriques, Zeynep Akata, and Samuel Albanic. 2022. Audio Retrieval with Natural Language Queries: A Benchmark Study. IEEE Transactions on Multimedia, pages 1-1. Conference Name: IEEE Transactions on Multimedia.
Daniele Micciancio and Panagiotis Voulgaris. 2010. Faster exponential time algorithms for the short- est vector problem. In Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '10, page 1468-1488, USA. Society for Industrial and Applied Mathemat- ics.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjin Liu. 2021. Less more: Clipbert for video-and-lane lnd--annggggee learning via

---

Antoine Miech, Jean- Dean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2021. Think- ing fast and slow: Efficient text-to-visual retrieval with transformers. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 9826-9836. Computer Vi- Sion Foundation/ IEEE.
(South), October 27 November 2. 2019, pages 3027-300 IEEE
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Antoine Miech, Ivan Laptev, and Josef Sivic. 2020. Learning a Text- Video Embedding from Incomplete and Heterogeneous Data. ArX1v:1804.02516 [es].
Haoran Wang, Di Xu, Dongliang, He, Fu Li, Zhong Ji, Jungong Han, and Errui Ding. 2022a. Boosting video-text retrieval with explicit high- level semantics. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa Portugal, October 10 14, 2022, pages 4887-4898. ACM.
Andreea- Maria Onceseu, A. Sophia Koepke, João F. Henriques, Zeynep Akata, and Samuel Albanie. 2021. Audio retrieval with natural language queries. In Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno Czechia, 30 August September pages 2411-2415. ISCA.
Xiaohan Wang. Linchao Zhu, Zhedong Zheng, Min- gliang Xu, and Yi Yang. 2022b. Align and tell: Boosting text-video retrieval with local alignment and fine-grained supervision. IEEE Transactions on Multimedia, pages 1-11.
Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Dar- rell, Yejin Choi, and Anna Rohrbach. 2022. Ex- posing the limits of video-text models through con- trast sets. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3574-3586, Seattle, United States, Association for Computational Linguistics.
Yimu Wang. Xiangru Jian, and Bo Xue. 2023. Balance act: Mitigating hubness in cross-modal retrieval with query and gallery banks.
Yimu Wang, Shiyin Lu, and Lijun Zhang. 2020a. Searching privately by imperceptible lying: A novel private hashing method with differential pri- vacy. In Proceedings of the 28th ACM International Conference on Multimedia, page 2700-2709.
Bryan A. Plummer, Liwei Wang. Chris M. Cervantes, Juan C. Caicedo, Julia Hockemmaier, and Svetlana Lazebnik. 2017. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. Int. J. Comput. Vis., 123(1):74- 93.
Yimu Wang and Peng Shi. 2023. Video-Text Retrieval by Supervised Multi-Space Multi-Grained Align- ment. ArXiv:2302.09473 [cs].
Yimu Wang, Xiu-Shen Wei, Bo Xue, and I. Lijun Zhang. 2020b. Piecewise hashing: A deep hashing method for large-scale gne-ggrined search. In Pattern Recognition and Computer Vision Third Chinese Conference, PRCV 2020, Nanjing, China, October 16-18, 2020. Proceedings. Part II. pages 432 444.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings Machine Learning Research, pages PMLR.
Yimu Wang, Bo Xue, Quan Cheng, Yuhui Chen, and Li- jun Zhang. 2021. Deep unified cross-modality hash- ing by pairwise data alignment. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IICAI-21, pages 1129-1133 135.
Milos Radovanovic, Alexandros Nanopoulos, and Mir- jana Ivanovi. 2010. Hubs space: Popular near- est neighbors in high-dimensional data. Journal of Machine Learning Research, 1 11(86))2487-2533
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR- VTT: A large video description dataset for bridg- ing video and language. on Computer Vision and Pattern Recognition, CVPR 2016. Las Vegas, NV. USA, June 27-30, 2016, pages 5288 5296. IEEE Computer Society.
2016 IEEE Conference
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mareus Rohrbach, and Douwe Kiela. 2022. FLAVA: A foundational language and vision aligmment model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans. LA. USA, June 18-24, 2022, pages 15617- IEEE.
Mouxing Yang, Zhenyu Huang, Peng Hu, Taihao Li, Jiancheng Lv, and Xi Peng. 2022. Learning with twin noisy labels for visible-infrared person re- identification. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14288-14297
Shupeng Su. Zhisheng Zhong, and Chao Zhang. 2019. Deep joint-semantics reconstructing hash- ing for large-scale unsupervised cross-modal re- trieval. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea
Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. 2023. Multimodal federated learn- ing via contrastive representation ensemble. In

---

The Eleventh International Conference on Learning Representations.
Sangwon Yu, Jongyoon Song, Heeseung Kim, Seong-
min Lee, Woo-Jong Ryu, and Sungroh Yoon. 2022.
Rare tokens degenerate all tokens: Improving neural text generation via adaptive gradient gat- ing for rare token embeddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29 29-45, Dublin, Ireland. Association for Computational Linguistics.
Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao. Qinli Yang, and Junming Shao. 2020. Revis- iting representation degeneration problem in lan- guage modeling.
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 518-527 Online, Association for Computational Lin- guistics.
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2022. Centerclip: Token clustering for efficient text-video retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '22, page 970- 970-981. New York, NY, USA. Associa- tion for Computing Machinery.
Zhun Zhong. Liang Zheng. Donglin Cao, and Shaozi Li. 2017. Re-ranking person re-identification with k-reciprocal encoding. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3652-3661. JEEE Computer Society.

---

A Related Work
through likelihood maximization with the weight- tying trick, especially with large training datasets, many of the learned word embeddings tend to de- generate and be distributed into a narrow cone. This limitation largely reduces the representation power of word embeddings. The representation degen- eration problem leads to an increase in the over- all similarity between token embeddings, which has negative effect on the performance of the models. was noted that Laplacian regulariza- tion can address this problem better than cosine regularization through theoretical proof and empir- ical experiments (Zhang et al., 2020). Subsequent work highlighted (Yu et al., 2022) that the training dynamics of the token embeddings focus on rare token embedding which leads to the degeneration problem for all tokens. To this end, they use adap- tive gradient gating which gates the specific part of the gradient for rare token embeddings and thus better alleviates the data degeneration problem.
We review prior work in cross-modal retrieval and representation degeneration, which are the two most related areas to our work.
Cross-modal Retrival. The goal of cross-modal retrieval is to learn a common representation space, where the similarity between samples from differ- ent modalities can be directly measured. Recently, inspired by the success of deep learning (Devlin et al., 2019; He al., 2016), numerous meth- ods based on deep neural networks have been proposed for image-text retrieval (Radford et al., 2021), video-text retrieval (Luo et al., 2022), and audio-text retrieval (Oncescu et al., 2021). Fur- ther, to learn better representation space, vision- language pretraining (Gan al., 2020; Li al., 2020; Singh al., 2022) large-scale unlabeled cross-modal data has been widely employed and have shown promising performance. Motivated by this, recent works have attempted to pretrain or fine-tune cross-modal retrieval models, e.g., image- text retrieval (Radford et al.. 2021; Li et al., 2020). video-text retrieval (Chen et al., 2020; Cheng et al., 2021; Gao et al., 2021; Gorti al., 2022; Lei et al., 2021: Ma et al., 2022: Park et al., 2022; Wang al., 2022a.b; Zhao et al., 2022), and audio-text retrieval (Oncescu et al., 2021) in an end-to-end manner.
Though representation degeneration has been explored in NLP, it remains unexplored in multi- modal learning for a long time. A recent work (Liang et al., 2022) shows that the representa- tion generated by common deep neural network is restricted to narrow cone and consequently, with two sentations from the two modalities are clearly apart during the whole training procedure. Further, they also show that varying the modality gap distance has a significant impact on improving the model's s downstream zero-shot classification performance and fairness.
encoders, the repre-
In contrast to the methods that focus on improv- ing the representation learning ability, another line of cross-modal retrieval research has focused on improving the effectiveness of retrieval, including k-d trees (Bellman, 2015), re-ranking (Zhong et al., 2017; Miech et al., 2021), query expansion (Chen et al., 2020), vector compression schemes based on binary codes (Su et al., 2019; Liong et al., 2017) and quantization (Gong et al., 2013) that help address the curse of dimensionality (Keogh and Mueen, 2017). However, recent study (Liang et al., 2022) shows that the representation degener- ation problem has significantly affected the perfor- mance of multi-modal learning. To investigate the influence of representation degeneration problem in the cross-modal retrieval, we show that repre- sentation degeneration problem widely exists in different datasets and models.
To step forward towards better representation learning in cross-modal retrieval, different from the previous methods in NLP which focus ad- dressing this problem in the training procedure, we propose novel method, namely INvGC, which proposes to avoid representation degeneration in a post-procsssing manner. Inspired by the repre- sentation aggregation induced by graph convolu- tion (Keogh and Mueen, 2017; Baranwal et al., 2023), we utilize the graph convolution in an oppo- site way to separate the data points that share sim- ilar representation. As the first method in solving the representation degeneration problem in cross- modal retrieval, INVGC does not require retraining the model or any other time-consuming operation. INvGC achieves better retrieval performance with a larger margin between different representations compared to the baselines and does this faster.
Representation degeneration. The representa- tion degeneration problem was first introduced in the natural language processing (NLP) area (Gao et al., 2019). It was found that when training a model for natural language generation tasks

---

where UPDATE and AGGREGATE are arbi- trary differentiable functions (i.e., neural networks) and x is the embeedding(representttion of node at k-th iteration. m is the "message" that is B Elaborations on Methodologies
For the sake of simplicity and explainability, we coneretize the above operation only with simple addition and self-loop, as follows.
B.1 Graph Convolution
Graph convolution is a mathematical operation that transforms the features of nodes in graph based on their local neighborhoods. The objective is to learn graph, which
x = xi + Σ Asxj-
(10)
function of signals/features takes into account the graph structure and the node features. It can be regarded as a generalization of convolutions to non-Euclidean data (Bruna al., 2014). The operation is first introduced and pop- ularized in the work of Graph Convolution Net- works (Kipf and Welling, 2017), which is consid- ered one of the most seminal papers in the area of graph learning.
Note that the only difference with Equation (10) and inverse convolution Equation (3) we apply in the study is that addiction is replaced with subtrac- tion, leading to the name 'inverse'.
B.2 Average Pooling in CNN
Average pooling is one type of pooling layers that conducts dimensionality reduction, reducing the number of parameters in the input (Goodfellow et al., 2016). Similar the convolutional layer, the pooling operation sweeps filter across the entire input, but the difference is that this filter does not have any weights. Instead, the kernel applies an aggregation function to the values within the recep- tive field. Specifically, the adopted average pooling calculates the average value within the receptive field of the filter as it moves across the input. Here in this study, the receptive field is subject to the size of the neighborhood of each data point.
The main idea behind graph convolution oper- ation is to generate new representation for each node that captures the local neighborhood informa- tion around it. This is usually achieved by aggregat- ing feature information from a node's direct neigh- bors, sometimes including the node itself. For- mally, it can be defined as special case of Mes- sage Passing Network (MPNN), in which vector messages are exchanged between nodes and up- dated using neural networks (Gilmer al., 2017). The basic operation of MPNN can expressed as (Hamilton, 2020)
B.3 Distribution of Data Representation
(k+1) (PDATE (x(*)
While we keep using the discrete gallery and query set like in Equation (3), they be regarded sampled results from the hidden continuous distri- bution dependent on the intrinsic properties of the corresponding dataset and the representation learn- ing method applied. Therefore, idcally, two gallery sets sampled from the same dataset will probably have different Adeg scores even with the same rep- resentation learning method due to the variance introduced by the sampling process. That is to say, performing the same inverse convolution operation on the two sampled gallery sets might have quite different effects, especially when the sample size is small. Also, since the proposed method in this study is a post-processing approach without any training, we need to control the magnitude of the convolution with the help of some hyperparameters. The reason for doing this is to control the change in the representation of the gallery data that has already been aligned with the embedding space of query data by the representation learning model. Given the sampled gallery set is small, this means a very large variance in the value of the best hyper- parameters as well.
AGGRECATE(S (5x.).JENN
= -UPDATE()) (N)
(1) (k)
N(O)
aggregated from i 's graph neighborhood (i).
In this study, we adopt a simple message passing operator that performs non-trainable propagation since we want to propose a post-processing method with any training. The adopt operator is actually the backbone of multiple GNN studics, which can be expressed (GraphConv),
x =
Cj Xj,
JEN(()
where 4 defines a custom aggregation scheme. x! is updated representation node and eji is edge weight between node j.

---

Unfortunately, it is quite common in practice that we only have a small sampled gallery set. Usually, when cross-modal retrieval carried out, we con- stantly cut down the size the gallery set with the help of some pre-ranking or indexing techniques. This process can be somehow regarded as sampling a set from the distribution that is empirically rep- resented by the whole gallery set. Also, during the evaluation of any method various datasets, the size of the test or evaluation gallery set is typically much smaller compared to the training set. Both cases make the result of the proposed methods sub- jected to potentially large variance.
2016), MSVD (Chen and Dolan, 2011), Activi- tyNet (Fabian Caba Heilbron and Niebles, 2015), and DiDemo (Hendricks et al., 2017)), two image- text retrieval benchmarks (MSCOCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2017)), as well as two audio-text retrieval benchmarks (Au- dioCaps (Kim et al., 2019) and CLOTHO (Drossos et al., 2020)). The details of the datasets are pre- sented below:
MSR-VTT (Xu et al., 2016): Comprises ap- proximately 10k videos, each accompanied by 20 captions. For text-video retrieval, we follow the protocol by previous works (Liu al., 2019; Croitoru et al., 2021; Luo et al., 2022; Ma al., 2022; Park et al., 2022), using both the official (full) split and the Ik-A split. The full split includes 2,990 videos for testing and 497 for validation, whereas the 1k-A split has 1,000 videos for testing and around 9,000 for training.
However, it would be more promising that our method is generally stable and robust to any size of the gallery set. The ideal case would be that can perform the inverse convolution similar Equa- tion (1) but based on the continuous distribution, where P(xy) is data point Xj to be sampled from the hidden distribution, as follows,
x;' x; x - Siy P(xj)xj.
(11)
MSVD (Chen and Dolan, 2011): Contains 1,970 videos and about 80k captions. The standard split used in prior works (Liu et al., 2019; Croitoru al., 2021; Luo al., 2022; Park al., 2022) is adopted for reporting re- sults, which includes 1,200 videos for training. 100 for validation, and 670 for testing.
However, it is impossible to have exact access to this hidden distribution in practice. The best approximation is the training (or validation) gallery set & since it is the largest one we can obtain. Therefore, we can perform the inverse convolution on G. Note that the distribution of the query set Q should theoretically be similar to that of G as this basic assumption in machine learning (Bogolin al., 2022). Therefore, is possible combine the (train or validation) gallery set G and the (train or validation) query set Q to be the even better esti- mation of the hidden distribution. Thus, we go on to refine INVGC as in Equation (4),
AetivityNet (Fabian Caba Heilbron and Niebles, 2015): Contains 20k videos and approximately 100K descriptive sentences. These videos are extracted from YouTube. We employ paragraph video retrieval setup as de- fined in prior works (Liu et al., 2019; Croitoru et al., 2021; Luo et al., 2022; Park al., 2022). We report results the vall split. The train- ing split includes 10,009 videos, with 4,917 videos allocated for testing.
In general, with reasonable and general assump- tions, we strike the importance of utilizing the data of both the modality from the training set when we want to capture more accurate and stable distribu- tion of data representation. The idea is not bound to the proposed methods and can be adopted by any future work on the prot-processing of cross-modal retrieval tasks.
DiDemo (Hendricks al., 2017): Includes over 10,000 personal vidcos, each lasting be- tween 25-30 seconds, along with over 40,000 localized text descriptions. The videos are di- vided into training (8,395), validation (1,065), and testing (1,004) sets.
Experiments
MSCOCO (Lin et al., 2014): Consists of 133k images, each accompanied by 5 captions. The 5k split is used for evaluation.
C.1 Datasets Details
The experiments are conducted on eight cross- modal benchmarks, which include four video- text retrieval benchmarks (MSR-VTT (Xu et al.,
Flickr30k (Plummer et al., 2017): This dataset contains 31 1,000 images collected from

---

Table 10: Implmentation details.
sentations gathered in a very narrow cone in the em- bedding space for basically all the datasets, meth- ods, and tasks. Also, though subject to the differ- ence between datasets and methods, we can witness that a more convex-shaped distribution usually gen- crally leads to a larger deg-
Publiic Code
Public Weights
CE+
Link
MSR EER. MSVD, DiDeMo, and ActivityNet
TT-CE=
Link
MSR-VTT, MSVD, DiDeMoo and AcrivityNer
CLIP4Clip
Link
NIA
CLiP2Nideo
Link
MSR-VIT and MSVD
X-CLIP
Link
CLIP
Link
CLIP
The results imply the universality of the repre- sentation degeneration problem. More quantitative results be found in RQ1 in Section 4.3 and in the Continuation on RQ1 section (Appendix C.7).
Osear
Link
MSCOCO and Flickr30k
AR
Link
AndioCaps and CLOTHO
Flickr, each accompanied by reference sen- tences provided by human annotators.
C.4 Detail Setting of AvGPOOL
AudioCaps (Kim et al., 2019): Comprises sound samples with event descriptions. We adopt the same setup prior work (Koepke et al., 2022) where 49,291 samples are used for training, 428 for validation, and 816 for testing.
Following Equation (9), the adjacency matrices pool and S poul of AvGPooL can be presented as, 59
1. s.mi(((n (g), Ri) P.(G.p)
0. else
and also,
CLOTHO (Drossos et al., 2020): Comprises of 4,981 audio samples of 15 to 30 seconds in duration and 24,905 captions of eight to 20 words in length (five captions for each audio sample).
{ 1. sim((ii qi) P.(..p)
(po (i.
e.ese,
where Pr(G,p) and P.( are the value of p- percentage largest similarity between node and all the nodes in set G and Q, respectively. We in- clude four values of p, namely [10, 25, 50, 100), to obtain a solid benchmark. All four benchmarks for the same task share identical hyperparameters Tg and rg.. Note that the worst scenario of tuning for AvGPooL is when there is no hyperparame- ter that can enable AvGPooL to beat the baseline performance of the original retrieval model. But it doesn't 't happen during the experiment since we can al always locate pair of hyperparameters with little effort with which AvGPOOL with at least one of four selected p values on at least one of the evaluation metrics mentioned in Section 4.2. This indicates the effectiveness of the idea of inverse convolution.
C.2 Experiment Details
The public codes and weights of all the tasks in this study are summarized in Table 10. Note that there is no available trained model for CLIP and X-CLIP. Therefore, we train both models on single A A100 GPU with the hyperparameters recommended by the original studies.
C.3 Prevalence of Representation
Degeneration Problem across Datasets and Methods
outperform the baseline
To show that the representation degeneration prob- lem prevails in all datasets, methods, and tasks, we perform the same analysis in Figure la. We uni- formly sample subset of the gallery set of both retrieval tasks (i.e., text to other modality or other modality to text, other modalities can be video, image, or audio depending on the dataset), and per- form PCA upon it to reduce the dimension of the representations down to 2, which is the first two principal dimensions. The results are presented in Figure 5. Note that Aeeg (i.e., the degree of repre- sentation degeneration problem) included in each figure the one for the complete gallery set instead of the sampled set used in the figure.
C.5 Computational Complexity
In the section, we discuss the computation com- plexity of the proposed INVGC and INVGC w/LOCALADI. As noted Section 2.1, N₂ is the size of gallery data, and No and N& are the size of the training (or validation) query and gallery set, respectively. We also assume N₂ is the size of the query.
Both methods need to precompute two adjacency matrices between the gallery set and both sets of training (or validation) data before performing the convolution, which costs ON + N&)) as
For this qualitative but very intuitive study, We firmly validate again that almost all the data repre-

---

Table 11: Retrieval performance on MSCOCO (5k split). Best in Bold and the second best is underlined.
Text-to-Iman Retrieval
Image-to- Text Retrieval
Normaizzaiion
R@
R@5 1
R@100
MdR 4
MnR 4
R@
R@5 7
R@10 10
MdR
MnR
CLIP
30.34
54.74
66.08
4.0
25.39
50.04
74.80
83.38
1.0
9.22
+ AVGPooi,(ratio0)) I)
30.37
54.77
66.
4.0
25.36
49.98
75.08
83.34
2.0
9.20
(5
30.37
54.77
66.14
4.0
25.36
49.98
75.06
83,30
2.0
9.20
+AVGPooI
30.38
54.77
66. 10
4.0
25.38
49.98
75.10
83,34
2.0
9.20
(Iatoo)))
30.39
54.77
66.11
4.0
25.38
49.98
75.06
83.36
2.0
9.20
+INVGC
32.70
57.53
68.24
4.0
24.35
51.04
75.18
83.24
1.0
8,93
+INVGC A/L
33.11
57.49
68.19
28.95
52.26
76.42
84.32
8.83
Oscar
$2.50
80.03
87.96
1.0
10.68
66.74
89.98
94,98
1.0
2.95
+ +AvGPooL(ruti-O.)
52.52
80.04
87.95
1.0
10,70
66.98
89.98
95,00
1,0
2.96
52.52
80.03
87.96
1.0
10.68
66.98
89.96
94,96
1.0
2.95
()
52.51
80.00
87.96
1.0
10,67
66.94
89.92
94.94
1,0
2.95
(raio)))
52.50
80.02
10.68
66.70
89.90
2.96
+INVGC
52.63
80.05
87.96
1.0
10.72
67.90
89.96
95.22
1.0
2.92
+INvGC WLOCALADI
52.93
80.05
87.78
1.0
11.09
67.68
90.24
95.20
1.0
2.94
Table 12: Retrieval performance on Flickr30k. Best in Bold and the second best is underlined.
Text-to- Text-to-Image Retrieval
Image-to- texx Retrieval
Normalization
R@ f
R@5 f
R@ 10
MdR 4
MnR 4
R@ 1
R@51
R 10
MdR
MnR
CLIP
58.98
83.48
90.14
1.0
6.04
78.10 10
94.90
98.10
1.0
1.98
(raiPoo,
59.10
83.56
6.04
78.30
95.00
1.97
59.10
83.56
90.18
1.0
6.04
78.40
95.00
98.20
1.0
1.97
+AVGPOOL (1atio00)))
59.10
83.54
90.18
1.0
6.05
78.40
95.00
98.20
1.0
1.97
(AviPooL(rattiol)
59.10
83.54
90.18
1.0
6.04
78.40
95.00
98.20
1.0
1.98
+INVGC
60.18
85,30
91.20
1.0
5,52
78.50
95.10
98.20
1.0
1.98
+INVGC w/L
60.48
85.30
91.10
1.0
5.59
80,20
95.10
98.40
1.0
1.97
Oscar
71.60
91,50
94.96
1.0
4.24
86.30
96.80
98.60
1.0
1.58
+ AVGPooL(rution 1)
71.62
91,
94.92
1.0
4.25
86.50
96.70
98.50
1.0
1.63
+ +AvGPooi
71.66
91.50
94.94
1.0
4.24
86.00
96.90
98.60
1.0
1.62
+ -.)
71.66
91.50
94,92
1.0
4.24
86.00
97.00
98.60
1.0
1.60
(Iatio))((aatio
71.62
91.52
94.96
1.0
4.24
86. 10
97.00
98.60
1.0
1,58
+INvGC
71.68
91.46
95.06
4.25
86.80
96.80
98.40
1.0
1.61
+INVGC ALLCCALADI
71.74
91.56
4.29
86.60
97.10
98.70
1.58
matrix multiplication. Then, the matrix multipli- cation of inverse convolution operation also in- curs cost of (N + N&))) for INvGC and NN N)) for INvGC w/LOCALADS, for the pruned matrix LOCALADI. Note that if choose small & for LOCALADI like 1% in the empirical study, the convolution step can much faster in practice (though the complexity doesn't change due to the pre-computation step).
C.6 More Quantitative Results
The results on retrieval performance for all the datasets with various methods are presented in Ta- bles 11 to 18. The results indicate that both pro- posed methods achieve significantly better retrieval performance all the tasks compared to the origi- nal baseline and AvGPoOL benchmarks.
Text- Leate and Image-Tex Retrieval. Results are presented in Tables 11 and 12.
After performing the proposed methods, need another (NN,.Ng) time calculate the similarity between the query and gallery set when performing the inference, given that we don't tt consider any advanced trick to perform argmax.
On the MSCOCO (5k split) dataset, the INVGC w/LOCALADI method outperforms other methods in both text-to-image retrieval and image-to-text Retrieval for both CLIP and Oscar models. Specifi- cally, for the CLIP model, IN GC w/LOCALADI achieves the best R@ and R @ in text-to-image retrieval and the best @ @5, R 10, and MnR scores in image-to-text retrieval. Similarly, for the Osear model. INvGC w/LOCALADI also delivers the best results, with the best R@ and R@5
In sum, both INVGC and INvGC w/LOCALADI incur computational cost of (N, + NN)) for inverse graph convolution (before inference) and O ON,N,, for inference.

---

Table 13: Retrieval performance on MSR- VTT (full split and 1k split). Best in Bold and the second best is underlined.
Text-to-Videe Retrieval
Video-to-Tex, Retrieval
Normalization
@@1
R@5
R 10
MdR
MnR
R @
R@5
R 10
MdR
MnR
MSR-
VTT (full split)
CE+
13.51
36.01
48.75
11.0
70.28
21.61
50.57
63.48
5.0
22.62
13.52
36.02
48.78
11.0
70.36
21.57
50.54
63.44
5.0
22.63
-AGGPooL
13.50
36.00
48.76
11.0
70.33
21.57
50.57
63,48
22.62
+ (
13.51
35.99
48.76
11.0
70.31
21.57
50.57
63.48
5.0
22.62
(AvoPooL(ratio=I)
13.51
36.01
48.75
11.0
70.28
21.61
50.57
63.48
3.0
22.62
+INVGC
13.57
36.01
48.74
11.0
71.34
22.47
50.67
63.68
5.0
22.42
+INVGC A/D
13.83
36.54
49.18
11.0
71.25
22.24
51.37
64.75
5.0
23.
TT-CE+
14.51
37.58
50.39
10.0
64.29
24.11
53.71
67.39
5.0
20.20
AAGGPoo..ratiooll
14.51
37.59
50.39
10.0
64.29
24.21
53.48
67.39
5.0
20.22
14.51
37.58
50.39
10.0
64.29
24.18
53.58
67.36
20.23
+AvGPooL -AvGPooi_ratto=00..
14.51
37.58
50.39
10.0
64.29
24.18
53.55
67.53
5.0
20.22
()viPooL(ratio=))
14,51
37.58
50.39
10.0
64.29
24.11
53.71
67.39
5.0
20.20
+INVGC
14.62
37.61
50.50
10.0
64.89
25.15
53.78
67.63
21.43
+INVGC w/LOCALADI
15.08
38.48
51.57
10.0
64.19
26.05
54.72
68.73
4,5
19.83
MSR-VIT (1k split)
CLIP4Clip
44.10
71.70
81.40
2.0
15.51
42,09
71.25
81.23
12.02
+ AvaPPol,(ratioo)))
44.20
71.60
81.50
2.0
15.55
42.39
70.65
80.24
2.0
12.34
AvGPooL(ratio00.25
44.20
71.60
81.50
2.0
15.55
42.39
70.55
79.94
2.0
12.37
AvGGooo.(ratio-f
44.20
71.50
81.50
2.0
15.54
42.39
70.75
79.84
2.0
12.33
(ratoo)))
44.10
71.60
81.50
2.0
15.52
42.29
70.75
80.04
2.0
12.30
+INvGC
44.40
71.90
81.60
15.36
44.66
72.13
81.72
11.59
+INVGC w/LOCALADI
44.40
71.70
81.20
2.0
15.65
42.79
71.44
80.83
2.0
12.08
CLIP2Video
46.00
71.60
81.60
72.73
82.51
10.20
+ (vaPooorraati:000))
45.90
71.70
81,50
2.0
14.52
43.87
72.83
82.41
2.0
10.21
(5
46.10
71.80
81.50
2.0
14.53
43.77
72.73
82.31
2.0
10.21
+ (3
46.00
71.70
81.50
2.0
14.53
43.77
72.73
82.31
2.0
10.21
(ratoo))(
45.90
71.70
81.50
2.0
14.52
43.77
72.73
82.41
2.0
10.21
+INVGC
46.20
71.70
81.30
14.44
44.66
73.22
83.10
10.07
+INVGC w/LOCALADI
46.60
72.10
81.70
2.0
14.50
45.06
70.65
79.64
2.0
11.65
X-CLIP
46.30
74.00
83.40
12.80
44.76
73.62
82.31
11.02
AvGPooLLratio00..)
46.50
74.00
83.40
2.0
12.88
44.92
73.64
82.23
2.0
11.07
-AvGPoot((atiooD))
46.40
74.00
83.50
12.85
44.96
73.52
82.41
10.87
+AvGPooI (1atio00)))
46.30
74.00
83.30
2.0
12.83
44.76
73.42
82.11
2.0
10.78
(AvcPooL(ratio:))
46,20
74.00
83.40
2.0
12.83
44.66
73.52
82.02
2.0
10.76
+INvGC
47.30
74.00
83.30
13.42
46.05
75.40
83.20
10.86
+INVGC w/LOCALADI
47.10
74.20
83,50
2.0
13.09
46.15
71.94
80.83
2.0
11.95
text-to-image retrieval.
split) show that INVGC w/LOCALADJ generally achieves the best performance in text-to-video re- trieval while INVGC presents superior results in video-to-text retrieval.
In the Flickr30k dataset, the INVGC w/LOCALADJ method generally shows the highest performance in both text-to-image retrieval and mmage-to-text retrieval across both CLIP and Osear models.
For the ActivityNet dataset, INvGC WLLOcALADiettbbis superior performance in text-to-video and video-to-text retrieval on R@ R @@ 5, and R R@ 10 with all four methods.
Text-Video Video and Video- Text Retrieval. Results are presented in Tables 13 to 16.
From the table for MSR-VTT (full split), can observe that INVGC w/LOCALADI demonstrated the best performance in both text-to-video and video-to-text retrieval on R@ I, R@S, and R@ 10 with both methods. The results for MSR-VTT (1k
For the MSVD dataset, the best-performing method for text-to-video retrieval is INYGC w/LOCALADI since has the best performance on R@I, R@5, and R 10 with all methods except C-P.I.. Meanwhile, INVGC method has the best

---

Table 14: Retricval performance on ActivityNet. Best Bold and the second best is underlined.
Text-to-Video
Video Retrieval
Vidco-to textt Retrieval
Normalization
R@
R
R 10
MdR
MnR
R 08
R
&
MdR
MnR
CE+
19.16
49.79
65.79
6.0
21.99
18.51
47.85
63.94
23.06
19.22
49.77
65.79
6.0
21.98
18.63
47.87
63.92
6.0
23.03
+AVGPoOLLrrato=00225
19.20
49.79
65.77
6.0
21.98
18.59
47.94
63.92
6.0
23.01
19.20
49.79
65.77
6.0
21.98
18.59
47.90
63.94
6.0
23.01
CWwGooo,(ratios))
19.16
49.79
65.79
6.0
21.99
18.51
47.85
63.94
6.0
23.06
+INYGC
19.34
49.58
66.02
6.0
21.33
18.71
48.12
64.10
22.75
+INVGC A/LOCALADI
19.89
50.21
65.47
5.0
23.05
19.22
48. 14
63.68
6.0
25.96
TT-CE+
23.29
56.42
73.78
4.0
13.59
22.49
56.38
72.67
4.0
13.90
+AVGPoOL(ratio=0..
23.29
56.46
73.81
4.0
13.59
22.45
56.21
72.56
4.0
13.89
23.27
56.44
73.81
4.0
13.59
22.49
56.25
72.63
13.89
23.25
56.42
73.81
4.0
13.59
22.49
56.27
72.63
4.0
13.89
#AVGPoolL(raio=tt)
23.29
56.42
73.78
4.0
13.59
22.49
56.38
72.67
13.90
+INvGC
23.47
56.66
73.93
4.0
13.51
22.57
56.34
72.54
4.0
13.87
+INGGC w/LOCALADI
23.90
58.12
74.31
4.0
13.53
24.51
57.41
73.34
4.0
14.59
CLIP4Clip
41.85
74.44
84.84
2.0
6.84
41.62
74.11
86.12
2.0
6.81
(rrGPoor
41.83
74.47
84.84
2.0
6.84
41.59
74.06
86.10
6.81
-AVGPP0LLratii=00255
41,80
74.44
84.84
2.0
6.84
41.54
74.09
86.10
2.0
6.81
--VGPool.(ratioo...
41.85
74.44
84.84
2.0
6.84
41.54
74.11
86.10
6.81
+AvoPooL(rrati=-1)
41.88
74.44
84.84
2.0
6.84
41.57
74.11
86.12
2.0
6.81
+INvGC
41.90
74.40
84.86
2.0
6.84
41.95
74.02
86. 12.
2.0
6.77
+INVGC w/LoCALADI
43.23
75.58
85.74
2.0
6.82
43.20
75.77
86.67
2.0
6.63
X-CLIP
46.25
76.02
86.05
2.0
6.37
45,20
76.07
86.57
6.40
(raiPooo
46.47
75.94
86.05
2.0
6.38
45.20
76.20
86.53
2.0
6.39
-AvGPo0LLfraii=00255
46.38
75.98
86.01
2.0
6.38
45.29
76.24
86.53
2.0
6.38
46.47
75.94
86.01
2.0
6.38
45.25
76.24
86.53
2.0
6.39
(A)tPoo))(ratio=t)
46.43
75.89
86.01
2.0
6.38
45.15
76.16
86.53
2.0
6.40
+INYGC
46.43
75.68
86.22
2.0
6.35
45.95
76.24
87.53
+INYGC w/LOCALADI
47.82
76.46
86.36
2.0
6.91
48.69
77.90
87.93
2.0
6,00
results for video-to tedeett--text retrieval with CLIP4CLIP, CLIP2Video, and X-CLIP.
C.7
More Ablation Study
Continuation on RQ1: Is the data degeneration problem alleviated?
On the DiDeMo dataset, INvGC w/LoCALADI with both methods achieve the best performance the text-to-vidco and video-to-text retricval on R@1 and R@5.
To strengthen the conclusion that both INvGC and INVGC w/LocALADthave strong capability to alleviate representation degeneration problem, conduct a comprehensive experiment all the datasets and methods as an extension Table and Table We have Tables 19 to 26 that presents the mean similarity within the test gallery set both tasks for three scenarios, the overall mean (MeanSim), the mean between the nearest neigh- bor(MeanSim@@ 1), and the mean between nearest 10 neighbors (MeanSim @ 10). And we have Ta- bles 27 to 34 include the similarity score from the test gallery
Text-Audio and Audio-Text Retrieval. Re- sults presented in Tables 17 and 18. On the CLOTHO dataset, the AR-CE method enhanced with the INvGC and INvGC w/LOCALADI tech- niques outperforms the earlier techniques, MoEE and MMT, and all other benchmarks. Specifically, either INVGC or INVGC A/LOCALADI achieves the best results in R @ 1, R R55, and R@ 10 in both text-to-audio and audio-to-text retrieval tasks.
the test query with the same evaluation metrics.
In the case of the AudioCaps dataset, INVGC w/LoCAL.ADI method performs best across multi- ple metrics in both text-to-audia audio and audio-to-text retrieval tasks.
It is quite obvious that both INYGC and INvGC w/LOCALADI help decrease the similarity score on all metrics, especially on McanSim@ l(ie.. A Adeg (G)). Also, both methods have better per-

---

Table 15: Retrieval performance on MSVD. Best in Bold and the second best is underlined.
Text-to-
Text-to-Video
Video Retrieval
Video-to- Text Retrieval
Normalization
R@1
R@5
R@
MdR
R@
R@5
R@
MnR
CE+
23.94
54.98
69.00
4.0
18.46
22.84
49.85
61.49
6.0
33.96
AVGPooL. I)
24.11
55.26
69.25
4.0
18.62
23.88
50.15
60.15
35.54
AVGPoOL(ratio=0225
24.01
55.10
69.09
4.0
18.60
23.13
50.15
60.60
5.5
34.86
-AVGPooL.(atio-D)
23.95
55.02
69.05
18.52
23.58
49.85
60.45
34.51
(AatPooL(ratio=I)
23.94
54.98
69.00
4.0
18.46
22.84
49.85
61.49
6.0
33.96
+INvGC
24.47
55.43
69.54
4.0
25.31
28.51
51.34
61.49
5.0
70.09
+INVGC w/LOCALADI
25.49
56.87
71.08
18.34
27.91
57.46
67.76
25.97
TI-CE+
24.42
56.20
70.44
4.0
17.16
25,22
55.07
64.63
4.0
29.94
24.60
56.47
70.72
17.22
25.07
54.93
64.63
29.99
AVGPP0LL(atio=00255
24.56
56.31
70.51
4.0
17.21
25.07
55.37
64.78
4.0
29.98
-AvGPooL,ratioee
24.49
56.18
70.50
4.0
17.19
25.07
55.22
64.48
4.0
29.95
())
24.42
56.20
70.44
17.16
25.22
55.07
64.63
29.94
+INVGC
25.36
56.51
70.97
4.0
17.22
28.81
55.52
65.22
4.0
29.92
+INVGC w/LOCALADI
26.36
58.34
72.29
4.0
16.99
28.06
57.16
68.36
4.0
24.49
CLIP4Clip
44.64
74.66
83,99
2.0
10.32
63.13
79.40
85.37
1.0
11.02
-AvGPooL(ratio-O..
44.87
73.89
83.07
63.28
79.55
85.22
+ -VrGPooL
45.06
74.04
83.49
2.0
11.29
62.84
79.55
85.22
1.0
10,99
-AvGPoo.
45.12
74.32
83.66
2.0
10.76
63.13
79.55
85.22
1.0
11.02
(AvoPooL(ratio=I)
45.09
74.54
83.91
79.55
85.37
+INVGC
45.43
74.82
84.00
2.0
10.42
68.36
84.93
90.30
1.0
8.75
+NvGCC w/LOCALADI
45.73
75.53
84.37
2.0
10.42
64.33
80.15
85.67
1.0
10.77
CLIP2Video
47.05
76.97
85.59
2.0
9.53
62.09
83.13
89.40
1.0
7.73
47.04
76.98
85.61
9.54
62.39
83.73
89.40
AvGPooL(ratio=0.55
47.07
76.98
85.61
2.0
9.54
62,09
83.28
89.40
1.0
7.74
+AGPooI 5
47.06
76.98
85.62
2.0
9.53
61,94
83.73
89.25
1.0
7.77
(AatPoo((ratio=1)
47.06
76.97
85.61
83.58
89.40
7.73
+INvGC
47.09
77.00
85.64
2.0
9.48
70.60
89.10
91.64
1.0
5.39
+INVGC w/LOCALADI
47.47
77.46
85.84
2.0
9.53
66.42
85.37
90.00
1.0
7.37
X-CLIP
46.31
76.84
85,31
2.0
9.59
65.52
83.73
89.85
1.0
8.15
-AaaPoot
46.36
76.70
85.28
65.52
83.88
89.55
8.16
46.33
76.71
85.28
2.0
9.64
65.37
84.03
89.55
1.0
8.17
())
46.35
76.71
85.25
2.0
9.62
65.22
83.73
89,55
1.0
8.19
(AvGPooL(ratio=I)
76.77
85.27
65.37
83.88
89.55
+INVGC
46.82
76.69
85,38
2.0
9.63
70.45
85.22
89.55
1.0
7.76
+INvGC w/LOCALADI
46.49
76.82
85.29
2.0
9.63
66.27
84.33
89.25
1.0
7.74
Table 16: Retrieval performance on DiDeMo. Best in Bold and the second best is underlined.
Text-to- Video Retrieval
Video-to-
Teeeo-ttoTexttt Text Retrieval
Method
Normalization
R@11
R@101
MnR J
10
CE+
18.23
42.63
56.08
8.0
42.05
18.82
42.73
55.78
8.0
37.27
+ AvaPPoor(ratio0O..
18.23
42.73
56.27
8.0
42.19
18.82
42.03
55.88
8.0
37.73
-AvGPooL
18.23
42.73
56.27
8.0
42.11
18.92
42.43
55.88
8.0
37.36
18.13
42.83
56.18
8.0
42.07
18.92
42.33
55.98
8.0
37.52
18.23
42.63
56.08
8.0
42.05
18.82
42.73
55.78
8.0
37.27
+INVGC
18.82
43.13
56.67
41.69
19.02
42.43
56.18
37.08
+INVGC w/LOCALADI
19.02
43.03
56.67
42.04
19.42
43.53
56.70
39.28
TE-CE+
22.31
49.20
62.15
6.0
31.18
21.41
47.11
60.66
6.0
99.67
+AvGPooL(ratio=O.)
22.31
49.50
62.85
6.0
31.18
21.41
47.21
60.46
6.0
29.74
+ AtaPooo
22.41
49,30
62.15
6.0
31.18
21.41
47.21
60.56
6.0
29.71
22.41
49.30
62.15
6.0
31.17
21.41
47.21
60.66
6.0
29.69
+AvGPooL(rutio=))
22.31
49.20
62.15
6.0
31.18
21.41
47.11
60.66
6.0
29.67
+INvGC
23.80
50.10
63.75
5.25
31.51
22.21
47.81
61.35
6.0
29.53
+INVGC w/LOCALADI
24.10
50.30
62.45
31.76
23.90
49.60
62.95
33.76

---

Table 17: Audio-text retrieval performance on CLOTHO. Best Bold and the second best underlined. **' refers to the results direct copied from (Koepke al., 2022). MoEE (Miech al., 2020), MMT (Gabeur al., 2020) are two methods used for audio-text retrieval.
Text-to-Audio Retrieval
Audio-to-Text Retrieval
Method
Normalization
R@1
R@5
R@101
MdR
MnR
R@1
R@5
R@101
MdR
MnR
MoEE
6.00
20.80
32.30
23.0
60.20
7.20
22.10
33.20
22.7
71.80
MMT
6,50
21.60
66.90
23.0
67.70
6,30
22.80
33.30
22.3
67,30
AR-CE
6.27
22.32
33.30
23.0
58.96
7.27
23.35
35.12
21.0
74.68
+AVGPooL//atioo0..
6.37
22.33
33.59
23.0
65.57
7.66
22.01
33.59
22.0
85 61
(5VGP00LL(atio-0.255
6.47
22.12
33.58
22.0
59.84
7.85
22.78
34.64
21.0
76.21
22.24
33.30
22.0
59.14
7.56
34.74
74.91
22.32
33.30
23.0
58.95
7.27
23.35
35.12
21.0
74.68
+INvGC
22.14
34.72
21.0
61.98
9.19
24.02
35.41
22.0
75.06
+INVGC w/LOCALADJ
6,58
22.35
34.64
22.0
58.51
9.00
24.50
36.17
21.0
71.69
Table 18: Text-audio retrieval performance on AudioCaps. Best in Bold and the second best is underlined. refers
to the results direct copied from (Koepke et al., 2022). MoEE (Miech al., 2020), MMT (Gabeur et al., 2020) are methods used for audio-text retrieval.
Text-to-Audio Retrieval
Audio-to-T Texxext Retricval
Method
Normalization
R@1
R@5
R@10
MdR
MnR
R@
R@5 f
R@10 10
MdR
MnR J
MoEE*
23.00
55.70
71.00
4.0
16.30
26.60
59.30
73.50
4.0
15.60
MMT
36.10
72.00
84.50
2.3
7.50
39.60
76.80
86.70
2.0
6.50
AR-CE
22.33
54.49
70.54
5.0
15.89
24.02
56.00
71.81
4.0
16.91
(rriPoot.
22.33
54.49
70.54
5.0
15.89
24.14
55.88
71.69
4.0
16.93
22.33
54.49
70.54
5.0
15.89
24.14
56.13
71.69
4.0
16.92
22.33
54.49
70.54
5.0
15.89
24.14
56.13
71.69
4.0
16.92
+AvcPoot.(ratio=)))
22.33
54,49
70.54
5.0
15.89
24.02
56.00
71.81
4,0
16.91
+INvGC
22.33
54.46
70.56
5.0
15.89
24.88
58.21
71.08
4.0
17.18
+INvGC w/LocALAD
24.07
55.69
70.20
4.0
16.54
27.57
59.31
74.75
4.0
14.21
Table 19: Similarity measures within gallery set (MeanSim @ equivalent Adng (G) on CLOTHO.
Text-to-Audio Retrieval
Audio-to-Text Retrieval MeanSim@1
Method
Normalization
MeanSim
MeanSim@
MeanSim 10
MeanSim
MeanSim 10
AR-CE
0.0436
0.7518
0.6652
0.0143
0.2462
0.2144
+INvGC
0.0391
0.5246
0.4567
0.0115
0.1261
0.1017
+INVGC w/LOCALADJ
0.0279
0.4739
0.4094
0.0114
0.1713
0.1420
Table 20: Similarity measures within gallery set (McanSim @ equivalent to Adeg (G)) on AudioCaps.
Text-to-Audio Retrieval
Audio-to- Retrieval
Method
Normalization
MeanSim
MeanSim@
MeanSim@
MeanSim
MeanSim@
MeanSim@
AR-CE
0.0358
0.6781
0.5622
0.0075
0.2656
0.2330
+INVGC
0.0358
0.6778
0,5620
0.006
0.1837
0.1582
+INVGC w/LOCALÁDI
0.0161
0.3264
0.2602
0.0054
0.1399
0.1142
Table 21: Similarity measures within gallery set (MeanSim @1 equivalent (Ge on MSCOCO (5k split).
Text-to- Imaage Retrieval MeanSim@
Image-to-Tedd Text Retrieval
Method
Normalization
McanSim
McanSim@
MeanSim
MeanSim@1
MeanSim 10
CLIP
0.4717
0.8211
0.7803
0.5161
0.9029
0.8596
+NNGC
0.4693
0.8142
0.7738
0.5137
0.8972
0.8541
INVGC w/LOCALADI
0.4646
0.8059
0.7647
0.5105
0.8924
0.8477
Oscar
0.0509
0.6014
0.4867
0.0166
0.7894
0.6601
+NVGC
0.0536
0.5960
0.4810
0.0172
0.7836
0.6523
INvGGC w/LOCALADJ
0,0703
0,5610
0.4459
0.0374
0.7658
0.6285

---

Table 22: Similarity measures within gallery set (MeanSim equivalent to Adey (G)) on Flickr30k.
Text-to- Tmxaeto-Image Retrieval
Image-to Teggetto-Text Retrieval
Method
Normalization
MeanSim
MeanSim@
MeanSim@
MeanSim
MeanSim@1
MeanSim
CLIP
0.4908
0.7639
0.7115
0.5119
0.8570
0.7882
+NNVCC
0.4868
0.7575
0.7056
0.5086
0.8515
0.7829
INVGC w/LOCALADJ
0.4671
0.7481
0.6942
0.4682
0.8358
0,7604
Oscar
0.0143
0.4155
0.3007
0.0102
0.7059
0.5050
INVGC
0.0147
0.4015
0.2854
0.0116
0.6819
0.4778
INVGC w/LOCALADI
0.0165
0.4077
0.2948
0.0139
0.6915
0.4880
Table 23: Similarity measures within gallery set (MeanSim @ equivalent to Aaeg (G)) on MSR-VTT (full split and
split).
Text-to- Daxttto-Video Retrieval
Viden-to- Text Retrieval
Method
Normalization
McanSim
McanSim@ I
McanSim 10
McanSim
McanSim@)
MeanSim 10
MSR-VTT (full split)
CR+
0.1525
0.6750
0.5516
0.0006
0.0250
0.0223
#INGGC +INGGC w/LOCALADI
0.1435
0.5974
0.4861
0.0005
0.0224
0.0199
0.0845
0.4722
0,3697
0.0007
0.0196
0.0171
TT-CE+
0,1751
0.6878
0.5696
0.0008
0.0295
0.0267
+IVGCC
0.1627
0,6080
0.5017
0.0007
0.0241
0,0216
+INGGC w/LoCALADI
0.1196
0.5176
0.4161
0.0007
0.0237
0.0212
MSR-VTI (Ik split)
CLIP4Clip
0.5416
0.7628
0.6073
0.8389
0.7922
INVGC
0,5460
0.8195
0.7618
0.6149
0.8403
0.7919
+INVGC w/LOOCLADI
0.5280
0,8098
0.7502
0.6090
0.8385
0.7921
CLiP2Video
0.5376
0.8179
0.7589
0.5769
0.8210
0.7702
+NNGGC
0.5379
0.8179
0.7588
0.5798
0.8216
0.7703
#NVGC w/LOCALADI
0.5314
0.8129
0.7528
0.5627
0.8077
0.7559
X-CLIP
0.3555
0.7484
0.6675
0.2626
0.5262
0.4608
+NVGC
0.3671
0.7367
0.6520
0.2653
0.5251
0,4600
INVGC w/LOCALADI
0.3054
0.6974
0.6065
0,2860
0.5344
0,4710
Table 24: Similarity measures within gallery set (MeanSim @ equivalent Adeg (G)) on ActivityNet.
Text-to-Vided Retrieval MeanSim@1
Video-to-Text Retrieval
Method
Normalization
MeanSim
MeanSim@ 10
MeanSim
MeanSim@
MeanSim@ 10
CE+
0.1245
0.6950
0.6095
0.0012
0.0164
0.0141
+INVGC
0.1177
0.6223
0.5453
0.001
0.0161
0.0139
+NNGGC WLOCCALDI
0.0601
0.4059
0.3489
0.0010
0.0129
0.01 10
TT-C+
0.1347
0.7502
0.6673
0.0021
0.0209
0.0187
+NNGCC
0312
0.7213
0.6412
0.0021
0.0206
0.0185
+NNGGC w/LOCALADI
0.0570
0.4082
0.3563
0.0018
0.0145
0.0128
CLIP4Clip
0.3411
0.8122
0.7566
0,2782
0.7638
0.6932
+INGGC
0,3412
0.8122
0.7566
0.2795
0.7635
0.6929
+NNGGC w/LOCALADI
0.3554
0.8029
0.7469
0.2800
0.7483
0.6754
X-CLIP
0.3229
0.8097
0.7460
0.1842
0.6635
0.5816
+NVVGC
0.3369
0.8052
0.7410
0.1667
0.6553
0.,7443
+INVGC w/LOCALADI
0.3033
0.7638
0.6926
0.1968
0.6464
0.5646
formance the retrieval task from text to other modalities, which is the more important task in practice compared the other direction.
significantly alleviate the representation degener- ation problem by significantly reducing Adeg (G) score.
Based on the results, we can safely draw the conclusion that both proposed methods are able to
Continuation
on RQ2: Is INVGC
W/LCCALAD DJ sensitive to the hyperparameter

---

Table 23: Similarity measures within gallery set (McanSim @1
equivalent to (G) on MSVD.
Text-to-Videe Video Retrieval
Video-to- Text Retrieval
Method
Normalization
McunSim
MeanSim@ I
MeanSim 10
MeanSim
McanSim @1
MeanSim@
CE+
0.1057
0.7050
0,5843
0.0155
0.0659
0.0641
+NVGC
0.0464
0.4200
0.3249
0.0075
0.0548
0.0534
+NVGC w/LOCALADI
0.0491
0.3884
0.3046
0.0064
0.0327
0.0310
TT-CF+
0.1176
0.7156
0.5968
0.0193
0.0758
0.0739
+NVGC
0.0493
0.4194
0.3228
0.0152
0.0565
0.0547
+INGGC w/LOCALADI
0.0478
0.3661
0.2864
0.0093
0.0415
0.0398
CLIP4Clip
0.6790
0.8916
0.8518
0.6485
0.9736
0,9527
+NVGCC
0.6918
0.8896
0.8479
0.6505
0.9737
0,9528
+INGGC w/LOCALADI
0.6731
0,8838
0. 0.8429
0.6481
0.9734
0.9522
CLIP2Video
0.5376
0.8179
0.7589
0.5769
0.8210
0.7702
+IVGCC
0.5379
0.8179
0.7588
0.5798
0.8216
0.7703
+INVGC w/LOOCLADI
0.5314
0.8129
0.7528
0.5627
0.8077
0.7559
X-CLIP
0.6931
0.8961
0.8587
0.5796
0.9165
0.8945
+INGGC
0.7035
0,8904
0.8491
0.5879
0.9159
0.8942
+IVGGC w/LOCALADI
0.6819
0.8885
0.8492
0.5600
0.9163
0.8913
Table 26: Similarity measures within gallery set (MeanSim equivalent dG) on DiDeMo.
Tex--to Video Retrieval
Video-to-Texi Retrieval MeanSim@1
Method
Normalization
McanSim
MeanSim@1
McanSim @
McanSim
MeanSim 10
CC+
0.1375
0.6183
0.4835
0.0021
0.0176
0.0144
#INGGC
0.1277
0.5686
0.4409
0.0019
0.0167
0.0136
+IVGGC w/LOCALADI
0.1374
0.6001
0.4690
0.0019
0.0150
TT-CE+
0.1609
0.6794
0.5580
0,0018
0.0304
0.0248
#NVGC
0.1432
0.5299
0.4292
0.0015
0.0244
0.0197
+IVGGC w/LOCALADI
0.0598
0.3680
0.2806
0.0018
0.0199
0.0161
Table 27: Similarity measures between test gallery and query set on CLOTHO. The nearest neighbor is considered with respect the gallery set.
Text-to-Audio Retrieval
Audio-to-Tex Retrieval MeanSim
Method
Normalization
MeanSim
MeanSim
MeanSim 10
MeanSim
MeanSim @ 10
AR-CE
0.0210
0.6028
0,5683
0.0210
0.5788
0,5212
+NNGGC
0.0195
0.3931 0.4520
0.3657
0.0185
0.3235
0.2830
+INGGC ADL
0.0146
0.4231
0.0104
0.4044
0.3600
Table 28: Similarity measures between test gallery and query AudioCaps. The nearest neighbor is considered with respect to the gallery set.
Text-to-Audio Retrieval MeanSim@
Audio-to- Retrieval
Method Nonmalization
MeanSim
McanSim@
MeanSim
MeanSim@1
MeanSim
AR-CE
0.0036
0.6229
0.5729
0.0036
0.5927
0.4883
+INVGC
0.0036
0.6228
0.5728
0.0037
0.4454
0.3595
+INGGC w/LOCALÁDI
0.0005
0.4058
0.3672
-0.0068
0.3568
0.2882
Tg and r
obtained from the MSCOCO dataset are depicted in Figure 6. Despite some observed variations, our method consistently outperforms the baseline, rep- resented by the red dashed line. This suggests that the proposed method continuously enhance per- formance even when parameters are varied over a broad range.
To address the question, like analysis INvGC, assess the R@) metrics of our proposed method under wide range of hyperparameters compared the optimal choice adopted. For each task, one rg or rg is fixed and the other is tuned to ob- serve its impact on the R metrics. The results

---

Table 29; Similarity measures between test gallery and query set on MSCOCO (5k split), The nearest neighbor is considered with respect the gallery set.
Text-to-Image Retrieval
Image-to-Text Retrieval
Method
Normalization
MeanSim
MeanSim
MeanSim 10
MeanSim
MeanSim@)
MeanSim @ 10
CLIP
0.1516
0.3282
0.3138
0.1516
0.3213
0,3009
+INVGC
0.1500
0,3245
0.3101
0.1511
0.3198
0.2994
+INVGC w/LOCALADI
0.0635
0.2214
0.2035
0.0921
0.2414
0.2208
Oscar
-0.0188
0.5423
0.4790
-0.0188
0.4893
0.3940
INVGC
-0.0192
0.5379
0.4744
-0.0193
0.4827
0.3873
INvGC w/LOCALADJ
-0.0257
0.4815
0.4172
-0.0368
0.4290
0.3361
Table Similarity measures between test gallery and query set on Flickr30k. The nearest neighbor is considered with respect to the gallery set.
Text-to-Iman Image Retrieval
Image-to- textext Retrieval
Method
Nonmalization
MeanSim
MeanSim@
McanSim@ 10
MeanSim
MeanSim@)
MeanSim 10
CLIP
0.1688
0.3167
0.2980
0. 0.1688
0.3030
0.2777
+INVGC
0.1681
0,3156
0.2969
0.1683
0.3020
0.2769
+INVGC w/LOCALADJ
0.0854
0.2539
0.2295
0.1150
0.2440
0.2190
Oscar
-0.073
0.4349
0.3552
-0.0073
0.3738
0,2695
+NvGC
-0.0074
0.4235
0.3440
-0.0078
0.3441
0.2403
INVGC w/LOCALADI
-0.0084
0,4259
0.3471
-0.0102
0.3562
0,2554
Table 31: Similarity measures between gallery and query MSR-VTT (full split and Ik split). The nearest neighbor is considered with respect to the gallery set.
Text-to-Vidhe Retrieval
Video-to- Text Retrieval
Method
Normalization
MeanSim
MeanSim@
MeanSim@
MeanSim
MeanSim@
MeanSim@
MSR-VTT (full split)
CE+
-0.0026
0.4262
0.4021
-0.0026
0.3830
0.3367
+NVGC
-0.0024
0.3906
0.3680
-0.0025
0.3474
0.3047
+INGGC w/LOCALADI
-0,0022
0.3610
0.3395
-0.0142
0.2680
0.2316
TT-CE+
0.0135
0.4632
0.4404
0.0135
0.4245
0.3774
+LvGCC
0.0130
0.4234
0.4023
0.0123
0,3533
0,3128
+INGGC w/LOCALADI
0.0103
0.3779
0.3578
0.0028
0.3273
0.2880
MSR-VTT (1k split)
CLIP4Clip
0.1074
0.2399
0.2160
0,1074
0.2388
0.2131
#NVGC
0.1078
0.2397
0.2159
0.1079
0.2394
0.2137
+NVGC
0.1061
0.2394
0.2149
0.0906
0.2176
0.1923
CLIP2Video
0.1214
0.2689
0.2344
0.1214
0.2686
0.2325
+NVGCC
0.1214
0.2688
0.2343
0.1217
0.2687
0.2327
+INGGC w/L.OCALADI
0.1
0.2629
0.2278
0.0502
0.1825
0.1470
X-CLIP
0.1241
0.2484
0.2209
0.1241
0.2459
0.2158
#NNGCC
0.1248
0.2479
0.2199
0.1241
0.2442
0.2142
+INVGC w/LOCALADI
0.1
0.2441
0.2154
0.0895
0.1955
0.1669
D Proofs
BrGb : {x E RR |||||| S a}
D.1 Proof of Theorem 1
To prove the theorem, we start with some prelimi- nary definitions. We denote the n-dimensional unit sphere as,
Then, n-dimensional ball representing the neigh- borhood of x with radius a can be denoted as Bn,x, : {y E R" lly 2112 & a). We go on to denote spherical caps C.,x.a as
Sa-1 = (XE ||x||2 1}
and n-diiensional ball,
C := Sn-1 Buxb

---

Table 32: Similarity measures between test gallery and query set on ActivityNet. The nearest neighbor is considered with respect the gallery set.
Text-to- Video Retrieval
Video-to to-ttt--TTee Text Retrieval
Method
Normalization
MeanSim
MeanSim@1
MeanSim @ 10
MeanSim
MeanSim@)
MeanSim 10
CE+
0.0210
0.4209
0.3703
0.0210
0.4154
0.3679
+INYGC
0.0204
0.4066
0.3578
0.0208
0.4090
0,3623
+NVGGC W/OCCALAD
0.0149
0.3304
0.2893
-0.0017
0,3049
0.2673
TT-CE+
0.0391
0.5070
0.4611
0.0391
0,5019
0.4591
+INGGC
0.0383
0.4885
0.4440
0.0389
0.4973
0.4550
+NVGG w/OCAAADI
0.0258
0.3796
0.3429
0.0148
0,3330
0.3017
CLIP4Clip
0.0538
0.3567
0.3223
0.0533
0.3567
0.3220
#NVGC
0.0533
0.3567
0.3222
0.0533
0.3553
0.3206
+IVGGC w/OCAALDI
0.0106
0.2821
0.2488
0.0107
0,2854
0.2512
X-CLIP
0,1233
0.3484
0,3135
0.1233
0.3464
0,3122
+NVGC INVGC W/L.CALADI
0.1233
0.3460
0.3111
0.1219
0.3397
0.3058
1080
0.3018
0.2683
0.0968
0.3005
0.2672
Table 33: Similarity measures between test gallery and query set on MSVD. The nearest neighbor is considered with respect the gallery set.
Text-to- Videoo Retrieval
Video-to- Retrieval
Method
Normalization
MeunSim
MeanSim I
MeanSim 10
MeanSim
McanSim @1
MeanSimé 10
CE+
0.1029
0.6456
0.6188
0.1029
0.6037
0.5235
+NVGC
0.0159
0.4832
0.4624
0.0358
0.3979
0.3408
+NNGCC w/LLCALADI
0.0450
0,4501
0.4279
0.0384
0.3393
0.2870
TTACE+
0.1170
0.6417
0.1170
0.5997
0.531
+NVGC
0.0613
0,5144
0.4924
0,1019
0,5075
0.4456
+INVGC w/[.OCALADI
0.0421
0.4581
0.4369
0.0574
0.3807
0.3347
CLIP4Clip
0.1273
0.2686
0.2606
0.1273
0.2483
0.2242
+IvGGC
0.1283
0.2674
0.2593
0.1273
0.2473
0,2233
+IVGGC w/LOCALADI
0.0635
0.1982
0. 1887
0.1167
0.2359
0.2119
CLIP2Video
0.1413
0,3032
0.2931
0.1413
0.2779
0.2500
#NYGC
0.1417
0.3031
0.2930
0.1416
0.2771
0,2492
#IVGGC w/LOCALADI
0.1073
0,2635
0.2526
0.0770
0.1992
0.1723
X-CLIP
0.1672
0.2943
0.2858
0.1672
0.2728
0.2487
+NNGC
0.1678
0.2940
0.2853
0.1675
0.2722
0.2482
+IVGGC w/LOCALADI
0.1621
0.2893
0.2805
0.1518
0.2564
0.2320
Table 34: Similarity measures between test gallery and query set on DiDeMo. The nearest neighbor is considered with respect to the gallery set.
Text-to- Video Retrieval
Video-to- Retrieval McanSim@1
Method
Normalization
McanSim
MeanSim @
McanSim @ 10
McanSim
McanSim @ 10
CL+
-0.0068
0.2960
0.2379
-0.0068
0.2888
0.2369
#NVGC
-0.0066
0.2824
0.2265
-0.0067
0.2825
0.2316
+NVGGC W/LOCALADI
-0.0135
0.2715
0.2165
-0.0148
0.2338
0.1897
TT-CE+
-0.002
0,4218
0.3485
-0.002
0.4067
0,3439
#INGGC
+0,0007
0.3374
0.2755
-0.0012
0.3747
0.3162
+INVGC W/LOCALADI
-0.0082
0.2832
0.2302
-0.019
0.2396
0.1983
Note that the radius here is not the neighbor- hood coverage where they have a relation as 2 2v 12 by simple triangle caleulation. Since the basis of CRXa an n dimensional hypersphere of radius and is the key factor we
interested in, denote the spherical caps using as,
Cap Cn,x,a
Since the cosine similarity applied is indepen-

---

dent of the norm of each point in G and Q, we can assume without loss of generality that all the points in G and Q are on the unit sphere, i.e. Vx E GUQ, x E Sn-1- Since query data follows an independent and identical uniform distribution in IR" therefore it still follows an independent and identical uniform distribution on Sn-1.
of CapRxx. is an n dimensional hypersphere of radius r b, denoted as Bn Therefore Capa,x6 includes a cone C1 with basis as Bn-1. 1,b and height h with the same basis but height 2 b includes Br,b-
v1- -1- Then, a cylinder C2
Note that we have h > 6² by Taylor expansion. Based on all the facts, we have:
Note that the probability of X1 is conrectly re- trieved is lower bounded by the probability that the corresponding query point q E Ca for g must be the nearest neighbor of X1.
V (Capnxx.) > V(C)) V (Bn-1.)
= VC2) h
2bn
Given uniform distribution, the probability is
h
((Capn.xy.b) A(S,-1) where the A is the operator for area of hypersurface
> V (Bmb))
2bn
b
V (Bn.)) 1n
Generally, bounding volume is easier than the surface area, so we have the following Lemma Lemma 3 to help establish the relationship between the two.
Therefore,
Lemma 3. Therefore, the relationship be- tween the surface area A(Capn,x.b) and volumn V(Capnxx)) of n-dimensional sphere cap is,
b. V ((Capnx.))
b
V (Bm,b)
An ((n-1)
1n
V 1Bn,1)
b
4n
A(Capm,,,b) = V(CaPn,).)) (12)
1 ga++1
Proof. We have the relationship between the sur- face area A(S,-1) 1) and volume (SSn-1) 6 of n- dimensional unit hypersphere as,
4n
Upper bound: Based on proof of Lower bound,
notice that half of Bn.5 includes Capa,x, we have
A(S,-1)= = d dr V(Sn-1)= = V(S..-1) (13)
(Bn,,)
V (Capax,6)
2
Therefore, the same relationship can be adapted to sphere cap as well.
Therefore,
V (Capnxo)
1
V (Bn,))
With Lemma 3, we only need to bound the vol- ume V(Capnxx6). which is given by the following Theorem 4.
((n-1)
2
V (Br,1)
1
8"
2
Theorem 4. Given n-dimensional unit sphere Sn and spherical caps Capaxx on it, we have
Finally, we finish the proof using Lemma 3.
((Capux,)) (Sn-1)
1 gm+1
D.2 Proof of Corollary 2
2 bb >
An
Proof. Using the result from Theorem 1, we have the following inequalities for the probability of successful retrieval of bj and b2.
Proof. Lower bound: Follow the proof of Lemma 4.1 in (Micciancio and Voulgaris, 2010). The basis
n
EEerry time mention the hypervolume of sphere V(Cap..x.)) where Cap,.x.> Cu,x.a = Sn-1 O B...... we actually always refer to that of V(B..1 B..x)). We keep using expressions like V(Cap.,,.a) to prevent possible confusion when dealing with area and volume at the same time
P(x, <
b7.
2
And,
1 bb+
P (1)
'When mentioning the hypervolume of a sphere Sa-1, we actually always refer to that of B
Therefore, we have,

---

b1)
2n by
b1
P(X1,b2) bz)
b2
Note that 2/b2 is constant with respect n, so we finish the proof.

---

10
0.7518
0,2462
Asgg
0,
0.0
0/2
-02
-0.5
-0.5
04
-02
(a) CLOTHO (AR-CE, T2A). (b) CLOTHO (AR-CE CE, A2T).
(C) MSCOCO (CLIP, T2D)
(d) MSCOCO (CLIP, 12T).
Agg 0.7894
Agg
0.9570
Aag 0.4155
Ago
0.8179
04
02
0.00
-0.0
-50
-25
25
-52
(c) MSCOCO (Oscar, 12T).
Flickr30k (CLIP, 12T).
Flickr30k (Osear, T2I).
(h) MSVD (CLIP2Video, T2V).
0,7484
0.0164
0.50
0.00
0.0
-0.1
-0.2
-0.75
-0.5
(i) MSVD (CLiP2Video, V2T). () MSR- VTT (X-,LIP, T2V).
(k) ActivityNet (CE+, V2T).
() MSVD (CE+, T2V)
Ase 0.7156
Assg 0,0758
0.4 Dasg
0.8916
Asgg 0.9736
0.
0.0
0.0
-03
1.0
0.0
(m) MSVD (TT-CE+, T2V)
(n) MSVD (TT-CE+ V2T)
(0) MSVD (CLIP4Clip. T2V),
MSVD (CLIP4Clip, V2T).
0.8961
0.9165
1.5
0.6183
02
0.1
02
0.0
-0.1
-02
-0.-mmm
-0.3
-10
0.0
(g) MSVD CLIP, T2V).
(r) MSVD (X CLIP, V2T).
DiDeMo (CE+, T2V).
(1) DiDeMo (CE+, V2T).
Argg 0.6794
Arg 0.0304
10
05
-2.
(u) DiDeMo (TT-CE+, T2V). (v) DiDeMo (TT-CE+, V2T).
Figure 5: The prevalence representation degeneration problem across various methods and datasets. Each figure illustrates the distribution of the gallery set's representation for both retrieval tasks. Dimension reduction is performed using PCA, and the first two principal dimensions are chosen.

---

33-
33
32
InvGC w/LocalAdj 0% - 121
322
(MSCOCO)
T21
InvGC w/oocalAd Us
31
31
(MSCOCO)
0.00
0.02
0.04
0.06
0.08
0.10
15
Fg
(a) Image-to Text R@ w.n. rg
Text
(h) Image- -oa--oo--eet-t--e Text R I w.r.t
12T R@1
12T R
InvGC w/LoralAd
51
CLP (MSCOCO)
48
InvGC W/LocalAd 0.011
CLIP (MSCOCO)
50
0.00
0.02
0.04
0.06
0.08
0.0
0.5
1.0
1.5
2.0
(c) Tmxae R@1 w.r.
(d) Text-to-lmage R@1 I w.r.t Tq
Figure 6: Hyperpapameters sensitivity of INVGC w/LOCALADI MSCOCO.


Reviews:
[{"review_id": "jjQzo719hb", "reviewer": "Reviewer_2BeH", "summary": "", "questions": "See Weaknesses", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. Below please find the responses (R) to some specific questions (Q).\n\n**Q(A) [Improvements are minor]:** \n\n>R: Different from finetuning or pretraining methods, our proposed $\\textbf{InvGC}$ is a general **post-processing method** that does **not need any training or finetuning** on the model. It offers a ‘free’ retrieval improvement in performance without any training. From experimental results, we observe consistent improvements over base methods [1,2,3] after employing InvGC. Specifically, the improvement of R@1 of $\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ over CLIP is **2.77%** and **2.22%**, respectively for text-to-image and image-to-text retrieval, on MSCOCO. \n\n**Q(B) [Reproducing SOTAs (previous method)]:** \n>R: Thank you for your question!\n>1. As our proposed $\\textbf{InvGC}$ is the **first post-processing** method addressing the data degeneration problem in the cross-modal retrieval, we design average pooling (i.e. $\\textbf{AvgPool}$, Appendix C.5, Page 17) to serve as a baseline. Specifically, $\\textbf{AvgPool}$ is the simplest variant of inverse graph convolution with binary adjacency matrix where the edge weight between a pair of nodes is 1 if they are neighbors, or 0 otherwise. \n>2. As for reproducing base methods, we use the public codes and default hyperparameters introduced in their papers. The details can be found in Table 10 (Page 16) in the appendix. \n\n**Q(C) [Parameters, training details]:**\n>R: Thank you for your comment! Our proposed $\\textbf{InvGC}$ is a **post-processing method that does not require any training procedure**. \n> - $\\textbf{Search space of hyperparameters}$: We set the search space of $r_g$ as [0,1] since it is used between the same modality, while for $r_q$, we use [0, 5]. We will report the hyperparameters selected in our experiments in the final version of our paper.\n> - $\\textbf{Determining hyperparameters}$: We did use a validation set for selecting the hyperparameters, i.e., $r_g$ and $r_q$. For the dataset that does not have a validation set, we use a subset of the training set to determine the hyperparameters.\n> - $\\textbf{Training details}$: As our proposed method $\\textbf{InvGC}$ is a post-processing method, it does not have any training process. It runs on the embeddings generated by trained base models [1,2,3]. The details of training base models can be found in Table 10 (Page 16) in the appendix of our paper.\n\n**Q(D) [Details of the experiments]:**\n>R: Thank you for the suggestion! Due to the space limitation, the details of experiments are presented in Appendix C (Page15  - Page17), including benchmarks, methods, implementation of the proposed methods, and computational complexity. Moreover, to facilitate further research, we will release our code with detailed instructions upon publication.\n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\n[1]Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293–304.\n\n[2] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, volume 12349 of Lecture Notes in Computer Science, pages 214–229. Springer.\n\n[3] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan,Ji Zhang, and Rongrong Ji. 2022. X-CLIP: end-  to-end multi-grained contrastive learning for video- text retrieval. In MM ’22: The 30th ACM  International Conference on Multimedia, Lisboa,  Portugal, October 10 - 14, 2022, pages 638–647.  ACM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper's overall structure is clear, the motivation described in the Introduction is easy to understand, and the previous work is summarized as well as possible in the Related Work.\n\nB. The issues concerned in the article are important. The overall idea is clear and easy to understand.\n\nC. Solid experiments and some theories support the paper, but some experimental details are not explained.\n\nD. The proposed method is described in sufficient detail.\n\nE. The ablation experiment was thoroughly studied.\n", "reasons_to_reject": "A. The results were an improvement over the previous baseline but not much. In my opinion, there is an emphasis on comparing to the state of the art and demonstrating strong performance and it takes effort to find the place where differences to prior work and main factors for the performance differences are described.\n\nB. The experimental scenarios can not simplify comparisons with other published works in the literature. I would have liked clear statements about how the results proposed here replicate or challenge previous experiments in other papers.\n\nC. The range in which the parameters were searched is not given and it is not clearly written how the (validation?) data were used for model selection. How was the training loop stopped? How many epochs need to be trained on different datasets? Any early stopping?\n\nD. The paper needs to give more details of the experiment to be convincing.\n", "questions_to_authors": "See Weaknesses", "paper_topic_and_main_contributions": "This paper addresses the \"representation degeneration problem\" in cross-modal retrieval. To address this problem, the authors propose a novel method called INVGC, which establishes a graph topology structure within the dataset and applies graph convolution to effectively increase the distance between data points and separate representations. They also propose an advanced graph topology structure called LOCALADJ to improve the efficiency and effectiveness of INVGC. Experimental results show that both INVGC and INVGC w/LOCALADJ significantly alleviate the representation degeneration problem and improve retrieval performance.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {}}, {"review_id": "WeUa7q4Lq3", "reviewer": "Reviewer_6sSa", "summary": "", "questions": "1. How to observe the representation degeneration problem of different datasets from Figure 4.\n2. How to set the percent p.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. Below please find the responses (R) to some specific questions (Q).\n\n$\\textbf{Question}$: \n\n**Q(1) [How to understand Figure 4]**:\n> R: Thank you for your question! The severity of the data degeneration problem is proportional to **the convexity of the cones** that the embedding of data is located in. We notice that a large score of data degeneration score (i.e. $\\Delta_{deg}$) always leads to more concentrated data as shown in Figure 4. For different methods with the same dataset (same dataset and same retrieval task), e.g., Figures 4(s) and 4(u), Figures 4(p) and 4(r), more gathering in a convex cone (similar to a triangle in 2d space), the larger the score, which can be clearly observed in the figure. However, it is hard to compare the degeneration problem between datasets since their distributions are different. \n\n\n**Q(2) [How to set percent $p$]**:\n> R: Thank you for the nice catch. $p$ controls the reception field of $\\textbf{AvgPool}$. We uniformly chose 4 values, as $[0.1, 0.25, 0.5, 1]$, to approximately find the best performance. In practice, the optimal value of $p$ can be selected via a validation set. However, we don’t recommend using $\\textbf{AvgPool}$ (served as a baseline) since it is beaten by our proposed methods, $\\textbf{InvGC}$, with a large margin based on experimental results.\n \n$\\textbf{Reason to reject}$:\n\n**Q(1) [The theory in section 2.2]**:\n> R: We agree with you. It’s likely the case that both similarities are high when data degeneration is severe. In practice, we normally only need top-$k$ neighbors when using recommendation systems. To simplify the proof and have a clear presentation, we only consider the similarity between nearest neighbors. And we believe that our theoretical findings can be easily extended to the top-$k$ scenario. \n\n\n**Q(2) [Computation of adjacency matrix]**:\n> R: Thank you for the good catch. We do not compute the adjacency matrix of test queries for three reasons:\n> 1. [Pratical settings]: The query data in practice is very **unstable** since it comes separately and independently (in practice each query cannot see each other), which is exactly the opposite case compared to gallery data.\n> 2. [Theoretical analysis]: The margin of correct retrieval is proved to be dependent on the distribution of the gallery set according to Theorem 1. The large distance between the representation of nearest neighbors in the gallery set, the exponentially large probability of achieving correct.\n> 3. [Experimental results]: Considering the convolution of the test query set when computing the adjacency matrix will **double the number of hyperparameters** in a post-processing method, which is very undesirable if it’s not able to significantly improve the performance.\nWe also add experiments to compare the performance of the **original setting** (only considering the convolution of the test gallery set) with the setting (both the convolution of the test query and gallery set, noted as **‘two-side’** here). Due to the limited space, we only present the Text-to-Image R@1 on MSCOCO and MSR-VTT. Results are presented in the following Tables 1 and 2.  For each task, all 4 hyperparameters of the two-side setting are tuned to achieve the best performance. Note that it’s very common that the optimal value of the hyperparameters for the query set is 0, with which the **two-side setting degrades to the original setting ($\\textbf{InvGC}$)**. Such cases are noted as degraded after R@1. \n>\n>In fact, two-side (computing adjacency matrix for test query and gallery) degrades to one-side (only computing adjacency matrix for test query, $\\textbf{InvGC}$) for all the following experiments. This experimental study supports the claims in the two reasons mentioned above that the inverse graph convolution of the gallery set is all that matters when computing the adjacency matrix. Doing so might hurt the practicality of the proposed methods by making them more time-consuming and unstable. \n>\n>Thank you for your suggestion! We will add a careful and complete discussion of this in the revised version of the paper.\n\n$\\textbf{Table 1: Performance of two-side setting on MSCOCO}$\n\n| Method           | Setting                                            | R@1 of Text-to-Image  |\n|------------------|----------------------------------------------------|--------------------------------|\n| $\\textbf{CLIP}$  |                                                    | 30.34                          |\n|                  | +$\\textbf{InvGC}$                                  | 32.70                          |\n|                  | +$\\textbf{InvGC}$ (two-side)                       | _32.70_ (degraded)             |\n|                  | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$            | 33.11                          |\n|                  | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ (two-side) | _33.11_ (degraded)             |\n| $\\textbf{Oscar}$ |                                                    | 52.50                          |\n|                  | +$\\textbf{InvGC}$                                  | 52.63                          |\n|                  | +$\\textbf{InvGC}$ (two-side)                       | _52.63_ (degraded)             |\n|                  | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$            | 52.93                          |\n|                  | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ (two-side) | _52.93_ (degraded)             |\n\n\n$\\textbf{Table 2: Performance of two-side setting on MSR-VTT}$\n\n| Method               | Setting                                            | R@1 of Text-to-Image  |\n|----------------------|----------------------------------------------------|--------------------------------|\n| $\\textbf{CE+}$       |                                                    | 13.51                          |\n|                      | +$\\textbf{InvGC}$                                  | 13.57                          |\n|                      | +$\\textbf{InvGC}$ (two-side)                       | _13.57_ (degraded)             |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$            | 13.83                          |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ (two-side) | _13.53_ (degraded)             |\n| $\\textbf{TT-CE+}$    |                                                    | 14.51                          |\n|                      | +$\\textbf{InvGC}$                                  | 14.62                          |\n|                      | +$\\textbf{InvGC}$ (two-side)                       | _14.62_ (degraded)             |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$            | 15.08                          |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ (two-side) | _15.08_ (degraded)             |\n| $\\textbf{CLIP4CLIP}$ |                                                    | 44.10                          |\n|                      | +$\\textbf{InvGC}$                                  | 44.40                          |\n|                      | +$\\textbf{InvGC}$ (two-side)                       | _44.40_ (degraded)             |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$            | 44.40                          |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ (two-side) | _44.40_ (degraded)             |\n| $\\textbf{X-CLIP}$    |                                                    | 46.30                          |\n|                      | +$\\textbf{InvGC}$                                  | 47.30                          |\n|                      | +$\\textbf{InvGC}$ (two-side)                       | _47.30_ (degraded)             |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$            | 47.10                          |\n|                      | +$\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$ (two-side) | _47.10_ (degraded)             |\n\n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a post processing method for cross modal retrieval, which can alleviate representation degeneration problem without any training process or additional data.\n2. Several CLIP based models are improved by InvGC.", "reasons_to_reject": "1. The theory in section 2.2 seems flawed. \"A higher \\delta_deg score leads to bad retrieval performance\" is counterintuitive. I think when the degeneration issue occurs, not only the similarity between the closest data is higher, but also the similarities between other samples are higher. Therefore, only observing the similarity between neighboring samples is one-sided.\n2. Why the adjacency matrix is computed between the test galary samples and training query/galary samples. What about the test query samples? Why are the presentations of test query samples not updated?", "questions_to_authors": "1. How to observe the representation degeneration problem of different datasets from Figure 4.\n2. How to set the percent p.", "paper_topic_and_main_contributions": "The paper proposes a post-processing technique InvGC for cross-modal retrieval, which is inspired by graph convolution and aims to solve the representation degeneration problem caused by CLIP, i.e., the representations are gathered in a very narrow cone in the embedding space.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xgEoxu4LxF", "reviewer": "Reviewer_LxR3", "summary": "", "questions": "\tThis paper seems to acquire some validation data. The practicality of the setting should be further clarified.\n\tThe mathematical notations are confusing. For example, $n$ in Theorem 1 refers to the dimension of the representation while it refers to the data number in Eq. (2). I strongly recommend the authors to carefully revise all ambiguous notations for correctness.\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. Below please find the responses (R) to some specific questions (Q).\n\n$\\textbf{Questions For The Authors}$:\n\n**Q(1) [Validation data is needed]:**\n>R: Yes, we do need validation data to select the hyperparameters, i.e., $r_g$ and $r_q$. For the dataset that does not have a validation set, we use a subset of the training set to determine the hyperparameters, which improves the practicality of the proposed methods. Similarly, in practice, researchers can split the training set into training and validation sets for selecting the hyperparameters.\n\n**Q(2) [The practicality of the method]:**\n>R: Thank you for your suggestion! We use a validation set, or a very small subset of the training set, to determine the hyperparameters, which is standard for a post-processing method. Also, we only have **2 (3)** hyperparameters used in  $\\textbf{InvGC}$( $\\textbf{InvGC}$ w/$\\textbf{LocalAdj}$) for each task, and the performance of retrieval is not sensitive to all of them as shown in Figures 5 and 6 (Page 29). Detailed settings and implementations including benchmarks, methods, implementation of the proposed methods, and computational complexity are presented in Appendix C (Page15  - Page17).\n\n**Q(3) [Ambiguous notations]:**\n>R: Thank you very much for your suggestion! We will thoroughly and carefully revise the paper to avoid any confusion that might occur.\n\n$\\textbf{Missing Reference}$:\n>Thank you for the suggestion!\n> - [1] proposed a novel and strong method, called DART, to tackle Twin Noise Labels in visible infrared person re-identification. Similar to our study, the problem is also related to the issue of data quality of cross-modal tasks but in a way of addressing noisy annotation and correspondence. \n> - [2] addressed a similar issue and proposed Noisy Correspondence Rectifier to learn with noisy correspondence. \n>\n>Though not dealing with the same problem as our study, both papers offered insightful ideas for tackling data-related problems and archiving robustness performance in cross-modal matching and retrieval, which serves as a pretty good reference to enrich our work.\n>\n> We will revise our paper carefully and add the discussion with these papers in our final version.\n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\n\n[1] M. Yang, Z. Huang, P. Hu, T. Li, J. Lv and X. Peng. Learning with Twin Noisy Labels for Visible-Infrared Person Re-Identification. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 14288-14297, doi: 10.1109/CVPR52688.2022.01391.\n\n[2] Huang, Zhenyu and Niu, Guocheng and Liu, Xiao and Ding, Wenbiao and Xiao, Xinyan and Wu, Hua and Peng, Xi. Learning with Noisy Correspondence for Cross-modal Matching. 35th Conference on Neural Information Processing Systems (NeurIPS 2021), page 29406--29419.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\tThis paper extends the scope of the limited convex cone (as representation degeneration) problem into the cross-modal retrieval task, revealing the influence of this problem in the task. I think it is an interesting and practical problem for researchers in the community.\n\tThe proposed method is simple but effectively alleviate the influence of representation degeneration problem for cross-modal retrieval task, as verified with the experiment results.\n\tExtensive empirical results validate the effectiveness of the proposed method.\n", "reasons_to_reject": "\tThis paper seems to acquire some validation data. The practicality of the setting should be further clarified.\n\tThe mathematical notations are confusing. For example, $n$ in Theorem 1 refers to the dimension of the representation while it refers to the data number in Eq. (2). I strongly recommend the authors to carefully revise all ambiguous notations for correctness.\n", "questions_to_authors": "\tThis paper seems to acquire some validation data. The practicality of the setting should be further clarified.\n\tThe mathematical notations are confusing. For example, $n$ in Theorem 1 refers to the dimension of the representation while it refers to the data number in Eq. (2). I strongly recommend the authors to carefully revise all ambiguous notations for correctness.\n", "paper_topic_and_main_contributions": "This paper studies the representation degeneration problem and reveals the influence of the problem for the cross-modal retrieval task. To solve the intractable problem, the authors propose a simple but effective method called INVGC, which effectively separates representations by increasing the distances between data points. Extensive empirical results validate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {"Missing_References": "\tSome papers (e.g., Learning with Twin Noisy Labels for Visible-Infrared Person Re-Identification, Learning with Noisy Correspondence for Cross-modal Matching) on robust cross-modal retrieval topics are encouraged to be discussed."}}]

Author rebuttal:

Meta-review: This research proposes a novel post-processing methodology to mitigate data degeneration in cross-modal retrieval.  

Pros:
*The research methodology is robust and clear. 
*The results and experiments are well-supported by ablation studies.

Cons:
* The feasibility of the setting requires additional elaboration.
* The mathematical notation is difficult to follow
Decision: Accept-Findings