Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks
Authors: Yimu Wang
Xiangru Jian
Bo Xue
Venue: EMNLP 2023 Main (2023)
Keywords: Cross-modal Retrieval
Hubness
TL;DR: We propose a novel method using query and gallery banks to mitigate hubness in cross-modal retrieval and as a by-product, the performance is also improved.

Abstract: In this work, we present a post-processing solution to address the hubness problem in cross-modal retrieval, a phenomenon where a small number of gallery data points are frequently retrieved, resulting in a decline in retrieval performance. We first theoretically demonstrate the necessity of incorporating both the gallery and query data for addressing hubness as hubs always exhibit high similarity with gallery and query data. Second, building on our theoretical results, we propose a novel framework, Dual Bank Normalization (DBNorm). While previous work has attempted to alleviate hubness by only utilizing the query samples, DBNorm leverages two banks constructed from the query and gallery samples to reduce the occurrence of hubs during inference. Next, to complement DBNorm, we introduce two novel methods, dual inverted softmax and dual dynamic inverted softmax, for normalizing similarity based on the two banks. Specifically, our proposed methods reduce the similarity between hubs and queries while improving the similarity between non-hubs and queries. Finally, we present extensive experimental results on diverse language-grounded benchmarks, including text-image, text-video, and text-audio, demonstrating the superior performance of our approaches compared to previous methods in addressing hubness and boosting retrieval performance.

Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks
Yimu Wang" and Xiangru Jian
Bo Xue
University of Waterloo
City University of Hong Kong boxue4-cêmy ci tyu. edu. hk
Cyimu. wang, xiangru. jaaaangr..iianuwwaaaaaae jiannnewattrrlo...... ca
Abstract
The Hubness Problem
Querrees
Retrieval Results
In this work, we present post-pree post-processing so- lution to address the hubness problem cross- modal retrieval, phenomenon where small number of gallery data points are frequently retrieved, resulting in decline in retrieval per- formance. We first thcoretically demonstrate the necessity of incorporating both the gallery and query data for addressing hubness hubs always exhibit high similarity with gallery and query data. Second, building on our theoret- ical results, we propose novel framework, Dual Bank Normalization (DBNoRM). While previous work has attempted to alleviate hub- ness by only utilizing the query samples, DB- NORM leverages two banks constructed from the query and gallery samples to reduce the occurrence of hubs during inference. Next, to complement DBNORM, we introduce two novel methods, dual inverted softmax and dual dynamic inverted softmax, for normalizing similarity based on the two banks. Specifi- cally, our proposed methods reduce the sim- ilarity between hubs and queries while im- proving the similarity between non-hubs and queries. Finally, present extensive experi- mental results on diverse munguage-grounde benchmarks. including text-image, text-video, and text-audio, demonstrating the superior per- formance our approaches compared to previ- ous methods in addressing hubness and boost- ing retrieval performance. Our code is avail- able at https: //gi: thub. com/y cww/ywimwangeeeeeeocooommmoeeeee hwwwwwwgwww//////////// Better_Cross_Modn Retrrivvl
Query
Results
6G
Cosine similarity
gillary da spario
Hluh Gallery
dexcles simiario
Inverted Softmax and Dynamic Inverted Softmax
Queries
Query Bank
Retrieval Results
Query
Normalized similarity
8s
Gallery
DES's soletions Roduce similari with bank.
Dual Bank Normalization (Ours)
Queriees
Query Bank
Gallery Bank
Retrieval Results
Query
Results
S
82
Dual bank similarity
Our solations: Mourcrssed by checoentcal roadiu, we rellac the disiiaiitt betwom bubs and quclies viille inpooving die similanty betwerr aot-lads guacnis
Sh
8+
8a
Gallery
The hubness problem (Radovanovic et al., 2010) in cross-modal retrieval where queries q1- 92, and qs are associated with their respective galleries gi, 82, and ga- The hub (g) the nearest neighbor multiple queries (qi, q2, and q3). resulting to subopti- mal retrieval performance. Middle: Previous methods employ a query bank normalize similarities. Bottom: Our NBNORM tackles the hubness problem by utilizing both gallery and query banks. It reduces the similarity between the hub gi and all queries while simultaneously enhancing the similarity between non-hubs gz and gs and all queries, resulting in improved performance.
Figure 1: Top:
Yu et al., 2021; Frank et al., 2021; Fang et al.. 2023b), innovative retrieval methods (Wang et al., 2020b,a; Luo et al., 2022b; Jian and Wang. 2023), and modsl benchmarks (Xu et al., 2016; Kim et al., 2023), significant advances have been made in image-text retrieval (Liu and Ye, 2019; Rad- ford et al., 2021; Wang et al., 2021; Luo al., 2022b), video-text retrieval (Xu et al., 2021; Wu et al., 2022; Park et al., 2022a; Chen al., 2022; Park et al., 2022a; Zhao et al., 2022; Fang et al., 2022, 2023a; Wang and Shi, 2023), and audio-text
1 Introduction
Cross-modal retrieval (CMR) facilitates flexible information retrieval across various modalities, in- cluding images, videos, audio, and text, by extract- ing diseriminative features and summarizing infor- mation from multiple modalities. Recently, sup- ported by the development of pre-training multi- modal models (Tsai et al., 2019; Li et al., 2020a;
"Corresponding author.

---

retrieval (Koepke al., 2022), achieving satisfac- tory retrieval performance.
hub exhibits high similarity with data from vari- ous modalities, which shows the necessity of using gallery data. The second research question is:
In the most popular CMR paradigm, deep neu- ral networks are employed to project data from diverse modalities into a shared high-dimensional vector space, enabling direct comparison through distance measures such as cosine similarity and similarity. However, within this paradigm, the occurrence of hubness (Radovanovic et al., 2010) poses a challenge. Hubness refers to the phe- nomenon where certain data points, known as hubs, frequently emerge as nearest neighbors of other points, resulting in decline in retrieval perfor- mance. To empirically illustrate the existence of hubness, we visualize the distribution of the fre- quency which each gallery data point is retrieved by queries, shown in Figure 3 and Figure 6 in the Appendix. We observe that hubness is prevalent video-text, image-text, and audio-text retrieval, significantly impacting the performance of retrieval systems (Bogolin et al., 2022).
RQ2: How can we leverage gallery data to mitigate hubness?
Motivated by our theoretical results, instead of only using query data QBNomm, we propose uni- fied framework, namely Dual Bank Normalization (DBNoRM) as shown in Figure 1, along with two novel normalizing methods, namely dual inverted softmax and dual dynamic inverted softmax, which leverage both query and gallery data banks to alle- viate the occurrence of hubs by reducing the simi- larity between hubs and queries and improving the similarity between non-hubs and queries.
Finally, to evaluate the effectiveness of our pro- posed DBNORM, we conducted experiments on several cross-modal benchmarks, including four video-text retrieval benchmarks (Chen and Dolan, 201 1; Fabian Caba Heilbron and Niebles, 2015; Xu ct al., 2016; Hendricks et al., 2017), two image-text retrieval benchmarks (Lin et al., 2014; Plummer et al., 2017), and two audio-text retrieval bench- marks (Kim et al., 2019; Drossos et al., 2020). Ben- efiting from gallery and query banks, DBNORM outperforms previous methods without requiring access to any additional data.
Prior research on mitigating hubness can be roughly categorized into training methods (Liu al., 2020) and post-processing methods (Suzuki al., 2013; Dinu et al., 2015; Smith et al., 2017: Lample et al., 2018; Huang et al., 2019; Liu and Ye, 2019), with particular attention in zero-shot learn- ing (Lazaridou et al., 2015; Huang et al., 2020) and bilingual word translation (Smith et al., 2017; Huang et al., 2019, 2020). Due to the limitation of space, detailed related works are presented in Appendix A. In this paper, focus on addressing hubness in post-processing manner.
In summary, our contributions are as follows!:
We propose a unified framework DBNORM and two novel post-processing methods, namely dual inverted softmax and dual dy- namic inverted softmax. These methods effec- tively mitigate hubness by reducing the sim- ilarity between hubs and queries with query and gallery banks.
However, hubness in the context of CMR re- ceived limited attention until the introduction of QBNorm (Bogolin et al., 2022), which is pio- neering method specifically designed for CMR and incorporates novel technique called DIS. It ad- dresses hubness by selectively reducing the similar- ity between queries and hubs, thereby mitigating the impact of hubs. It follows the principle that hubs exhibit high similarity with query data to iden- tify hubs. This prompts us to investigate whether hubs also demonstrate high similarity with gallery data. Therefore, the first research question is:
Our proposed methods achieve state-of-the- art performance on cight cross-moue modall re- trieval benchmarks, outperforming previous approaches across multiple evaluation met- rics.
We are the first to theoretically demonstrate that hubs exhibit high similarity with data from different modalities and the necessity of utilizing both query and gallery data to ad- dress the issue of hubness.
RQI: Do hubs exhibit high similarity with gallery data?
To answer it and provide a theoretical validation for the principle of QBNorm, we theoretically demonstrate that hubness is a universal character- istic across different modalities. In other words, a
'The code is released at link.

---

2 Our Methods
Theorem 1. Assuming that x1 and x2 are sampled from distribution X with mean p. if x1 is closer to ia than X2, then X1 is more likely to be a hub than X2 on the space X, such that,
In this section, we first address RQI through a theo- retical demonstration of the universality of hubness and the necessity of incorporating both query and gallery data to address hubness. Next, to answer RQ2, we propose a novel method called DBNORM. which builds upon our theoretical analysis, along with two novel similarity normalization techniques: Dual Inverted Softmax and Dynamic Dual Inverted Softmax. These methods effectively utilize both query and gallery data to mitigate hubness.
E [[|x2 x11*] E
[ vx X.
Next, we demonstrate that hub will have high similarity (small distance) with points from another space (distribution).
Theorem 2. Assuming X1,X2 X and y V similarly, if X1 is closer to the mean point paz of the space X than x2, such that [|x2- -all ||
2.1 Preliminaries
p then even in a different space (distribution) y, x1 is still more likely to be a hub than X2, such that:
In this paper, we focus on cross-modal retrieval (CMR), aiming to learn a pair of encoders that map data from different modalities into a common space where they can be directly compared.
E
[1xx y|
EE ||| - yl vy
The query and gallery modalities are denoted X' and y. The (test) gallery, denoted by G (g...... &No ). contains all the embeddings of the gallery data, where NG is the size of the gallery data. In cross-modal retrieval, the gallery data does not overlap with the training data. Moreover, as our proposed methods require training data to address hubness, we define the sets of embeddings of train- ing query and gallery data as Q = (ÂI,..., ...,.. (.
Based on the previous theoretical results, we introduce the following corollary.
Corollary 3. Under the lg distance measure, hub will exhibit high similarity (small distance) with any point from X or
Remark 1. This corollary implies that a hub will be frequently retrieved by both the query and gallery points, as the similarity between that point and any other point will be high (small distance).
and G = {g)..... EN ). where No and No are the sizes. Finally, the query data is denoted as q.
Following QBNorm, we focus on practical sce- nario where only single query occurs a time and queries are unable to observe each other. In such a situation, we are limited to utilizing the training data for constructing banks.
Additionally, we extend our theoretical findings beyond the lz distance measure and explore the use of hyper-spheres, which can be regarded as using cosine similarity for measuring the distance between data points in CMR. This is particularly relevant since the cosine similarity is widely used in recent work (Luo et al., 2022a; Gao et al., 2021).
2.2 RQ1: Theoretical Analysis
In cross-moda retrieval, we consider two spaces, denoted X' and Y, that contain embeddings or representations of data points from two modalities. We assume that the two spaces, X' and V, follow symmetric distributions although they may have different means, variances, and distribution types. Specifically, the mean of X and y are denoted as fy and My.
Theorem 4. Hubs will have high similarity with any point from X and y under the cosine distance measure.
Now, we have shown that it is necessary to use gallery and query data to identify whether point is a hub, as the hub will exhibit high similarity with any point under E and cosine metrics. Fur- thermore, our theoretical results also suggest that hubness can be better mitigated by utilizing larger data banks demonstrated by Smith et al. (2017); Bogolin al. (2022). All the proofs deferred to Appendix C due to the limitation of space.
First, we demonstrate that points (embeddings representations) close the mean point more likely to exhibit hubness compared points situ- ated farther away from the mean. We use the & distance measure to compute distances.
This assumption is reasonable since various symmetric distributions, such as Uniform, Normal, or Laplacian distrihu- tions, exist.
'With more data, the estimation of the mean will he more precise.

---

Algorithm 1 DBNORM
efficiency.
Require:
the query point q, the gallery set G. I: Construct dual banks Q and G from the train- Dual Bank
2.3.1 Normalization Methods
ing or validation sets. Construction.
As existing methods are either designed for only using the query bank or cannot be directly incor- porated into DBNORM', we propose two novel methods efficiently implement DBNORM, outlined below.
2: Calculate the unnormalized similarity sq.g: - sim(q. g.), Vi E [Ng]]
> -Similarity
Computation.
3: Calculate normalized similarity $q.g. Ví E [Nc], with DualIS (Equation (1)) or DualDIS (Equation (2)). Similarity Normalization. 4: return Ranking argsortic[Ne] $q.8.
First, drawing inspiration from IS (Smith et al., 2017), by inverting the softmax, and normalizing the probability over query and gallery banks rather than the test gallery, we propose Dual Inverted Softmax (DualIS). Given the query q and gallery gi. the normalized similarity &q.g. is calculated follows.
2.3 RQ2: DBNORM
To mitigate the pervasive hubness problem ob- served in various cross-modal retrieval bench- marks and tasks (Figures 3 and 6), we propose a novel approach named Dual Bank Normalization (DBNORM), summarized in Algorithm 1. DB- NORM first constructs banks from the training data of both modalities and then normalizes the similar- ity. Specifically, DBNORM reduces the similarity between the query and hubs in the gallery data, ie., data points that appear as nearest neighbors of many queries, and increases the similarity between the query and non-hub gallery data. DBNORM includes the following steps.
9g
65 q.g.
69
exp(B1sq.8)
5g
9.gi
Σ JN expl 8 (118g))))
(1)
exp(B28q.g)
65 9.g.
EjE[Ma] expl (328g.4.)))
where 31 and B3 are temperatures and 8q.g. is the similarity between and g,. We employ multipli- cation as an aggregation method to combine the normalized similarities from two banks because it effectively summarizes the information from both sides. A detailed comparison with other aggrega- tion methods presented in Appendix B.5. The intuition behind DualIS is to measure the proba- bility that the corresponding gallery retrieves the query data, rather than testing whether the query retrieves the corresponding gallery.
Dual Bank Construction. In order to iden- tify hubs in the gallery, DBNORM constructs two banks, Q = (qI....,qN, .N.... and G (êi....,ár ), where No and NG are the sizes of the two banks. As suggested by our theoret- ical results, using both query and gallery banks simultaneously allows for a better estimation of the hubness of a data point.
Previous research (Bogolin al., 2022) has shown that when the bank fails to effectively rep- resent the space, possibly due to sampling bias, the performance significantly degraded, even falling below that of unnormalized similarities, be- cause the similarities with non-hubs might be inac- curately normalized by IS and DualIS. To address this issue, we introduce the Dual Dynamic Inverted Softmax (DualDIS). Firstly, we precompute two activation sets, Ag and Ag. storing the indices of gallery data that might be hubs. Then, DualDIS only normalize the similarity with the gallery points in these sets to avoid inaccurately normalizing the similarity with non-hubs. Specifically, for a query
Similarity Computation. The similarity be- tween the query and the gallery is calculated as $q.. - sim(q. g:), Vi E [Na]. where sim(..- is a similarity metric.
Similarity Normalization. To normalize the similarity, we introduce two novel methods: Du- allS (Equation (1)) and DualDIS (Equation (2)).
These methods allow for the normalization of the similarity $q,g. Vi E [Nc].
Finally, with the normalized similarity $q.8. Vi E [Nc], we can obtain the ranking of gallery data. is worth noting that in practice. the query and gallery banks should be precom- puted and reused across all queries, resulting in significant improvements in computational
'In practice, queries are isolated from each other, making it infeasible to include CSLS (Lample et al., 2018) and GC (Dinu et al., 2015) 5) as they require queries to he visible to cach other. A detailed comparison with CSLS and GC is in Appendix B.6.

---

q. Vi [Nc]] we have,
Methods
Normalization
RØ51
10
Melk 4
Mal
MSR MSR-VTI VTT (rull splio
sq.g
sgg q.g:
* qqg.
RoME
Froven
32.5
10.7
41.2
71.2
17.0
if arg max $q.8, E Ag
12.62
53.87
46. 46.38
12.0
74.99
15
13.31
35,00
47.46
12.0
76.16
s
sc[Nol
13:31
57.46
4:81
$ gg. otherwise
+ DuallS
14.88
57.36
90.00
11.0
70.13
DuaIDI
14,88
57.36
50,00
11.0
70.13
68 4g.
if arg max sq.83 E Aq JE[NGl
14.11
50.78
10.0
63.31
IS
16.58
40.75
53.44
9.0
60.52
TT-CE=
4 DIS
16.99
40.73
53.44
9.0
60.51
8g.. otherwise
DualiS
17.06
41.63
54.25
5.0
60.10
DuilDS
17.02
8.0
61001
MSR-VIT (1k sphi)
where A, farg maxte & E [N.]]. and arg max ja jE[No] &g,, return the top k indices that maximize & (* é (g. q}). In other words, A, contains the indices of the top & retrieved gallery points by any bank point. As the activa- tion sets can be computed before inference and reused, thereby not increasing the computational complexity of inference. On the other hand, ben- efiting from these two activation sets, DualDIS is shown to more robust than DualIS.
Discrteoodebronk
43.4
72.3
81.2
14.8
43.5
2.0
ConteCLiP
46.1
71.5
52.1
2.0
15.1
Align&TTn
45.2
73.0
82.9
2.0
CLIPITV
2.0
N-Pool
46.9
72.8
52.2
2.0
14.3
TSD-Nes
47.0
74.5
83.8
2.0
13.0
CAMuR
47.3
2.0
44:10
71.20
R1:40
2.0
15.51
+15
44.20
71.70
$1.60
2.0
15.64
CLIP4CTip
44.30
81.
2.0
15.64
DualtS
45,00
2.0
DualiDSs
45.00
72.50
N2,10
2.0
15.32
46.00
31,60
81.
2.0
14.51
47.00
82.10
2.0
CLIP2Video
+ DIS
47.00
72,80
K2
2.0
1391
+ DualSS
47.30
73.20
K2.30
2.0
13.90
DualDDs
47,30
K2,30
2.0
3 Experiments
46.30
74.00
83.40
2.0
12,80
+ 15
18.60
24.10
$4.10
2.0
13:25
In this section, we first conduct experiments to demonstrate that DBNORM can efficiently improve the retricval performance as a by-product. Next, our ablation studies indicate that DBNORM effec- tively mitigates hubness by reducing skewness and exhibiting robustness to hyperparameters. For clar- ity, we use IS, DIS, DualIS, and DualDIS to rep- resent QBNorm (IS), QBNorm (DIS), DBNorm (DualIS), and DBNorm (DualDIS), respectively. Following QBNorm, queries remain independent of each other as in practice, queries do not always occur at the same time. Detailed comparison with GC and CSLS is deferred to Appendix B.6.
X-CLIP
48.60
84.10
2.0
13:35
DualtS
2.0
DualDDS
48.70
74.30
84.10
2.0
13.30
Table I: Text-to-Video Retrieval performance on MSR- VTT (full split and 1k split). Best in Bold and the second best underlined.
TT-CE+ (Croitoru et al., 2021), CLIP4Clip (Luo et al., 2022a), X-CLIP (Ma et al., 2022), and CLIP2Video (Park et al., 2022a), two image-text retrieval methods: CLIP (Radford et al., 2021) and Oscar (Li et al., 2020b), and one audio-text retrieval method: AR-CE (Koepke et al., 2022).
For evaluation metrics, we employ recall at Rank K (R t.@K. K, higher is better), median rank (MdR, lower is better), and mean rank (MnR, lower is better) as commonly utilized retrieval metrics in previous retrieval works (Radford al., 2021: Luo et al., 2022a; Ma et al., 2022).
3.1 Datasets, Experimental Settings, and Evaluation Metrics
While we mainly focus our experiments on stan- dard benchmarks for text-video retrieval, i.e., MSR- VTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011), ActivityNet (Fabian Caba Heilbron and Niebles, 2015), and DiDemo (Hendricks et al., 2017), we also explore the generalization of DB- NORM on two text-image retrieval benchmarks (MSCOCO (Lin et al., 2014) and Flickr30k (Plum- mer et al., 2017)) and two audio-text retrieval benchmarks (AudioCaps (Kim et al., 2019) and CLOTHO (Drossos et al., 2020)). We test DB- NORM with DualIS and DualDIS on five video- text retrieval methods: CE+ (Liu et al., 2019),
3.2 Quantitative results
In this section, we provide quantitative results for eight cross-modal retrieval benchmarks. We com- pare methods with IS (Smith al., 2017) and DIS (Bogolin et al., 2022), which are two represen- tative post-processing methods. Due to limitations of space, some retrieval results and detailed analy- ses are presented in Appendix B.3. Text-video retrieval. The text-to-vides results

---

Methods
Normalization
R
R6B5 1
MdR L
M J
Methods Notmalizatiin
Rei
R@:5 1
Re110 T
MdR 1 MaR
FSE
11.5
31.8
77.7
13.0
MSCOC0 (SK split)
HET
27.7
58.6
94.7
4.0
VILT
40.4
70.0
$1.1
VCM
40.8
72.8
98.2
ConterCT IP
43.9
74.6
85.8
2.0
6.7
ALIGN
45.6
69.8
78.6
Tell
42.6
73.8
98.7
CODIS
53.9
79.5
87.1
CLIPTTV
44.1
75.2
98.4
2.0
6.5
ALBEF
60.7
84.3
90.5
TS2-Net
41.0
73.6
20.31
$4.74
66.12
4.0
25.39
CAMoE
$1.0
77.7
IS
35.15
60:71
21.20
21.69
19.16
47.79
65.79
CLIP
DIS
25.16
60.70
71.19
3.0
21.69
IS
20.13
50.58
66.44
5.0
21.07
DualiS
37,93
63.36
73.37
3.0
21.23
CE+
20.20
50.50
DualDIS
37.92
63.35
73.37
3.0
21.17
DuallS
22.66
52.21
68.0 68.09
5.0
19.02
$2.50
80.03
87.96
1.0
10.68
DualDDS
22.35
51.96
19:22
52.77
80.03
87.99
10.94
23.29
56.42
73.78
4.0
13.59
Oscar
DDIS
53.47
80.02
87.09
1.0
12.34
IS
24.69
57.98
74.42
DuallS
53.86
80.39
88. 09
1.0
11.83
IT:CE+
DIS
24.65
57.84
74.31
4.0
13:35
DualDiS
53,91
80.57
88,10
1.0
11.59
DuallS
27.15
60.81
76.67
DualDiS
26.90
10.16
16.35
4.0
12.18
Flickr30k
41.85
VILT
55.0
82.5
89.8
IS
45.93
77.52
87.07
2.0
6.39
UNITER
68.7
89.2
93.9
CLIP4Clip
46.02
77.29
ALIGN
75.7
93.8
96.8
CODIS
79.7
94.8
97.3
DuallS
46.71
77.47
87.35
2.0
6.01
ALBEF
85.6
97.5
98.9
DuallS
46.76
87.28
58.98
83.48
90.
1,0
6.04
46.25
76.02
86.05
2.0
6.37
IS
57.78
83,40
90.16
1.0
6.20
IS
49.36
79.16
59. 36
CLIP
DIS
59.02
83.40
90.10
1.0
6.04
X-CLIP
+ DIS
49.39
78.60
87.97
2,0
5.80
DuallS
58.02
83.40
90.22
6.20
DuallS
49.96
78.51
88.62
DualDDS
50.17
78.03
88.06
1.0
5.61
DualDDS
59.02
83.42
90. 10
1.0
6,04
71.50
91.50
94,96
1.0
4.24
Table 2: Text-to-Videe Retrieval performance on Activ- ityNet. Best in Bold and the second best is underlined.
72.26
91.74
94
4.26
Oscar
DIS
22.26
91.74
94.88
1.0
4.26
DualIS
73,00
91.74
1.0
4.19
DualDIS
73,00
91.70
94.78
1.0
4.21
Methods
Normalization
R@1
R@ R
MdR 1 MaR
Table 4: Text-to-Image Retrieval performance on MSCOCO (5k split) and Flickr30k. Best in Bold and the second best is underlined. Due to the limitation of the computational resources, we only use 20% of data
ESE
18.2
44.8
89.1
7.0
Frozon
33.7
64.7
76.3
3.0
Con
47.3
76.9
Align&Tell
47.1
77.0
85.6
2.0
CLIPITV
47.0
76.5
X-Pool
47.2
77.4
86.0
2.0
9.3
from the training data to construct the dual banks.
CAMoE
49.8
79.2
23.94
51.98
69.00
4.0
18:16
IS
24.64
56.64
70:45
Methods
Normalizarion
R@11
R@51
R@10
MdR J MiR
CB+
+ DIS
24.66
$6.64
70.46
4.0
20.42
AudioCaps
DuillS
25.04
56.01
20.70
+ DialDIS
25.06
56.02
69.65
4.0
20.64
MoGE MMT
23.00
55.70
71.00
4.0
16.30
24.42
36.10
72,00
84,50
7.50
15
26.55
59.68
72.85
4.0
17.72
22.23
$4.49
70.54
5.0
15.89
TT-CB+
59.68
+15
23.19
55.96
70.05
4.0
20.44
+ DuilSS
27.21
60.23
73.35
4.0
16.88
CR
DISS
23.19
55.93
20.05
4.0
20.45
DwiIDS
60.20
41.64
74.66
83,99
2.0
10.32
+ -DualS
24.04
56.84
71.03
4.0
18:44
75.60
DuulDDS
56.81
71.03
15.44
CLINClip
+ DIS
46.05
75.60
N4.37
2.0
10.16
CLOTHO
DialIS
46.33
75.91
84.35
MoBE
6.00
20.80
32,30
23.0
60.20
+ DiiIDIS
46.34
75.95
84.34
2.0
10.12
MMT
6.50
21.60
66.90
67.70
47.05
76.97
85.59
I5
47.52
77.95
86.01
2.0
9.47
6.27
22.32
33.30
23.0
58.95
CLiP2Video
DIS
47.52
77.95
+1
6,83
23,50
35.04
22.0
56.52
AR-CE
DIS
6.83
23.46
34,97
56.61
+ DuilSS
47.95
77.95
86.22
2.0
9.30
DailDiss
47.97
77.93
86.20
+ DualIS
7:06
24.42
36.29
21.0
54,12
46.31
36.34
85.31
2.0
9.59
DualDIS
7.12
24.50
36.17
21.0
54.85
15
47.06
85.22
X-CLIP
+ DIS
47.06
77.43
85.22
2.0
10.34
Table 5: Text-to-Audio Retrieval performance on Au- dioCaps and CLOTHO. Best in Bold and the second best is underlined.
DialIS
47.95
78.36
86.00
+ DuiIDS
47.95
78.35
86,00
2.0
9.70
Table 3: Text-to- Video Retrieval performance on MSVD. Best in Bold and the second best is underlined,
space, the results for DiDemo and video-to-text retrieval results on MSR-VTT ActivityNet, and MSVD are deferred to the Appendix. Moreover, we compare our results with ofate-of-the-ann meth-
on MSR-VTT, AcvitityNet, and MSVD are pre- sented in Tables I to 3. Due to the limitation of

---

ods in video retrieval, i.e., FSE (Zhang et al., 2018), HiT (Liu et al., 2021b), ROME (Rony al., 2022), Frozen (Bain et al., 2021), DiscreteCodebook (Liu et al., 2022a). VCM (Cao et al., 2022), Center- CLIP (Zhao et al., 2022), Align& Tell (Wang et al., 2022b), CLIP2TV (Gao et al., 2021), X-Pool (Gorti et al., 2022), TS2-Net (Liu et al.. 2022b), and CAMoE (Cheng et al., 2021). Notably, our ap- proach, DBNORM, outperforms previous normal- ization methods by significant margin on all four benchmarks. Specifically, with DualTS and DualDIS, the retrieval performance CLIP4Clip yields the best performance, achieving R@l of 45.00 and R@5 of 72.50. Similar observations can be obtained on other benchmarks. However, it is important to note that using the activation sets (DualDIS) may have adverse effect on retrieval performance, possibly due to the biased construc- tion of these sets.
(Text) Query: cartoons sponge squid starfish Retrieval Results by CLIP4Clip
Retrieval Results by CLIP4Clip + IS/DIS
Retrieval Results CLIP4Clip DualIS/DualDIS
Text-image retrieval. Quantitative results are shown in Table 4 and Table 10 in the Appendix. Similar observations can be obtained MSCOCO and Flickr30k. DBNorm (DualIS and DualDIS) effectively improves R@ compared with IS and DIS. We also compare our results with ALBEF (Li et al., 2021), ViLT (Kim et al., 2021b), ALIGN (Jia et al., 2021), UNITER (Chen et al., 2020b), and CODIS (Duan et al., 2022).
Top-3 Text-to- Text-to-Viden retricval examples.
Query
CLIP4Clip
a woran talking in the tv channel
news reader deseribing about
gard scletioen
shst
tx.l bortz aal imnge dopkenng scro
Text-audio retrieval. Results are presented in Table and Table 11 the Appendix. We observed that employing DualIS and DualDIS leads to sig- nificant improvements in R@ I for text-to-audio retrieval on two benchmarks. Specifically, R@) on AudioCaps is increased to 24.04 with DualIS and DuaIDIS. Besides, we compare with MoLÉ (Miech al., 2020), MMT (Gabeur et al., 2020).
CLIP4Clip
CLIPPClipp DDDDDSD
cnn report is talking about their dogs
gad the dros warth spaaking and scectsion erier viaa aack com sxa bosde oll isnge dopliing woon
gati doan veating rins
news reader describing
huads stll
about a news
reader describing
talking the channel
about
(b) Top-3 Vidco-to retrieval examples.
Figure 2: Retrieval examples on MSR-VTT. More ex- amples are presented Figure 5 due to the limitation of space.
3.3 Qualitative Results
To qualitatively validate the effectiveness of our proposed methods, we present examples of video- to-text and text-to-video retrieval on MSR-VTI -TT in Figure 2 and Figure 5 in the Appendix, respectively. The retrieval results demonstrate that our proposed DualIS and DualDIS, leveraging gallery and query banks, exhibit better performance and robustness compared to IS and DIS. Specifically, Figure 2 (a) highlights the capability of DualIS and DualDIS to distinguish three distinct characters, while Figure (b) showcases specific case where the vanilla CLIP4Clip and CLIP4Clip with IS/DIS fails to cap- ture the presence man wearing a black shirt in
the background. In contrast, DualIS and DualDIS excel in capturing such intricate details, thereby yielding superior retrieval performance.
3.4 DBNORM
In this section, tions of DBNORM MSR-VTT and ActivityNet with CLIP4Clip. Due to the limitation of space, the questions on the sensitivity to B1, 82, and k in DualDIS, the relationship between skewness and performance, aggregation methods, compar-
several research ques-

---

MSRVTT
ActivityNet CLIP4Clip
MSCOCO
Normalization
Best
CF+
TT-CE+
CLIP4Clip
CLiP2VideD
X-CLIP
CE+
TT-CE+
X-CLIP
CLIP
Oscar
1.38
1.28
1.13
0.84
1.24
0.76
0.83
0.98
2.71
0.55
0.82
0.34
0.18
0.32
0.74
0.67
0.51
0.55
0.57
0.90
0.24
-DIS
0.83
0.34
0.18
0.33
0.74
0.68
0.52
0.42
0.44
0.90
0.22
2
DuaalS
0.37
0.33
0.57
0.26
0.70
0.54
0.37
0.36
0.42
0.42
0.31
DDualDDS
0.37
0.28
0.57
0.26
0.70
0.50
0.40
0.46
0.43
0.43
0.29
5
Table 6: The hubness (skewness score) on lext-video/image retrieval with CE+, TT-CE+, CLIP4Clip, CLIP2Video, X-CLIP, CLIP, and Oscar is better reduced after applying our proposed DualIS and DualDIS than IS and DIS. The Lower is better. Best in Bold and the second best is underlined. The hubness scores on other benchmarks, methods, and image/videoo/uudi-texx retrieval are deferred to Table 13 in the Appendix due to the limitation of space.
MSRAVTT (CE+)
MSRVVT (TT-CE+)
MSRVVT (CUP4CIPp)
MIR-VTT(-CCUL
1-ccurence
200
300-
medan- -2.0
max
L-accuences
200
100
400
max modaa 120
1-oecurence
medion 1.0
max
10.0
7.5
5.0
max medinn - -10
100
100
2.5
500
1500 2000
500
1000
1500
200
400
6.0
500
800
number videos
number videos
number
number
(a) MSR-VIT (CB+).
(b) MSR-VTT (TT-CE+).
(c) MSR-VTT (CLIP4Clip).
(d) MSR-VTT (X-CLIP).
ActivityNoot (CE+)
ActivityNot (TT-CE+)
ActivityNaat (CLIP4CIP))
ActivityNet (X: CLIP)
1-accurance
10-
max 33
L-accurence
10
max 12
1-occurence
max
L-cccuren
maxK
mechon 2.0
medion 3.0
median
1.0
2
1500
500
1500
1000 150010501500
300 number video
200
400
number videos
number videos
number video
(e) ActivityNet (CE+).
(f) ActivityNet (TT-CE+).
(g) ActivityNet (CLIP4Clip).
(h) ActivityNet (X-CLIP).
Figure 3: Hubness is prevalent across different methods, datasets, and tasks. These figures illustrate the distribution of the number of times each (test) gallery video retrieved by (test) queries. Columns (different models): Retrieval distributions for CE, TT-CE+ CLIP4Clip, and X-CLIB Rows (different datasets): Retrieval distributions for the same model on MSR-VTT and ActivityNet. The statistics (maximum, minimum, and median values) of 1-occurrence are shown in the figures. More illustrations are deferred to the Appendix (Figure 6).
Text-Video R81
M5R-VTT (CUP44Cpp)
Activity Not (CUP4CIp)
Tecto-ilee Reel
<5
MSR-VTT (CL/P4CPp)
MSR-VIT (CUPACIpp)
40
8n
bastline
baseline Dunls
baseline Duals
baseine
30-
Dualt
DualDI5
30
42
Duals
30
10
107
10a
10"
'0'
10a
10
10"
10a
101
10
10a
10t
10
10
10'
107
10
10th
10'
Size of Banks (Mg and No)
Size of Banks (Ng and Nal
Size of Gullery Bank No
Size of Query Bank Ma
(a) R@ w.t No and No-
(b) R@ w.r. N& and No-
(c) R@I w.r.t No.
(d) R@ w.r. Ng-
Figure 4: Text-to-Viee Retrieval R@1 (1 w.r. the size of gallery and query banks on MSR-VTT and ActivityNet using DualIs and DualDIS with CLIP4Clip.
ison with GC and CSLS, and the computational complexity are presented in the Appendix.
resulting in a negative impact on performance, which empirically validates the prevalence of hub- ness. To quantitatively evaluate hubness, following Radovanovic et al. (2010), we report the skewness (details are presented in Appendix B.5) of text-to- video/image/audio and video//magecauddiott--ess retrieval in Table 6 and Table 13 in the Appendix. The skewness is better reduced after employing DualIS and DualDIS than IS and DIS, proving the effectiveness our methods in alleviating hubness.
RQI: Can DBNORM alleviate hubness? First, to illustrate the presence of the hubness problem, present 1-occurrence in text-to- video/image/audio retrieval in Figure 3 and Fig- ure 6 in the Appendix. Through the visual- ization across various benchmarks and methods, we consistently observe that small subset of videos//mages/audio is retrieved by multiple times,

---

How much data is desired in the banks? To address this question, we conduct experiments on scaling the size of query and gallery banks by uniform sampling. The results of @@l I for text- video retrieval using DualIS and DualDIS are pre- sented in Figures 4 and 7. Our observations in- dicate that the size of the two banks increases, the performance improves. However, even with relatively small number of samples in the query and gallery banks, we still observe satisfactory per- formance. Moreover, we examined the individual impact of the query and gallery bank sizes by inde- pendently sampling them at different scales. The results demonstrate that the size of the query bank has a greater influence on performance compared to the gallery bank, although bigger gallery bank also leads to better performance.
RQ2:
MSCOCO, and AudioCaps datasets. Furthermore. we found that the hubness score (the skewness score) does not have an absolute correlation with the retrieval performance, as a low hubness score can still result in poor retrieval performance. the future, it would be interesting to improve the robustness of bank-based normalization techniques and explore alternative effective metrics for mod- eling the relationship between hubness (skewness) and retrieval performance.
A Acknowledgments
We first sincerely thank all the reviewers and chairs for their efforts helping improve our paper's pre- sentation and organization. Next, we thank Tamer for the computational support and Charlie for re- viewing the paper. Last but not least, we thank all the collaborators, friends, and computer science administrators for their support in Spring 2023.
4 Conclusion
In this work, we addressed the issue of hubness in cross-modal retrieval through a post-processing ap- proach. First, we theoretically proved that hubs exhibit high similarity with data from both the query and gallery modalities. Then, motivated by our theoretical results, we proposed a novel post-processing method, Dual bank Normaliza- tion, along with two novel methods, i.e., Dual In- verted Softmax and Dual Dynamic Inverted Soft- max, which leveraged gallery and query banks to reduce similarities with hubs and improve similari- ties with non-hubs gallery data. Finally, with exten- sive experiments across a range of tasks, models, and benchmarks, we demonstrated the superiority of our proposed methods over previous methods in addressing hubness and improving performance.
References
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zis- serman. 2021. Frozen time: joint video and image encoder for end-to-end retrieval. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 1708-1718, 18. IERE
Richard Bellman. 2015. Adaptive Control Processes A Guided Tour (Reprint from 1961), volume 2045 of Princeton Legacy Library. Princeton University Press.
Simion-Vlad Bogolin, Toana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. 2022. Cross modal trieval with querybank normalisation. IEFE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5184-5195. IEEE.
Limitations
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Hossert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learner. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
First, our proposed method, DBNORM, is capa- ble of tackling hubness in range of cross-modal and single-modal retrieval tasks. However, in this study, we focused on evaluating DBNORM in the context of text-video, text-image, and text-audio retrieval, while excluding audio-image, audio-text, and audio-image image retrieval tasks. It would be in- teresting to test the empirical efficiency of our methods on these tasks. Second, we observe that the performance improvement of NORO com- pared to single bank normalization is not signifi- cant on certain benchmarks, such as CLOTHO and MSVD. However, DBNORM has demonstrated sat- isfactory performance on MSR-VTT, ActivityNet,
Shugiang Cao, Bairui Wang. Wei Zhang, and Lin Ma. 2022. Visual consensus modeling for video-text re- trieval. In Thirty-Sixhh AAAI Conference on Artificial

---

Intelligence, AAAI 2022, Thirty-Fourthh Conference on Innovative Applications of Artificial Intelligence. TAAI 2022, The Twelveth Symposium Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 167-175. AAAI Press.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: an Audio Captioning Dataset. In ICASSP 2020 2020 IEEF International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 736-740. ISSN: 2379-190X.
Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang. Xu, Belinda Zeng, and Trishul Chilimbi. 2022. Multi- modal Alignment using Representation Codebook. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15630-15639, New Or- leans, LA, USA. IEEE.
David Chen and William Dolan. 2011. Collecting
highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 190-200, Portland, Ore- gon. USA. Association for Computational Linguis- tics.
Bernard Ghanem Fabian Caba Heilbron, Victor Escor- cia and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity un- derstanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-9700
Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2022. LiteVL: Effi- cient video-language learning with enhanced spatial- temporal modeling. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7985-7997, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. 2022. Multi-moda cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. 2020a. Fine-grained video-text retrieval with hierarchical graph reasoning. In 2020 IEFF/CVT Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA. June 13-19, 2020, pages 10635-10644. Computer Vision Foundation / IEEE.
Xiang Fang. Daizong Liu, Pan Zhou, and Guoshun Nan. 2023a. You can ground carlier than see: An effective and efficient pipeline for temporal sentence ground- ing in compressed videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2448-2460.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Fll Kholy. Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. UNITER: UNiversal Image- TExt Representation Learning, In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, vol- ume 12375 of Lecture Notes in Computer Science, pages 104-120. Springer.
Xiang Fang, Daizong Liu, Pan Zhou, Zichuan Xu, and Ruixuan Li. 2023b. Hierarchical local-globe gocallglobal trans- former for temporal sentence grounding. IEEE Trans- actions on Multimedia.
Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021. Vision-and-language vision-for- language? on cross-modal influence multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9847-9851 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. 2021. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. CoRR, abs/2109.044 09499.
Ioana Croitoru, Simion-Vias Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. 2021. Teachtext: Crossmodal generalized distillation for text-video retrieval. In 2021 IEEE/CVF International Conference on Com- puter Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 11563- 11573. IBEE.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020. Multi-modal transformer for video retrieval. In Compuder Vision - ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part IV. volume 12349 of Lecture Notes in Computer Science, pages 214-229. Springer.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adver- sarial training for rision-and-language representation learning. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. 2021. CLIP2TV: an empirical study on transformer-basee video-text retrieval, CoRR, abs/21 11 1.05610.
Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. ArXiv: 1412.6568 [cs].
methods for

---

Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. 2013. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 35(12))2916-922)
Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling Up Vi- sual and Vision-Language Representation Learning With Noisy Text Supervision. In Proceedings of the 38th International Conference on Machine Learning. pages 4904-4916. PMI.R. ISSN: 2640-3498.
Satya Krishna Gorti, Noel Vouitsis, Junwei Ma, Key- van Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022. X-pool: Cross modal language- video attention for text-video retrieval. In JEEE/CVF Conference Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1996-5005 IEEE.
Xiangru Jian and Yimu Wang. 2023. Invge: Robust cross-modal retrieval by inverse graph convolution.
Ming Jiang, Qiuyuan Huang. Lei Zhang. Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jian- feng Gao. 2019. TIGEr: Text-to-image grounding for image caption evaluation. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-ICCNLP), pages 2141-2152 Hong Kong, China. Association for Computational Linguistics.
Kazuo Hara, Ikumi Suzuki, Masashi Shimbo, Kei Kobayashi, Kenji Fukumizu, and Milos Radovanovic. 2015. Localized centering: Reducing hubness in large-sample data. In Proceedings of the Twenty- Ninth AAAI Conference on Artificial Intelligence, Jan- uary 25-30, 2015, Austin, Texas, USA, pages 2645- 2651. AAAI Press.
Eamonn Keogh and Abdullah Mueen. 2017. Curse of Dimensionality, pages 314-315. Springer US. Boston, MA.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In 2016 JEEE Conference on Computer Vision and Pultern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Chris Dongioo Kim. Bycongchang Kim, Hyunmin Lee. and Gunhee Kim. 2019. AudioCaps: Generating cap- tions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Pupers), pages I 119-132, Minneapolis, Min- nesota. Association for Computational Linguistics.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing Moments in Video with Natural I Language. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5804-5813, Venice. IEEE
Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xi- ang Yu, Stan Sclaroff, Kate Saenko, and Manmo- han Chandraker. 2021a. Learning modal con- trastive features for video domain adaptation. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 202 Montreal, QC, Canada, October 10- 2021, pages 13598-13607 IEEE.
Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, and Jie Lin. 2021. Learning cross- cross-modal retrieval with noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5403-5413, Computer Vision Foundation / IEEE.
Taehoon Kim, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Mark Marsden, Alessandra Sala, Seung Hwan Kim, Bohyung Han, Kyoung Mu Lec, Honglak Lec, Kyounghoon Bae, Xiangyu Wu, Yi Gao, Hailiang Zhang, Yang Yang. Weili Guo, Jianfeng Lu, Young- tack Oh, Jac Won Cho. Dong jin Kim, In So Kweon, Junmo Kim, Wooyoung Kang. Won Young Jhoo, Byungseok Roh, Jonghwan Mun, Solgil Oh, Ke- nan Emir Ak, Gwang- Gook Lec, Yan Xu, Mingwei Shen, Kyomin Hwang, Wonsik Shin, Kamin Lee. Wonhark Park, Dongkwan Lee, Nojun Kwak, Yujin Wang, Yimu Wang, Tiancheng Gu, Xingchang Lv, and Mingmao Sun. 2023. Nice: Cvpr 2023 challenge zero-shot image captioning.
Jaji Huang. Xingyu Cai, and Kenneth Church. 2020. Improving bilingual lexicon induction for low fre- quency words. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 1310-1314, Online. As- sociation for Computational Linguistics.
Jiaji Huang, Qiang Qiu, and Kenneth Church. 2019. Hubless nearest neighbor search for bilingual lexicon induction. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4072 4080, Florence, Italy. Association for Computational Linguistics.
Wonjae Kim, Bokyung Son, and Iddoo Kim. 2021b. ViLT: Vision-and-Languaee Transformer Without Convolution or Region Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 5583-5594. PMLR. ISSN: 2640- 3498.
Hervé Jégou, Hedi Harzallah, and Cordelia Schmid. 2007. A contextual dissimilarity measure for ac- curate and efficient image search. In 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), 18-23 June 2007, Minneapolis, Minnesota, USA JEEE Computer Society.
A. Sophia Koepke, Andreea- Oncescu, Joao Hen- riques, Zeynep Akata, and Samuel Albanie. 2022.

---

Audio Retrieval with Natural Language Queries: Benchmark Study. IEEE Transactions on Multime- dia, pages 1-1. Conference Name: IEEE Transac- tions on Multimedia.
In IEEE International Conference on Computer Vi- sion, ICCV 2017, Venice, Italy, October 22-29, 2017. pages 4097-4105. IEER Computer Society.
Alexander Liu, Young Jin, Cheng-T Lai, Andrew Rouditchenko, Aude Oliva, and James Glass, 2022a. Cross-modal discrete representation learning. In Pro- ceedings of the 60th Annual Meeting the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 3013-3035, Dublin, Ireland. Associa- tion for Computational Linguistics.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovie Denoyer, and Hervé Jégou. 2018. Word translation without parallel data, In Interna- tional conference on learning representations.
Angeliki Lazaridou, Georgiana Dinu, and Marco Ba- roni. 2015. Hubness and pollution: Delving into cross-space mapping for zero-shot learning. In Pro- ceedings of the 53rd Annual Meeting of the Asso- ciation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 270-280, Beijing, China. Association for Computational Lin- guistics.
Fangyu Liu and Rongtian Ye. 2019. A strong and robust baseline for text-image matching. In Proceedings of the 57th Anmual Meeting of the Association for Com- putcational Linguistics: Student Research Workshop. pages 169-176, Florence, Italy, Association for Com- putational Linguistics.
Fangyu Liu, Rongtian Ye. Xun Wang, and Shuaipeng Li. 2020. HAL: Improved Text- Image Matching by Mitigating Visual Semantic Hubs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages I 11563-11571.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L.. Berg, Mohit Bansal, and Jingjin Liu. 2021. Less is more: Clipbert for video-and and- language learning via sparse sampling. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2027, pages 7331-7344 Computer Vi- sion Foundation/ / IEEF
Haoliang Liu, Tan Yu, and Ping Li. 2021a. Inflate and shrink:enriching and reducing interactions for fast text-image retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 9796-9809, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in neural information processing sys- tems.
Song Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen, Wenkui Ding, and Zhongyuan Wang. 2021b. HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval. In 2021 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 11895-11905, Montreal, QC, Canada, IEEE.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020a. HERO: Hierarchical reni-representation
encoder for Video+Language pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2046-2065, Online. Association for Computational Linguistics.
Yang Liu. Samuel Albanie, Arsha Nagrami, and Andrew Zisserman. 2019. Use what you have: Video retrieval using representations from collaborative experts. In 30th British Machine Vision Conference 2019, BMVC 2019, Cardiff. UK. September 9- 2019, page 279. BMVA Press.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020b. Oscar: Object-semantic aligned pre- training for vision-language tasks. In Computer Vi- sion ECCV 2020- - 16th European Conference, Glas- gow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 121-137. Springer.
Yuqi Liu, Pengfei Xiong. Luhui Xu, Shengming Cao. and Qin Jin. 2022b. Ts2-net: Token shift and selec- tion transformer for text-video retrieval. In Computer Vision FCCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, volume 13674 of Lecture Notes in Computer Science, pages 319-335. Springer.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. Computer Vision ECCV 2014- 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V. volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer.
Huaishao Luo, Lei Ji, Ming Zhong. Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022a. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing. 508:293- 304.
Ziyang Luo, Yadong Xi, Rongsheng Zhang, GongZheng Li, Zeng Zhao, and Jing Ma. 2022b. Conditioned masked language and image modeling for image- text dense retrieval. In Findings of the Association
Venice Erin Liong, Jiwen Lu, Yap-Peng Tan, and Jie Zhou. 2017. Cross-modal deep variational hashing.

---

for Computational Linguisties: EMNLP 2022. pages 130-140, Abu Dhabi, United Arab Emirates. Associ- ation for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High- High-Performance Deep Learn- ing Library. In Advances in Neural Information Processing Systems, pages 8024-8035. Curran Asso- ciates, Inc.
Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022. X-CLIP: end-to- end multi-grained contrastive learning for video-text retrieval. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, Octo- ber 10 14, 2022, pages 638-6668888 063.-47.444 ACM.
Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic. and Andrew Zisserman. 2021. Think- ing fast and slow: Efficient text-to-visual retrieval with transformers. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 9826-98868 9836. Computer Vi- sion Foundation/ / IEEE.
Mandela Patrick, Yao Huang, Yuki Markus Asano, Florian Metze, Alexander G. Hauptmann, João F. Henriques, and Andrea Vedaldi. 2021. Support-set bottlenecks for video-text representation learning. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Antoine Miech, Ivan Laptev, and Josef Sivic. 2020. Learning a Text-Video Embedding from Incomplete and Heterogencous Data. ArXiv:1804.0255 [cs].
Bryan A. Plummer, Liwei Wang. Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. Int. J. Comput. Vis.. 123(1):74- 93.
Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K. Roy-Chowdhury. 2018. Learn- ing joint embedding with multimodal cues for cross- modal video-text retrieval. In Proceedings of the 2018 ACM on International Conference on Multime- dia Retrieval, ICMR 2018, Yokohama, Japan, June 11-14, 2018. pages 19-27. ACM.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, 1CML 2021, 18 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Andreea-Maria Onceseu, A. Sophia Koepke, João F. Henriques, Zeynep Akata, and Samuel Albanie. 2021. Audio retrieval with natural language queries. In In- terspeech 2021, 22nd Annual Conference of the Inter- national Speech Communication Association, Brno, Czechia, 30 August 3 September 2021, pages 2411- 2415. ISCA.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, and Antonio Torralba. 2016. Ambient sound provides supervision for visual learn- ing. In Computer Vision ECCV 2016 14th Eu- ropean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part I, volume 9905 of Lecture Notes in Computer Science, pages 801-816. Springer.
Milos Radovanovic, Alexandros Nanopoulos, and Mir- jana Ivanovi. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Ma- chine Learning Research, 11886):44772552 -2331.
Milos Radovanović, Alexandros Nanopoulos, and Mir- jana Ivanović. 2010. On the existence of obstinate results in vector space models. In Proceedings of the 33rd international ACM SIGIR conference Research and development in information retrieval. pages 186-193, Geneva Switzerland. ACM.
Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach. 2022a. Exposing the limits of rideo-text models through contrast sets. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 3774-5586,56 Seattle, United States. Association for Computational Linguistics.
Md Rashad Al Hasan Rony. Liubov Kovriguina, Deban- jan Chaudhuri, Ricardo Usbeck, and Jens Lehmann. 2022. RoMe: A robust metric for evaluating natural language generation. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 5645- 5657, Dublin, Ireland. Association for Computational Linguistics.
Yookoon Park, Mahmoud Azab, Seungwhan Moon, Bo Xiong, Florian Metze, Gourab Kundu, and Kir- mani Ahmed, 2022b. Normalized contrastive learn- ing for text-video retrieval. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 248-260, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Dominik Schnitzer, Arthur Flexer. Markus Schedl, and Gerhard Widmer. 2012. Local and global scaling

---

reduce hubs in space. J. Mach. Learn. Res., 13:2871- 2902.
rideo-text retrieval with explicit high-level semanties. In MM '22: The 30th ACM International Conference Multimedia, Lishoa, Portugal, October 10- 14, 2022. pages 4887-4898. ACM.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba. Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: A foun- dational language and vision alignment model. In IEEE/CVF Conference on Computer Vision and Put- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15617-15629, IEEE.
Xiaohan Wang, Linchao Zhu, Zhedong Zheng, Min- gliang Xu, and Yi Yang. 2022b. Align and telli Boosting vixeo retrieval with local alignment and fine-grained supervision. IEEF Transactions on Multimedia, pages 1-11.
Samuel L.. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offlin bilingual word vectors, orthogonal transformations and the inverted softmax. In Sth international conference on learning representations, ICLR 2017, toulon, france. april 24-26, 2017, conference track proceedings. OpenReview.net. Tex.bibsource: dblp computer science bibliography. https://bllp.og tex. biburl: https://dblp.orgg/eee hwt.:///wc/cooo//c///iiiiitiiiii tex. tex. Thu, 25 Jul 2019 14:25:47 +0200.
Yimu Wang. Shiyin Lu, and Lijun Zhang. 2020a. Searching privately by imperceptible lying: A novel private hashing method with differential privacy. In Proceedings of the 28th ACM International Confer- ence on Multimedia, page 1700-2709
Yimu Wang and Peng Shi. 2023, Video-tes text retrieval by supervised multi-space multi-grained alignment.
Yimu Wang, Xiu-Shen Wei, Bo Xue, and Lijun Zhang. 2020b. Piecewise hashing: A deep hashing method for large-scale fine-grained search. In Puttern Recog- nition and Computer Vision Third Chinese Confer- ence, PRCV 2020, Nanjing, China, October 16-18, 2020, Proceedings, Part II. pages 432 444.
Shupeng Su, Zhisheng Zhong, and Chao Zhang. 2019. Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval. In 2019 IEEE/CVF International Conference on Com- puter Vision, ICCV 2019, Seoul, Korea (South), Octo- ber 27- - November 2, 2019, pages 3027-3035. IEEE,
Yimu Wang. Bo Xue, Quan Cheng, Yuhui Chen, and Lijun Zhang. 2021. Deep unified cross-modality hashing by pairwise data alignment. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1129-1135.
Ikumi Suzuki, Kazuo Hara, Masashi Shimbo. Yuji Mat- sumoto, and Marco Saerens. 2012. Investigating the effectiveness of laplacian- based kernels in hub reduc- tion. In Proceedings of the Twenty-Sixth AAAI Con- ference on Artificial Intelligence, July 22-26, 2012, Toronio, Ontario, Canada. AAAI Press.
Xing Wu, Chaochen Gao, Zijia Lin. Zhongyuan Wang, Jizhong Han, and Songlin Hu. 2022. RaP: Redundancy video-language pre-training for text-video retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics
Ikumi Suzuki, Kazuo Hara, Masashi Shimbo, Marco Saerens, and Kenji Pukumizu. 2013. Centering sim- ilarity measures to reduce hubs. In Proceedings of the 2013 Conference on Empirical Methods in Nat- ural Language Processing, pages 133-22,,65 Seattle, Washington, USA. Association for Computational Linguistics.
Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettle- moyer, and Christoph Feichtenhofer. 2021. Video- CLIP: Contrastive pre- tre-training for zero-shot video- text understanding. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 6787-6800. Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Yao-Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang, J. Zico Kolter, Couis-Phllippe Morency, and Rus- lan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6558- 6569, Florence, Italy. Association for Computational Linguistics.
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. 2015. Translating Videos to Natural Lan- guage Using Deep Recurrent Neural Networks. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1494-1540 1504, Denver, Colorado. Association for Computational Linguistics.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR- VTT: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5288-5296 IEEE Computer Society.
Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. 2023. Multimodal federated learning via con- trastive representation ensemble. In The Eleventh International Conference on Learning Representa- lions.
Haoran Wang, Di Xu, Dongliang He, Fu Li, Zhong Ji, Jungong Han, and Errii Ding. 2022a. Boosting

---

Tiezheng Yu, Wenliang Dai, Zihan Liu, and Pascale Fung. 2021. Vision guided generative pre-trained language models for multimodal abstractive summa- rization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3995-4007, Online and Punta Cana, Domini- Republic. Association for Computational Lin- guistics.
Lihi Zelnik- Manni and Pietro Perona. 2004. Self-tuning spectral clustering. In Advances in Neural Informa- tion Processing Systems [Neural Information Pro- cessing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada], pages 1601- 1608.
Bowen Zhang, Hexiang Hu, and Fei Sha. 2018. Cross- Modal and Hierarchical Modeling of Video and Text. In Computer Vision ECCV 2018, volume 1 11217. pages 385-401, Cham. Springer International Pub- lishing. Series Title: Lecture Notes in Computer Science.
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. 2022. Centerclip: Token clustering for efficient text- video retrieval. In Proceedings of the 45th Inter- national ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '22, page 970-981, New York, NY, USA. Association for Computing Machinery.
Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi
Li. 2017. Re-ranking person re-identification with k-reciprocal encoding. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolula, HI, USA, July 21-26, 2017, pages 3652-3661. IEEE Computer Society.

---

A Related work
Instead of improving cross-modal retrieval through better representation, retrieval techniques, or more data, we aim to advance cross-modal re- trieval by addressing the hubness problem in post- processing manner. This problem has been empiri- cally demonstrated to be prevalent among various cross-modal retrieval tasks and datasets, as shown in Figure 3 and Figure 6 in the Appendix.
In this section, we present prior work from the literature that lies in cross-modal retrieval and the hubness problem, which are the two most related areas to our work.
Cross-modal Retrival. Cross-moda retrieval methods aim learn common representation space, where the similarity between samples from different modalities can be directly mea- sured. Early methods for cross-modal retrieval include Gaussian Mixture Models (Owens et al., 2016) modelling translation via EM (Croitoru et al., 2021), CCA (Mithun et al., 2018), and KCCA (Patrick et al., 2021).
The Hubness problem. This problem was ini- tially characterized by Radovanovic al. (2010). They noticed that the distribution becomes consid- erably skewed as dimensionality increases under commonly used assumptions, leading to the emer- gence of hubs. Hubs are points with very high k-occurrences that effectively represent "popular" nearest neighbors. In other words, the distribu- tion of "k-oceurrences" (the number of times a point appears in the nearest neighbors of other points) skews heavily to the right. As an inherent property of data distributions in high-dimensional vector space, it might cause the degradation of the retrieval (Bogolin et al., 2022) and bilingual word translation performance (Huang et al., 2020, 2019; Smith al., 2017).
Inspired by the tremendous success of deep learning (Devlin al., 2019; He et al., 2016), nu- merous methods have been proposed for image- text retrieval (Radford et al., 2021), video-text retrieval (Luo et al., 2022a), and audio-text re- trieval (Oncescu et al., 2021). With the rise of self-supervised pretraining methods (Brown et al., 2020; Devlin et al., 2019; Radford et al., 2019), vision-language pretraining (Gan et al., 2020; Li al., 2020b; Singh et al., 2022) on large-scale unlabeled cross-moda data has shown promising performance in various tasks, such as image re- trieval (Liu et al., 2021a), image captioning (Jiang al., 2019), and video retrieval (Park et al., 2022b). Inspired by this, recent works have attempted to pretrain fine-tune cross-modal retrieval mod- cls, e.g., image-text retrieval (Radford et al., 2021; Li et al., 2020b), video-text retrieval (Chen al., 2020a; Cheng et al., 2021; Gao et al., 2021; Gorti et al., 2022; Lei et al., 2021; Ma et al., 2022; Park et al., 2022a; Wang et al., 2022a,b; Zhao et al., 2022), and audio-text retrieval (Koepke et al., 2022) in an end-to-end manner.
In the past few decades, algorithms for mitigat- ing the hubness problem can be categorized into three classes. The first line of research focuses on rescaling the similarity space to symmetrize nearest neighbor relations (Schnitzer et al., 2012), includ- ing local (Jégou et al., 2007: Zelnik Mennir and Perona, 2004) and global (Schnitzer et al., 2012) scaling. Another line of research has focused on mitigating the hub tendency of centroids of the data through Laplacian- based kernels (Suzuki et al., 2012) and centering (Suzuki et al., 2013; Hara et al., 2015). Later, while CENT (Hara et al., 2015) has better complexity, Bogolin et al. (2022) show that CENT is not effective in terms of retrieval perfor- mance.
Another line of research has been focused on improving the effectiveness of retrieval, including k-d trees (Bellman, 2015), re-ranking (Zhong et al., 2017: Miech et al., 2021), query expansion (Chen et al., 2020a), vector compression schemes based on binary codes (Su et al., 2019; Liong et al., 2017) and quantization (Gong et al., 2013) that help address the curse of dimensionality (Keogh and Mueen, 2017). In addition, extracting addi- tional information from noisy data (Hu et al., 2021). incorporating more datasets (Yu et al., 2023), and mitigating the domain adaptation problem (Kim et al., 2021a) have been other solutions for cross- modal retrieval.
The last line, which is also closest to our pa- per, is the normalization of similarity by matching queries with a set of data points (Bogolin et al., 2022; Dinu et al., 2015; Lample et al., 2018; Smith et al., 2017). Dinu et al. (2015) first proposed a globally-correcter (GC) method to take into ac- count the proximity distribution of potential neigh- bors across many mapped vectors to tackle the hubness problem. Following that, Lample et al. (2018) proposed cross-domain similarity local scal- ing (CSLS) using a bipartite neighborhood graph. Next, inverted softmax (IS) (Smith et al., 2017)

---

and QBNorm (DIS) (Bogolin et al., 2022) were proposed to normalize query similarities by consid- ering the similarity between a query bank and the gallery points.
10k videos, each with 20 captions. For text- video retrieval, following prior works (Liu al., 2019; Croitoru et al., 2021: Luo et al., 2022a; Ma et al., 2022; Park et al., 2022a), we use the official split (full) and the Ik-A split. The full split contains 2,990 videos for testing and 497 for validation while the 1k-A split contains 1,000 videos for testing and around 9,000 for training.
In contrast to IS and QBNorm, which are the two most related works to ours, our DBNORM is built upon detailed theoretical analysis (Section 2.2) and normalizes similarity considering points from both modalities, motivated by our theoretical analysis. By constructing banks from two modali- ties, DBNORM achieves better performance with- out access to any additional data on several dif- ferent benchmarks across various cross-modal re- trieval tasks.
MSVD (Chen and Dolan, 2011) has 1,970 videos and around 80k captions. The results are reported on the standard split used in prior works (Liu et al., 2019; Croitoru et al., 2021; Luo et al., 2022a; Park et al., 2022a) which consists of 1,200 videos for training, 100 for validation and 670 for testing.
Hubness and similarity concentration. It is known that the problem of hubness is re- lated to concentration, the tendency of pairwise similarities between elements in set to con- verge to constant as the dimensionality of the space increases (Radovanovic et al., 2010). Later, Radovanović et al. (2010) show that it also holds for the widely employed cosine similarity as the ex- pectation of pairwise similarities becomes constant and the standard deviation converges to 0.
ActivityNet (Fabian Caba Heilbron and Niebles, 2015) contains 20k videos and has around 100K descriptive sentences. The videos are extracted from YouTube. We use paragraph video retrieval defined in prior works (Liu et al., 2019; Croitoru al., 2021; Luo al., 2022a; Park et al., 2022a). We re- port results on the vall split while the training split consists of 10,009 videos, while there are 4,917 videos for testing.
B Experiments
To demonstrate the empirical efficiency of our DB- NORM, we test alongside DualIS and DualDIS on five video-text retrieval methods: CE+ (Liu et al., 2019), TT-CE+ (Croitoru et al., 2021), CLIP4Clip (Luo et al., 2022a), X-CLIP (Ma et al.. 2022), and CLiP2Video (Park et al., 2022a), two image-text retrieval methods: CLIP (Radford et al., 2021) and Oscar (Li et al., 2020b), and one audio- text retrieval method: AR-CE (Koepke et al.,
DiDemo (Hendricks et al., 2017) includes over 10,000 25-30 second long per- sonal videos with over 40,000 localized text descrip- tions. Videos are split into training (8,395), validation (1,065), and testing (1,004) sets.
MSCOCO (Lin et al., 2014) consists of 123k images with 5 captions for each sentence. We the 5k split for evaluation.
2022).
B.1 Dataset details
Flickr30k (Plummer et al., 2017) dataset con- tains 31,000 images collected from Flickr, to- gether with reference sentences provided by human annotators.
Experiments are conducted on eight cross- modal benchmarks, including four video-text retrieval benchmarks (MSR-VTT (Xu et al., 2016), MSVD (Chen and Dolan, 2011), Activi- tyNet (Fabian Caba Heilbron and Niebles, 2015), and DiDemo (Hendricks al., 2017)), image- text retrieval benchmarks (MSCOCO (Lin al., 2014) and Flickr30k (Plummer et al., 2017)), and two audio-text retrieval benchmarks (Audio- Caps (Kim et al., 2019) and CLOTHO (Drossos et al.. 2020)). The details of the datasets are shown below,
AudioCaps (Kim et al., 2019) dataset which comprises sounds with event descriptions. We use the same setup as prior work (Koepke et al., 2022) where 49,291 samples are used for training, 428 for validation, and 816 for testing.
CLOTHO (Drossos et al., 2020) consists of 4,981 audio samples of 15 to 30 seconds in duration and 24,905 captions of eight to 20 words in length (five captions for each).
MMS-VTT (Xu et al., 2016) contains around

---

Text-to- Video Retrieval
Videeo
Video-to-Tew Retrieval
Methods
Nornalization
R@
R@51
R@10
MdR
MaR +
R@ 1
RØ5 R@101
MdR
MnR 4
MSR-VTT (full split)
RoME
10.7
29.6
41.2
17.0
Frozen
32.5
61.5
71.2
12.62
46.38
19.83
60.94
29.42
13.31
35.00
47.46
12.0
74.16
23.68
52.25
65.59
24.75
CF+
DIS
13.31
35.00
47.46
12.0
74.16
23.68
52.25
65.59
5.0
24.75
DualIS
14.88
37.36
50.00
11.0
70.13
25.48
56.68
69.89
4.0
21.76
DuaIDIS
14.88
37.36
50.00
11.0
70.13
25.48
56,68
69.89
4.0
21.76
14.61
37.81
50.78
10.0
63.31
24.48
54.11
67.59
5.0
20.22
16.58
40.75
29.40
72.11
16.56
TT-CE+
16.59
40.73
53.44
60.51
26.69
69.06
18.90
DualIS
17.06
41.63
54.25
8.0
60.10
29.83
62.04
73.68
3.0
15.70
DualIDIS
17.02
41.6 1660
54.23
8.0
60.01
27.53
57.39
69.77
4.0
18.63
MSR-VTT (Ik split)
DiscreteCodehook
43.4
72.3
81.2
14.8
42.5
71.2
81.1
12.0
VCM
43.8
71.0
2.0
14.3
45.1
72.3
82.3
2.0
10.7
CenterCLIP
44.2
71.6
82.1
2.0
15.1
42.8
71.7
82.2
2.0
10.9
Tel
45.2
73.0
82.9
2.0
43.4
70.9
81.8
2.0
CLIPZTV
45.6
71.1
80.8
2.0
15.00
43.9
70.9
82.2
2.0
12.0
X-Pool
46.9
72.8
82.2
2.0
14.3
TS2-Net
47.0
74.5
83.8
2.0
13.0
45.3
74.1
83.7
2.0
9.2
CAMoE
47.3
74.2
84.5
2.0
11.9
49.1
74.3
84.3
2.0
9.9
44.10
71.70
81.40
15.51
42.09
71.24
81.23
12.01
44.20
71,70
$1.60
15.64
44.86
72.04
82.02
11.56
CLIP4Clip
44,20
71.70
81.60
15.64
44.86
72.04
82.11
11.61
DualIS
45,00
72.50
82.10
2.0
15.32
45.45
73.02
81,42
2.0
11.56
DuaIDIS
45.00
72.50
82.10
2.0
15.32
45.45
73.02
81.42
2.0
11.59
46.00
71.60
81.60
14.51
43.87
72.73
82.51
10.20
+S
47.00
72.80
82.10
13.91
46.15
72.63
81.92
11.15
CLIP2Video
47.00
72.80
$2.10
13.91
46.15
72.53
81.92
11.14
DualIS
47.20
73.20
82,30
2.0
13.90
46.74
72.53
81.82
2.0
10.93
DuaIDIS
47.20
72.70
82.30
2.0
13.90
46.74
72.43
81.82
2.0
10.90
46.30
74.00
83.40
12.80
44.81
73.69
82.39
10.99
74.10
84.10
13.35
46.69
83.28
10.52
X-CLIP
48.60
74.10
84.10
13.35
46.69
83.28
10.52
DualIS
48.50
74.30
84.10
2.0
13.30
46.88
74.38
83,38
2.0
10.44
DualDIS
48.70
74.30
84.10
2.0
13.30
46.88
74.28
83,38
2.0
10.44
Table 7: Retrieval performance on ASR-VTTTO (full split and 1k split). Best in Bold and the second best is underlined.
B.2 Experimental Details
the models single A A100 card using the hyper- parameters recommended in their papers. We set constructing activation sets.
The experiments are conducted using the PyTorch framework (Paszke et al., 2019). We utilize the fol- lowing models and their respective weights: CE+ (Model Weights: MSR-VTT, MSVD, DiDeMo, and ActivityNet); TT-CE+ (Model Weights: MSR-VTT, MSVD, DiDeMo, and ActivityNet); CLIP2Video (Model Weights: MSR-VTT and MSVD); CLIP (Model Weights: MSCOCO and Flickr30k); Oscar (Model Weights: MSCOCO and Flickr30k); AR-CE (Model Weights: AudioCaps and CLOTHO) as they have released official code and weights. However, since CLIP4Clip and X- CLIP do not provide their trained models, we train
B.3 More Quantitative Results
Text-video retrieval. The vidco-to-text results on MSR-VTT, ActivityNet, and MSVD are presented in Tables 7 to 9, the results on DiDeMo are in Table 12. Similar results can be observed. Our proposed DualIS and DualDIS outperform the pre- vious methods, i.e., IS and DIS, by large margin across different benchmarks.
Text-image retrieval. The image-to-text results on MSCOCO and Flickr30k are shown in Table 10. Text-audio retrieval. The audio-to-text results

---

Viden-to- Text Retrival
Image-too Text Reurivall
Methods
Normalization
Mechods Normalizaiio
Rd
R@s
R
MJR
MrR
R@I
R@5
R@10 f MdR
MaR
FSE
12.6
33/2
77.6
12.0
MSOOCO
VCM
42.6
74.9
86.2
ConterCT IP
44.5
75.7
86.2
2.0
6.5
VIET
56.5
82.6
89.6
Allg
43.5
73.6
98.3
ALIGN
58.6
89.7
CAMoE
49.9
77,4
CODIS
71.5
91.1
95.5
ALBEF
77.6
94.3
97.2
18:51
47.85
63
IS
19.36
50.13
65.5 65.87
5.0
20.81
50.02
74.82
83.34
1,0
9.23
CE+
-IS
19.52
65.81
IS
50.68
74,80
83.10
1.0
9.39
CLIP
DDS
50.68
74.80
83.10
DuallS
21.92
53.49
68.25
5.0
17.57
DualDDS
21.72
DuallS
53.26
77.00
85.25
1.0
8.44
DualDDS
53.32
76.95
85.30
1.0
8.32
22.49
56.38
72.67
4.0
13.50
IS
23.18
57.01
66.74
89.98
94.98
2.98
TT:CE+
DIS
23.43
57.05
73.48
4.0
13:48
IS
68.36
90.16
32
1.0
2.91
27.46
61.13
76.71
Oscar
DDS
69.48
90.54
2.99
DualDDS
22.11
50.79
75.86
4,0
11:42
DuallS
70:72
91
95 66
1.0
2.81
41.62
74.11
86.
DualDDS
70,93
91.44
95.84
1.0
2.76
IS
46.23
76.72
87 26
2.0
6.46
Flickr30k
CLIP4Clip
76.48
VILT
73.2
93.6
96.5
DuallS
46 59
78.04
88.15
2.0
6.05
UNITER
83.6
95.7
97.7
DuallS
46.73
77.90
ALIGN
88.6
98.7
99.7
45.20
76.07
86.57
2.0
6.40
CODIS
91.7
99.3
99.8
+15
$1.13
78.95
88. 14
1.0
3.62
ALBEF
95.9
99.8
100.0
X-CLIP
DIS
50.52
78.60
87.84
1,0
5.71
78.10
94.90
96_10
1,0
1.98
DuallS
50.92
79.42
89.36
1.0
5.32
IS
77.80
92.90
97.40
1.0
2.15
DualDDS
50.22
79.12
$8.62
1,0
3.44
CLIP
DIS
77.90
92.90
97.40
2.14
DuallS
$1.60
95.40
97:90
1.0
1.92
Table 8: Retrieval performance on AetivityNet. Best in Bold and the second best is underlined.
DualDDS
81.50
95.40
97.90
1.0
1.92
86.30
96
1.58
-S
87.10
97.70
99. 10
1.0
1.49
DDS
87.10
99.
1.49
DuallS
87.10
97.
99.30
1.0
1.49
DualDDS
87.10
97.60
99.30
1.0
1.49
Video-to-Text Retrivval
Methods
Normalization
R@L1
R@51
MJR
MrR
Table 10: Image-to-Tex: Retrieval performance on MSCOCO (5k split) and Flickr30k. Best in Bold and the second best is underlined. Due to the limitation of the computational resources, only use 20% of data
FSB
16.7
43.1
88.4
7.0
CeIterCLIP
63.5
86.4
92.6
1.0
3.8
Align&/Tell
61.8
87.5
92.7
1.0
22.84
49.85
61.49
6.0
33.96
IS
26.22
54.48
65.92
4.0
27.89
from the training data to construct the dual banks.
CE+
DIS
21.63
52.99
64.63
5.0
32:57
+ DuillS
29.85
$9.40
67.46
4.0
25.41
DunIDIS
25.67
53.58
65.07
5.0
32.26
Audio-to-Too Retrieval
Medhods
Normalication
25.22
55.07
64.63
4.0
29.94
R
R@5
R 10
MJR
MaR
IS
24.63
52.39
65.07
4.0
27.31
AudioCaps
TT.CE+
+ DIS
23.43
55.22
65.37
4.0
28:25
MoEE>
26.60
59.30
73.50
4.0
15.60
DunlIS
28.06
$7.46
67.61
4.0
25.64
MMT
39.60
76.80
86.70
6.50
DuiDDS
24.18
55.97
65.82
4.0
26.16
24.02
56.00
71.81
4.0
16.91
63.13
79.40
85.37
1.0
11.02
I5
67.61
$4.48
89.70
1.0
7.61
IS
29.90
61.64
74.75
13.95
CLIPACip
DIS
61.48
$1.49
87.16
L.0
11.10
AR-CE
DIS
29.90
61.27
74.75
4.0
15.01
+ DuilIS
67.76
$1.18
89.25
1.0
8:07
+ DuallS
30.02
62.13
75.12
4.0
14,80
DunIDS
64,78
81.64
87.16
1.0
11.12
DualDDS
61.76
75.12
14.85
62.09
83.13
89.40
1.0
7.73
CLOTHO
+S
66.52
82.84
88.36
1.0
7.94
MobI
7.20
22.10
33.20
22,7
71.80
CLiP
DIS
62.39
82.8 84
88.51
1.0
MMT
22.80
33.30
+ DialIS
69.70
86.27
89,70
1.0
6.35
7.27
23.35
35.12
21.0
74.68
DuilDD
63.13
83.58
89.40
1.0
+IS
8:52
23.64
$2.20
20.0
72.99
65.67
83.73
89.85
1.0
8.15
AR-CE
DIS
8.33
23.35
36,84
20.0
73.35
+5
64.63
81.64
87.16
1.0
9.84
X-CLIP
* DI8
64.78
82.84
88.51
1.0
8.46
+ DuallS
9,09
25.55
38,28
19.0
71,33
+ DuulDDS
8.13
23.64
37.04
20.0
72.71
DuiltS
67.91
83.88
88.96
1.0
DaalIDSS
66.27
83.33
88.81
1.0
8.27
Table II: Audio-to-Tew Retrieval performance Au- dioCaps and CLOTHO. Best in Bold and the second best is underlined.
Table 9: Retrieval performance on MSVD. Best in Bold and the second best underlined.
B.4 More Qualitative Results
on AudioCaps and CLOTHO are presented in Ta- ble 11.
This section presents retrieval results for text-to- video and video-to-text retrieval tasks, as illustrated

---

Text-lo- Video Retrieval
Video-to- Text Retrieval
Methods
Normalization
R@1 1
R@5
R@
MdR J MrR 1
R@L
R@5 f
R@ 10 1
MdR
MnR J
S2VT
11.9
13.0
13.2
33.6
76.5
15.0
FSE
13,9
36.0
78.9
11.0
13.1
33,9
78.0
12.0
18.22
42.63
8.0
42.05
18.82
42.73
55,78
37.27
IS
21.22
46.31
$9.56
7.0
37.06
20.72
45.72
59.86
7.0
33.84
C
DIS
21.02
46.31
59.56
7.0
37.37
20.72
45.72
59.76
7.0
33.87
DualIS
21.81
46.12
59.66
7.0
37.07
20.92
45.62
59.66
7.0
33.84
DualDIS
21.71
46.31
59.56
7.0
37.37
20.72
45.82
59.56
7.0
33.90
22.31
49.20
62.15
6.0
31.18
21.41
47.11
60.66
6.0
29.67
+S
23.80
51.29
65.04
5,0
28.21
24.20
51.59
65,54
5.0
26.09
TT-CF+
DIS
23.71
51.29
64.84
5.0
28.40
24.30
51.49
65.54
5.0
26.23
DuallS
25.60
50.30
64,44
5,0
28, 49
24.40
52.59
66.33
5.0
25.01
DualDIS
25.60
50.40
64.44
5.0
28.88
24.60
52.49
66.24
5.0
25.08
Table Retrieval performance on DiDeMo. in Bold and second best is underlined. S2VT (Venugopalan et al., 2015) and FSE (Zhang et al., 2018) are two representative methods on DiDeMo.
Figures and 5, respectively. Upon observing both text-to-video and video-to-text retrieval sce- narios, a notable observation emerges regarding the susceptibility of IS and DIS to be misled by the query bank as shown in Figure 5. In contrast, our proposed methods, DualIS and DualDIS, capital- ize on the utilization of both query and gallery banks, leading to an improved retrieval perfor- mance achieved by effectively reducing the sim- ilarity of hubs and preserving the similarity of non- hubs.
EiENN (x,((x)= = EiE(N, I(x the nearest neighbours of the i-th gallery data) and I(cond) is the indicator function. Positive skewness indicates that the distribution is right-tailed and higher skewness scores mean a severer hubness problem occurs. As shown in Tables 6 and 13, the hubness score consistently exhibits a relatively high value across various methods and benchmarks. However, upon employing our DualIS and DualDIS, we observed a notable reduction in hubness scores across different benchmarks. This reduction serves as empirical evidence supporting the effectiveness and efficiency of our proposed methods in addressing the hubness issue.
B.5 DBNORM
Continued RQ1: Can DBNORM alleviate hub- ness? (Empirical Observation of Hubness in Cross-Modal Retrieval) We first illustrate the hub- ness phenomenon across several benchmarks and methods of cross-modal retrieval in Figure and Figure We notice that, hubness prevalent across different methods, datasets, and tasks as the queries frequently retrieve small number of gallery data.
Continued RQ2: How much data desired in the banks? The results R @ for text-video re- trieval using DualTS and DualDIS are presented Figures 4 and 7. Our observations indicate that as the size of the two banks increases, the performance improves. owever, even with relatively small number of samples in the query and gallery banks, still achieve satisfactory performance. More- over, we examined the individual impact of the query and gallery bank sizes by independently sam- pling them at different scales. The results demon- strate that the size of the query bank has a greater influence on performance compared to the gallery bank, although a bigger gallery bank also leads to better performance.
Moreover, to demonstrate how severe the hub- ness problem is in specific benchmark and method, following Radovanovic al. (2010), we employ skewness of the k-oceurrences distribution, with &o = 10. Specifically, the skewness of the k-occurrences distribution is defined as
SNZ [x[NN((x u
(3)
RQ3: Is DBNORM sensitive to B1 and 32? In order investigate the sensitivity of DBNORM 31 and 3z, we conducted an evaluation as shown in Figure 8. Our findings indicate that the perfor- mance of DualDIS demonstrates limited sensitivity
where MN2 and ON. are the mean and standard deviation of N Nx(x) refers
to the k-occurrence distribution, given by

---

(Text) Query: computer generated cartoon figure
operates control panel while amother character sleeps in the background
Retrieval Results by CLIP4Clip
(Text) Query: man giving interview in show
Retrieval Results CLIP4Clip
Retrieval Results by CLIP4Clip IS/DIS
Retrieval Results CLIP-Clip IS/DIS
Retrieval Results by CLIP4Clip DuualS/Ds
Retrieval Results
CT CI.IP4Clip
DualiS/DuiDDS
(a) Top-3 toxt-oo-video.
(b) Top-3 Text-to-vided
Query
CLIP4Clip
Query
CLIP4Clip
young girl shopping
playing with toy kitchen accessorics
child in pink watches white bird in an open box
a woman making playdoh
a gid walking down path
someune is assembling small hello kitty oven toy
CLIP4Clip IS/DIS
CLIP44Tip DualIS/DuaDDIS
CLIP4Clip IS/DIS
CLIP4Clip DualDSSDuaDDIS
child pink watches white bird in open box
a young girl shopping
is assembling
playing with toy kitchen accessories
small hello killy loy
child pink watches
young girl shopping
playing with toy kitchem accessorics
someone is ussembbling small hello kitty oven toy
white in an open girl talks ahout photos and her life
girl talks ahout photos and her life
woman making playdoh
woman making playdoh
Top-3 Video-to-text.
Top-3 Video-to-text.
Figure 5: Top-3 Text-to-video and Video-to-text retrieval results on MSR-VTT.
to changes in both 31 and 82. Adjusting these hy- perparameters within reasonable range does not result in significant variations in the overall perfor- mance.
RQS: Is hubness score related to perfor- mance? Our current findings reveal inverse relationship between the hubness score and per- formance. Specifically, we observed that as the hubness score decreases, the performance tends to improve. This aligns with the expectation that lower hubness scores indicate a reduced concen- tration of nearest neighbors, leading to improved retrieval performance. However, it is important to note that in certain cases (X-CLIP on MSVD), the relationship between the hubness score and retrieval performance may differ from this trend.
RQ4: Is DuaIDIS sensitive to the Top-k hyper- parameter? We conducted experiments to observe the influence of in the Top-k selection used to construct the gallery activation sets. The results are shown in Figure 9, We observed that choosing k offers good trade-off between performance and computational efficiency. Therefore, we used k = 1 for all reported experiments.

---

MSVD (CE+)
MSVD (TT-CE+)
MSVO (CLIP4Ciİa)
MSVD DX-CLIP)
Loccurence
mise 265.3
1-occurncce
mox - 26532
1-occurence
300
mix 3A7 man
1-occurnce
mas 26485 me
mi
mine
median 25.0
200-
median
median
10000
10000
100
10000
200
400
600
200
400
600
100
200
300
200
400
600
number of videos
number of videos
number of video
number video
(a) MSVD (CE+).
(b) MSVD (TT-CE+))
(c) MSVD (CLIP4Clip).
(d) MSVD CL-PL..)
1-0ccuencce 16
MSR-VIT (CUP2Video)
(CUP2Videe)
DiDeMo (CE+)
DiDeMo (TTCE++)
1-ocurence
200
545
Trocurencs
30.0
1-ccuunnce
mi modao 35.0
mih
min medion 1.0
median
200-
modian
5.0-
100
2.5
1000
200
300
400
50 100 150 200 250 300
50
100
350
200
250
50
100
150
200 250
number of video
number of video
number of videos
number of videos
(c) MSR-MSRTTTT (CLIP2Video).
(f) MSVD (CLIP2Vidco).
DiDeMo (CF+).
(h) DiDeMo (TT-CE++)
CLI (MSCOCO)
OSCAR (MSCOCO)
CLIP (Flickr30k)
OSCAR (Flickr30k)
1-occurence
150
30
1-occurence
1-occuancc
200
median 4.0
mio
20
median 5.0
median - 5.0
min
mith modian - 5.0
50-
1.0
10
1000
2001
1000
5:0
1000
1500
2000o
0
100
250
3á0 400
100
150 200
bember
Images
number of images
number images
number of
(i) MSCOCO (CLIP).
i MSCOCO (Oscar).
(k) Flickr30k (CLIP).
(I Flickr30k (Oscar).
AudicCaps [AR CE)
CLOTHO (AR-CE)
1.ccuncnce
30-
max
80-
maz 99 m
mesion - 4,0
00-
medkan 6,0
20
100
audios
number audi
(m) AudioCaps (AR-CE).
(n) CLOTHO (AR-CE).
Figure 6: Hubness is prevalent across different methods, datasets, and tasks. These figures illustrate the distribution of the number of times each (test) gallery video/image/audio retrieved by (test) set queries. In all visualization, severe hubness can he observed, as a small number of galleries are retrieved disproportionately often, which might reduce the retrieval performance.
Video-haat Rg1
MER.VTT (CLIP401P))
Viteo Toxt R@a
ActivityNet (CLI44CI())
Vi Tedeo-Toox Reg
MSR.VTT (CLP44CP))
Wottoodnoo RS3
ActMittyNet (CLIP4CIlp)
16
65.0
1.0
Duals DusiDIS
Dualis
30
Duals
baseline
04
Duai5
Baseinc
DusiDIS
20
Dualais
DusiDs
16
10"
"10"
10
Size (N Na)
Banks and Na)
Size Gallery Bank No
Size Gallery Bank Na
Video-Tbet R@l
ActivityNatt (CLIP4CIpp)
Mdde Toxk Rgu
MER.VIT (CUP4CIlA)
coctedde Re1
ActivityNat (CL/P4CIl))
Mid00- Toxt R81
ActivityNott (CLIP4CIip)
96
85
65.0
baseliee
8.3
84
Dusl5
Dualis DusIDIS
Duals DualDis
42.5
Duall
62-
40.0
60-
00.0
DuaiDIS
10" 10F
10"
111"
'107
Sizc Gallery Bank Na
Query Bank
Query Bank Ne
Query No
Figure 7: Text- Video Retrieval R@ w.n. the size of gallery and query banks on MSR- VTT and ActivityNet using Dualls and DuaIDIS with CLIP4Clip.
This can be attributed to potential biases present in the dataset and the models, which can influence the observed patterns. Therefore, we should carefully interpret the hubness score as the sole indicator
of retrieval performance, particularly when dataset biases are significant. Further investigations are needed to gain a comprehensive understanding of the interplay between the hubness score and re-

---

MSRVTT CLIP4Clip CLio X-CLIP CE
ActivityNet
MSCOCO
Normalization
CE+
TECE++
TT-CE+
CLIP4Clip
X-CLIP
CLIP
Oscar
Texi-to Video/lmmgm///uuii
1.38
1.28
1.13
0.84
1.24
0.94
0.76
0.83
0.98
2.71
0.55
0
0.82
.DIS
0.83
0.34
0.18
0.33
0.74
0.68
0.42
0.44
0.90
0.22
DuallS
0.37
0.33
0.57
0.26
0.70
0.54
0.37
0.36
0.42
0,42
0.31
7
DualDDS
0.37
0.28
0.57
0.26
0.70
0.50
0.40
0.46
0.43
0.43
0.29
5
VideoInags/Audio-too -oxxt
3.65
4.02
2.13
2.13
2.66
1.06
0.70
1.11
1,58
3.78
1.45
0
+ 11S
2.29
2,34
0.43
0.28
1.58
0.63
0.56
0.77
0.44
2.21
1.08
DDIS
2.50
0.65
DuallS
2.48
2.38
1.12
0.28
1.46
0.30
0.27
0.32
0.37
1.53
0.98
DualDDS
2,51
2.47
1.16
0.31
1488
0.34
0.29
0.40
0.37
1,50
1.06
2
DiDeMo
Flickr30K
AudioCaps AR-CE
CLOTHO AR-CE
Normalization
Best
CE+
TE-EE+
CLIP4Clip CLIP2Video X-CLIP CE+
TT-CB+
CLIP
Oscar
Text-to- Text-to-Vidas Vidensmmage/Auuio
7.95
7.95
0.83
0.84
0.65
1.23
0.86
2.78
0.32
0.22
1.09
+ 18
7.88
7.92
0.42
0.80
1.87
0.44
0.39
4.06
0.01
0.02
0.68
4
+ IDI
7.92
7.92
0.42
0.80
1.87
0.46
0.40
2.81
0.01
0.01
0.68
+ DuallS
7.93
7.92
0.20
0.68
0.53
4.14
0.15
0.46
DuaaDDSS
7.93
7.92
0.37
0.21
0.68
0.32
2.81
0.01
0.15
0.55
Vadeo/lnag//Audiooooi
3.18
4.18
4,54
6.02
4.47
1.01
2.23
1,16
1.65
2.43
+ -S
3,45
3.06
2.97
2.78
3.84
0.53
0.72
1.62
1.02
1.00
2.03
4
+ DDS
3.42
4.05
3.47
4.39
4.64
0.52
0.72
1.62
1.03
0.99
2.02
2
+ DuallS
2,60
3.42
3.01
3.00
3,92
0.54
0.59
1.08
0.99
1.06
1.53
DualDDS
1.60
3
Table 13: The hubness (skewness score) significantly reduced after applying our proposed DualIS and DualDIS than IS and DIS Best in Bold and the second best is underlined.
MSR-VTT (CUP4CIp)
Activity (CUP4ClPp)
MSR.VTT (CUP4CI))
ActivityNet (CUP4CIp)
Video R08
Video-Texxt Roy
Text-Vide R081
45
60
baseline
Dunlt
Toxt-Wideo
86
baseline Dusl5
Duals
baseline
35
40
baselie
Dunls
DuiiD15
Dualbi5
30
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
81
9:
8:
(a) V2T R@I w.l. 1.
(b) V2T R@ w.r. 81.
(c) V2T R@1 w.r. 82.
(d) V2T R w.r.t 82.
Figure 8: Video-to-Text Retrieval recall w.rt 31 and 32 on CLIP4Clip and ActivityNet using DualIS and DuaIDIS.
(CUP4Ci))
Activity/vet
ActivityNet (CUP<Cip)
Test Mideo R881
Viden-Text: Rel
Text Vitt-ideooo Rea1
40.8
45
48
46-
44.6
48
44
40.2
DuaiDI5
baselre
43
boseline DualD15
baseline DualDi5
DiaDD5
beseine
0
10
20
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Top-k
Top-k
Top-k
Tap-
(a) T2V
w.n. Top-k on(b) T2V R@1 w.n. Top-k on(e) T2V R@ w.nt Top-k on Ac(d) V2T R w.n. Top-k on Ac- tivityNet.
MSR-VI VTT.
MSR-VTT
tivityNet.
Figure 9; Video-Text Retrieval performance with respect to top-k hyperparameter in DuaIDIS.
trieval performance under different conditions and datasets.
allS (Equation (1)) and DualDIS (Equation (2)). multiplication is employed to aggregate normal- ization results from the query and gallery banks. To explore altemative aggregation methods, we
RQ6: How about aggreagating normalization results by adding instead of multiplying? Du-

---

Text-to- Video Retrieval
Video-to to Text Retrieval
Agg
R@
R@5 1
@@ 10 f
MdR
MnR
R@L
R@5 f
R @ 10
MdR
MnR
MSR VTT (Ik)
Clip4Clip
44.10
71.70
81.40
2.0
15.51
42.09
71.24
81.23
2.0
12.01
DuallS
44.20
71.70
81.60
2.0
15.64
44.86
72.04
82.02
2.0
11.56
DualDIS
45.00
72.50
82.10
2.0
15.32
45.45
73.02
81.42
2.0
11.59
DuallS
45.00
72.50
82.10
2.0
15.32
45.45
73.02
81.42
2.0
11.56
DualDIS
x
45.00
72.50
82.10
2.0
15.32
45.45
73.02
81.42
2.0
11.59
ActivityNet
CLIP4Clip
41.85
74.44
84.84
2.0
6.84
41.62
74.11
86.12
2.0
6.81
DualIS
46.40
77.85
87.21
2.0
6.14
47.02
77.59
87.49
2.0
6,16
DualDDS
46.26
77.78
86.90
2.0
6.11
47.09
77.21
87.38
2.0
6.31
DuallS
x
46.71
77.47
87.35
2.0
6.01
46.59
78.04
88.15
2.0
6.05
DualIDIS
46.76
77.48
87.28
2.0
6.01
46.73
77.90
88.06
2.0
6.05
Table 14; Retrieval results on MSR-VTT and ActivityNet with DBNORM(DualIS) and DBNORM(DuaDIS)) "Agg" refers to the aggregation methods, "x" and "4" represent multiplying and addition.
Teat-to Video Retrical
Video-to-Teed Retrival
QB
GB
wile test
Ri-11
MIR
MaR
Stewmess
R
R
RG
MiR
MoR
Skewness
MISRVVTT (Ik)
ClijsClip
44.10
71.70
1.38
3,65
GC Ouperr hound)
0.45
70:35
18:71
GC
40.20
65.10
75.60
2.0
24.32
3:33
40.14
67.33
76.59
2.0
15.58
4.71
CSLS (upeer bound, 10000)
70
70.70
80.70
2.0
15.03
0.81
42.50
70.35
80.43
2.0
12.84
0.85
with
with 100
70.60
2.0
0.92
41.11
CSS S with 1000
70
71.00
80.70
2.0
1601
0.50
42.19
70.36
80.43
2.0
13.11
0.92
IS
44.20
71.70
81.60
2.0
15.64
0.82
44.86
72.04
$2.02
2.0
11.56
2.20
DIS
11.61
2.54
DualiS
72.50
2.0
0.37
73.02
Kl,42
2.0
2.48
DualDES
45.00
72.50
82.00
2.0
15.32
0.37
45.45
73,02
81.42
2.0
11.59
2.51
ActivityNa
CLIP4Clip
74.44
0.94
1.06
GC (upper bound)
2.0
0.13
47.09
77
GCC
33.89
63.22
76.74
3.0
9.91
2.19
32.23
61.18
75.39
3.0
10.11
56,88
CSLS (upper bound, (222)
40.29
73.00
84.46
2.0
7.32
0.79
40.90
73.38
85.53
2.0
7.04
0.92
70:46
1,30
72.50
2.0
7.36
0.90
73.05
7.11
CS1S 1000
40.15
72.93
84.41
2.0
7.34
0.81
40.74
73.38
85.53
2.0
7.04
0.92
CSLS
40 29
72.93
86.49
2.0
7.32
0.79
40.83
73:33
85.41
2.0
7,05
0.94
77.52
02
77.29
86.83
2.0
0.68
46.
76
2.0
DuallS DuilSS
46.71
77.47
87.38
2.0
6.81
0.59
46.99
76.04
$8.15
2.0
6,05
0.30
46.76
77.48
87.78
2.0
6,01
0.50
46.73
77.90
$8.06
2.0
6.85
0.34
Table 15: Retrieval results on MSR-VTT and ActivityNet with GC, CSLS, IS, DIS, DBNORM(DualIS) and DBNORM(DuaIDIS) "QB", "GB", and "w/o test" refers the query bank, gallery bank, and without the access all test gallery data (only observing one gallery data at time).
conducted experiments by replacing multiplication with addition. The retrieval results are presented in Table 14. It is observed that, for both multiplication and addition yield compara- ble retrieval performance, whereas for ActivityNet, multiplication outperforms addition, providing em- pirical evidence for the superiority of multiplica- tion.
to address hubness in Natural Language Processing. Note that, GC and CSLS requires the access all queries.
However, in real-world settings, queries do not always occur simultaneously, and is common for methods not to have access to all queries at once. This aligns with the practical conditions where queries are generated in sequential or asyn- chronous manner. To test the compatibility of GC and CSLS with such scenarios, CSLS accordingly and conduct experiments on the MSR-VTY and ActivityNet datasets.
B.6 Comparison with GC and CSLS
adapt GC and
Globally-correeted (GC) (Dinu al., 2015) and Cross-Domain Similarity Local Scaling (CSLS) (Lample et al., 2018) are two representa- tive post-processing methods specifically designed
GC returns the gallery element g that has the highest rank for query q. The similarity is normal-

---

ized as follows,
We notice that,
8q.8. = Sq.88 Rg Q.q).
(4)
Ex
X2
x x]
where (g, Q. q) returns the rank of q in Q when the query is gi and Q contains all the queries. To investigate the applicability of GC without ac- cessing other test queries, we modify GC as fol- lows,
Ex
X|
/ (x 12)1
F,x
Xx2
14| |] + [[|x 1e]
21E
(xx (4) T (x -11)
=E [|x [2] E [||x / 122]
$q.g. = Sq.g. Rank(g Q (q).a)) q)
(5)
The last equality is because the mean of X2 and x
where Rank(g; g: Q UUq. q, q) returns the rank of q in Q U q when the query is g..
is μ.
Similarly, we have
CSLS utilizes a query bank that consists of all the test queries and a gallery bank. To adapt CSLS for use in our method, we modify it to incorporate (train validation) queries and normalize the sim- ilarity sq,g, between query q and a gallery point gi as follows,
E [x] x1|²]
(9)
E |X]] - |el + E [lI - x x|
Next, we have,
A - X2 1e + E [1] x1
Σ KE|K]
8q.g: = 28q.8;
K
8q.8:
K
8q.9:
E |||x] [=]²] + E 1 x|| (10)
EE|K]
(6) = X 1e| | - |x| 14| > 0.
where (gi and (qth)c( are the k gallery and query samples in banks that are most similar to query q.
Now, we have completed the proof.
The quantitative results of GC and CSLS on MSR-VTT and ActivityNet are shown in Table 15, We denote GC and CSLS with access to test queries GC (upperbound) and CSLS (upperbound) re- spectively. It noteworthy that the retrieval perfor- mance significantly drops without access to all test queries, and the performance of CSLS is propor- tional to the hyperparamter k:
C.2 Proof of Theorem
The difference A is defined as,
A Ey x.xxx Mells - licllli--uaa = |||x--
yl ||| --1
(11)
We have,
Considering the unsatisfactory retrieval perfor- mance of GC and CSLS when lacking access to all test queries, we exclude these two methods from our approach.
E
X2 y|
E
||xc - M
(y 12))
EE
Xx2
|l 2 |y 1
C Proofs
2
(x2 1xx) (y 11xz)
C.1 Proof of Theorem 1
=E
x2 uzl 2 + EE Iyy p 11
Consider two data points and x2 sampled from (H),, which satisfy
Similarly, we have,
Xx2 e11l ||x1 1 > 0.
(7)
E |x| y| = E 1all2 +
Ely
1all?
The expected difference between the squared Euclidean distances from x₁ and X2 to a point x sampled from the same distribution, is defined as
Inserting the above two equality into Eq. (11), we have
A= =
Ex
X2
x||
|x]
x||²]
A - E |||x yl12 ||x -
(8)
Ex
X22
x
x
Ex
||X|
x
12
- X2 - M 2 - |x] - /1rl1 > 0.

---

C.3 Proof of Corollary 3
Proof. Consider two data points x1 and x2 sam- pled from a symmetric distribution X' and ran- dom point y sampled from a symmetric distribu- tion y on the surface of a hypersphere satisfying |x| = ||x2|22 = lyllz = r, and the following
We
only
need
to
prove that
E
x2 y||
|x| - y
0. we have
E
X2 - -x|
|x| - =x
0, where
As E [[x2 - -y| ||x| y|1
>
0, with the
holds true
proof of Theorem 2, have
COs(XL, (2) (xs( Mt) > 0.
(14)
|x2 pz |||| Fx ||2 0.
where M4 is the center of the distribution X. That means under the cosine metric, X1 is closer to the mean point than X2.
In other words, as ||2-
plx
||
|x| Mel > 0.
we have
The expected difference between the cosine dis- tances from x1 and X2 to the random point y, is defined
1|2 Mxll |||1
pex > O.
With
Theorem
1,
we
have
E ||x x|| ||| x| 0.
A = E [cos(x1, y) - cos(x2 - y)]
Now, we complete the proof.
E Xiy" - X2X 1]
72
C.4 Proof of Theorem 4
1 E[x1 x21 - 1b2
Similarly, we first present the following theorems. Theorem Assuming that X1 and X2 are sampled from a symmetric distribution X on a hypersphere with mean μ. if xı closer to than X2 (i.e., cos(X1, 14) cos(x2, 1)), then Xı is more likely to be a hub than on the space X.
(15)
22
1
72 B[xi x2 T
= cos((1, [e) - cos((ox. (x2, u2z) > 0.
Now, we have completed the proof.
Proof. Consider two data points x₁ and X2 sam- pled from symmetric distribution the surface of hypersphere satisfying ||x1||2 ||x la and the following holds true
Combining the above two theorems and with the proof of Corollary 3, have completed the proof.
( p1) cxs(X2, u1) > 0.
(12)
where μ is the center of the distribution. That means under the cosine metric, x is closer to the mean point than X2.
The expected difference between the cosine dis- tances from X1 and X2 to random point x sampled from the same distribution, is defined as
[cos(x1,, x) cos(x2 (x2 x]]
E x x2x
(13)
E|x1 x22le
= cos((x1, 11) cos X2, (1) 0.
Now, we have completed the proof.
Theorem 6. Assuming that Xı and X2 are sam- pled from symmetric distribution with mean piz- if X1 is closer to a than X2 (i.e., cos(xe X1; v.) cos(x2, 4)), then x1 is more likely to be hub than X2 on another space y


Reviews:
[{"review_id": "AcvDDRdIop", "reviewer": "Reviewer_rioa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. We have carefully considered your comments and have provided responses to specific comments.\n\n**Reasons To Reject [1, Statistics about hubness in CMR]:**\n\n> Thank you for your suggestion! \n> 1. In the submitted paper, following [1], we included the **skewness score, or hubness score**, as a quantitative measure of the hubness problem. Comprehensive data on these scores across various benchmarks can be found in Table 6 (Page 7) of the main paper and Table 13 (Page 22) in the appendix. Notably, we observed a notable reduction in the skewness score, after employing DualIS and DualDIS, proving the effectiveness of our methods in alleviating hubness. \n> 2. To provide a visual representation of the **distribution of 1-occurrence**, we have included them in Figure 3 (Page 8) in the main paper and Figure 6 (Page 21) in the appendix. Intuitively, these figures indicate a clear pattern where a majority of samples are retrieved only once, while a smaller subset of samples experiences frequent retrievals. \n\n**Reasons To Reject [2, Analysis about some lower performance of DualDIS compared to our proposed DualIS]:**\n\n>Thank you for your suggestion! Due to the limitation of space, we were regrettably unable to present a comprehensive analysis of our proposed methods. We will include a detailed analysis in the final version of our paper.\n>\n>1. As DualDIS only normalizes certain similarities within a subset chosen by a small bank, the sampling bias of that bank could lead to worse performance under certain scenarios compared with DualIS. As suggested by our theorems, a larger bank always leads to better performance because a more precise estimation of distribution can be made.\n>\n>2. On the other side, in our evaluation, we did observe that DualDIS, from a statistical perspective (hubness score), performs nearly as well as DualIS in mitigating the hubness issue. Both sets of results are illustrated in Table 6 and Table 13. We believe that the small performance drop of DualDIS compared with our proposed DualIS is due to the misalignment in retrieval performance and the hubness score. This is discussed in the Limitations section, where we acknowledge that the hubness score (skewness score) does not exhibit an absolute correlation with retrieval performance (Line 512 to Line 515). \n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\n[1] Milos Radovanovic, Alexandros Nanopoulos, and Mirjana Ivanovi. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research, 11(86):2487–2531.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. theoretical analysis of the hubness problem in Cross-Modal Retrieval.\n2. the experimental results on multiple task.\n3. the post-processing method that integrates multiple improvements.", "reasons_to_reject": "1. The paper lacks statistics about the hubness problem in Cross-Modal Retrieval.\n2. It lacks the analysis about some lower performance with DBNorm (DualDIS).", "questions_to_authors": "", "paper_topic_and_main_contributions": "**Topic:** This paper aims to address the hubness problem in Cross-Modal Retrieval.\n\n**Contribution:**\n1. provide the  necessity of incorporating both the gallery and query data;\n2. propose a post-processing method that  contains  Dual bank Normalization, Dual Inverted Softmax and Dual Dynamic Inverted Softmax;\n3. provide experimental results on multiple task.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qjyo7qiQ4q", "reviewer": "Reviewer_htxD", "summary": "", "questions": "Q1: The proposed method shows significant improvements on the VTT task, but I notice the improvements are not significant on the other tasks, i.e., MSVD task. What leads to this difference in the same Text-to-Video setting?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. We have carefully considered your comments and have provided responses to specific comments.\n\n**Questions For The Authors [1, Performance] / Reasons To Reject [1]:**\n\n> Thank you for your question! \n>\n> 1. Our proposed methods, DualIS and DualDIS, rely on the **normalization of similarities** based on both query and gallery banks. It's important to acknowledge that the **sampling bias of query and gallery banks** could lead to unstable performance under certain scenarios. Additionally, in the design of DualDIS, which normalizes only specific similarities within a subset chosen by a small bank, the sampling bias has been amplified which could lead to suboptimal distribution estimation for embeddings. As suggested by our theorems, a larger bank can lead to better performance because a more precise estimation of distribution can be made. \n>\n>2. On the other side, in our evaluation, we did observe that DualIS and DualDIS, from a **statistical perspective (hubness score)**, perform better than QBNorm (IS and DIS) in mitigating the hubness issue as shown in Table 6 and Table 13. We believe that the retrieval performance is due to the misalignment in retrieval performance and the hubness score, along with the limitation posed by the number of available test data. This is discussed in the Limitations section, where we acknowledge that the hubness score (skewness score) does not exhibit an absolute correlation with retrieval performance (Line 512 to Line 515). \n>\n> We will include this analysis in our final version paper. \n\n**Reasons To Reject [2, Effectiveness of gallery bank]:**\n\n>Thank you for your comments! We agree. The effectiveness of the gallery bank is somewhat overshadowed by the presence of a modality gap, as highlighted in [1]. Modality gap contributes to the challenges in accurately estimating distribution and alignment. But we do believe that a large bank (a more precise estimation) will better reduce the hubness problem.  \n\n**Typos Grammar Style And Presentation Improvements:**\n\n>Thank you for your suggestions! We will carefully revise our paper based on your suggestions! \n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\n[1] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Advances in neural information processing systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 Two post-processing methods are proposed to mitigate the hubness by reducing the similarity between hubs and queries.\n2 The authors conduct extensive experiments to evaluate their proposed method on eight cross-model retrieval datasets. \n3 The paper is well-written and easy to understand.", "reasons_to_reject": "* The proposed methods (DualIS and DualDIS) are not generic on some cross-model retrieval tasks, i.e., the performance in MSVD (Table 3) shows minor improvements. \n* I think the proposed gallery bank is supplementary and less effective compared to the query bank to address hubness issues in cross-model retrieval tasks. This conclusion is also verified by the authors' experiments.", "questions_to_authors": "Q1: The proposed method shows significant improvements on the VTT task, but I notice the improvements are not significant on the other tasks, i.e., MSVD task. What leads to this difference in the same Text-to-Video setting?", "paper_topic_and_main_contributions": "The authors proposed a framework named Dual Bank Normalization to address the hubness issue in cross-modal retrieval tasks. \nThe main contribution can be summarized as follows:\n * The authors find the hubness issue also related to the gallery, which is neglected in existing cross-modal solutions.\n * Two post-processing methods are introduced to normalize the similarities in query bank and gallery bank. \n * The authors give a theoretical demonstration that hubs express high similarity in different data modalities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall, the paper is well writen to read and understand.\n* I suggest more details can be provided in algorithm 1 and improve its presentation.\n* The 4th contribution (open-source) can be moved to the footnote.\n* Highlight RQ1 and RQ2 in bold should be better (pp. 2)."}}, {"review_id": "YxyYGveSTP", "reviewer": "Reviewer_GfvP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time, efforts, and your detailed and positive comments. We have carefully considered your comments and have provided responses to specific comments.\n\n**Reasons To Reject:**\n\n>We deeply appreciate your thorough feedback and would like to begin by highlighting our approach to mitigating the hubness problem [1,2]. Our initial step involves employing larger banks to normalize similarities, a strategy demonstrated to effectively reduce the hubness issue [1,2].\n\n>Building on this foundation, our contributions stem from two key directions:\n>\n>1. [**Theoretical foundation**] Similar to the query bank, the gallery bank holds the potential for identifying hubs that can contribute to hubness alleviation. This observation is underscored by Theorems 1 and 2, providing a theoretical basis for our approach. Moreover, our theoretical findings also lay the groundwork for QBNorm [2].\n>2. [**Novel framework**] In line with the theoretical insights, we propose DBNorm, supplemented by the DualIS and DualDIS. These techniques harness the power of dual banks to address the hubness problem with enhanced efficacy.\n\n> The fusion of these theoretical foundations and novel methodologies has enabled us to present a holistic solution to the hubness problem in CMR, thereby enhancing the robustness and applicability of our work.\n\nWe sincerely thank you for your thoughtful comments and are open to further discussion on these matters.\n\n[1] Milos Radovanovic, Alexandros Nanopoulos, and Mirjana Ivanovi. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research, 11(86):2487–2531.\n\n[2] Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. 2022. Cross modal retrieval with querybank normalisation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5184–5195. IEEE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed DBNorm approach addresses a fundamental problem common to most CMR models and has the potential to improve the performance of cross-modal retrieval systems in general.\n2. The paper provides a rigorous theoretical demonstration of the context of the problem and the motivation for the key idea of the solution (namely, normalizing on both query and gallery data). Similarly, the proposal of DualIS and DualDIS is well-motivated by prior work.\n3. The paper provides quantitative experimental results of the method on 8 language-based benchmarks of different cross-modal modality settings (4 text-video retrieval benchmarks, 2 text-image retrieval benchmarks, and 2 audio-text retrieval benchmarks), and convincingly demonstrates that the proposed DBNorm approach outperforms previous normalization methods based on SOTA models in a variety of settings.", "reasons_to_reject": "The designs of the main contributions appear to be a small increment from the previous research (Bogolin et al., 2022), which identified the hubness problem in CMR and proposed the single-bank normalization method DIS (on which DualDIS is based). This is partially mitigated by the theoretical foundation of DBNorm and the extensive experimental studies demonstrating superior performance, which improves the soundness of this work by providing a clear motivation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Dual Bank Normalization (DBNorm), a general post-processing solution that addresses the hubness problem in cross-modal retrieval, which is a phenomenon where a small number of gallery data points are frequently retrieved, resulting in a decrease in retrieval performance. The main contributions of the paper are:\n1. introducing DBNorm, a theory-grounded novel framework, as a general solution to mitigate the hubness problem in the context of cross-modal retrieval (CMR), along with two post-processing methods: Dual Inverted Softmax (DualIS) and Dual Dynamic Inverted Softmax (DualDIS),\n2. demonstrate through experimental results on a diverse set of benchmarks that DBNorm outperforms previous methods in addressing hubness and improving retrieval performance.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a  new query and gallery banks method to mitigate hubness in cross-modal retrieval. The research is original and the methodology clear. The results and experiments are well-supported.

Pros:
* The theoretical analysis of the Hubness in a Cross-Modal Retrieval scenario
* Extensive experimentation for different Cross-Modal retrieval datasets.

Cons:
* Minor improvements when compared to previous research.
Decision: Accept-Main